feat: improve reliability #8

Open

phbaer wants to merge 9 commits from feat/improve-reliability into main

pull from: feat/improve-reliability

merge into: phbaer:main

phbaer:main

phbaer:renovate/astral-sh-setup-uv-10.x

phbaer:feat-add-support-for-litellm

Conversation 0 Commits 9 Files changed 180 +33651 -4212

phbaer commented 2026-09-25 05:23:50 +02:00

Owner

• Copy link

No description provided.

phbaer force-pushed feat/improve-reliability from 74cea98af6 to a693b3a70c 2026-09-25 08:13:47 +02:00 Compare

phbaer force-pushed feat/improve-reliability from a693b3a70c to 8ff8372dcc 2026-09-25 08:34:44 +02:00 Compare

phbaer force-pushed feat/improve-reliability from 8ff8372dcc to e7263c8465 2026-10-23 15:00:56 +02:00 Compare

phbaer force-pushed feat/improve-reliability from 1571cd516f to 411692c036 2026-10-27 05:34:19 +01:00 Compare

phbaer force-pushed feat/improve-reliability from 68d2c19471 to 85f9431cd7 2026-10-27 05:44:56 +01:00 Compare

phbaer force-pushed feat/improve-reliability from 7235d85d86 to 935fde68cc 2026-10-30 08:44:47 +01:00 Compare

phbaer added 1 commit 2026-10-30 11:30:24 +01:00

feat: inspect published Maven dependencies and release sources 9821faa0a4

phbaer added 3 commits 2026-11-02 11:46:02 +01:00

feat(analysis): run source review as a budgeted agentic investigation ... e47c152d26

Replace the fixed four-round, three-calls-per-turn research loop with a
budgeted investigation driven by the model. Reachability, deep, and
transitive reviews now continue until the model answers or a turn,
tool-call, tool-output, or stage-time budget ends the loop, configured by
AGENTYZER_RESEARCH_MAX_TURNS, AGENTYZER_RESEARCH_MAX_TOOL_CALLS,
AGENTYZER_RESEARCH_MAX_TOOL_CALLS_PER_TURN,
AGENTYZER_RESEARCH_MAX_TOOL_OUTPUT_CHARS, and
AGENTYZER_SOURCE_REVIEW_TIMEOUT_SECONDS. A midpoint budget check and a
"stop when decided" prompt rule steer the model to finalize once version
evidence or a verified path settles the result.

Add eight bounded, read-only repository tools over the indexed snapshot:
directory listing, glob, regex or literal grep with context, file outline,
enclosing-function or line-range reads, symbol references, literal search,
and Git history. Tool output is line-numbered plain text inside the
existing untrusted-evidence markers and records served spans for citation
verification. Regex search rejects backreferences, nested quantifiers, and
alternation inside quantifiers and stops after a ten-second budget. Index
application directories before vendored trees so node_modules cannot
exhaust the symbol budget before src/ is reached.

Rewrite the reachability and deep-review prompts as investigation
protocols (orient, locate, trace callers to entrypoints, verify
activation, check mitigations, finalize), treat pre-selected excerpts as
starting hints, allow working notes during tool turns, and state that
project code resembling the weakness is not the advisory's vulnerable
implementation. Structured final contracts are unchanged.

Implement native Ollama tool calling through /api/chat with
AGENTYZER_OLLAMA_TOOL_CALLS and automatic fallback to text directives for
models that reject tools. Surface an investigation summary (tool turns,
tool calls, files read, searches) in step evidence, and let synchronous
CLI assessments wait up to one hour.

fix(evidence): classify vendored manifests, pin Maven POM versions, surface runtime images ... df229a48ea

Live assessments showed verdicts limited by inventory handling rather than
by missing evidence. Three evidence rules change:

Manifests and locks below vendored trees (node_modules, vendor,
site-packages, and similar) belong to installed packages, not to the
project. They no longer make a dependency direct; a nested manifest that
declares the assessed package is recorded under nested_manifests with the
installed package's name and constraint, which identifies the intermediary
and keeps the transitive call-path review enabled. Version inventories
ignore vendored manifests in the worktree and in historical refs.

Maven has no lock file. When no lock inventory exists and every pom.xml
declares the direct dependency with one exact, fully resolved version, the
scanner records manifest-pinned dependency instances and the version
analyzer records the same manifest_pinned_version instead of the "no
resolved lock inventory" gap, propagated into the assembled version
summary. Ranges, unresolved ${property} values, LATEST/RELEASE, and
disagreeing POMs leave the inventory incomplete. The exclusion recheck
accepts a lock-free proof only when both sides agree. SBOM versions that
all compare outside the affected ranges are noted rather than recorded as
an inventory gap.

Dependency scanning collects declared runtime images from Dockerfile FROM
references and Compose image entries with file and line provenance, flags
images whose name matches the assessed package, and supplies them to the
review prompts, step evidence, and Markdown report as declared host
runtime context. They are never a verified deployed artifact or
dependency version and do not change the deterministic verdict.

feat(verdict): accept verified version exclusions with unknown reachability ... f6aa0239d6

A patched version is patched regardless of whether its API is reached. The
independent version-exclusion recheck therefore no longer treats unknown
first-pass reachability or an uncertain transitive review as blocking;
a reported reachable path, YES/LIKELY exploitability, or a YES/LIKELY
transitive path still contradicts the version evidence and keeps the
result unresolved. The audit view records unresolved reachability as a
note instead of a concern when the exclusion is verified, and the
exclusion rationale states that reachability was not established and
names a Maven POM pin when it applies.

The decision, its drivers, the alternatives considered, consequences, and
confirmation criteria are recorded in the analysis reference. A live
assessment of a Keycloak extension pinned above the fixed release moved
from Inconclusive to Not Affected with a passing audit; the end-to-end
gate test on a pinned Maven fixture covers both directions.

Lower the default investigation turn budget from 24 to 12 after live runs
showed the extra turns re-confirming decided facts; the midpoint reminder
and tool budgets are unchanged.

phbaer force-pushed feat/improve-reliability from f6aa0239d6 to 69bef6e625 2026-11-02 12:52:24 +01:00 Compare

phbaer added 2 commits 2026-11-02 13:48:45 +01:00

feat(analysis): read unindexed dependency sources on demand ... d6970f6993

The syntax index is budgeted and, on Node projects, exhausted its symbol
budget inside node_modules before reaching the packages that matter. Local
investigation tools only served indexed files, so the model could request
the intermediary's request helper and be told the directory was not part
of the snapshot.

- Build the index with the installed packages on the lock-file dependency
  chain to the assessed component first (dependency_source_prefixes,
  SourceIndex.build priority_prefixes), then application directories, then
  the remaining vendored trees.
- Let list, find, grep, outline, read, and history fall back to disk for
  repository paths the index skipped, under the same path-safety rules (no
  symlinks, no traversal, text files only) and separate file, byte, walk,
  and time budgets. On-demand output is labelled "not indexed", counts as
  served evidence for citation checks, and is reported as on_demand_files
  in coverage without inflating the indexed-file count.
- Update tool descriptions and investigation instructions so the model
  knows vendored sources are readable when it names their path.

feat(verdict): keep unverified deep-review mitigations out of Affected ... 54ee6795c3

When the deep source review reports EXPLOITABLE NO with a cited code-level
mitigation that the independent exclusion checks could not verify, the
first-pass and transitive reviews reporting the dependency API as reachable
no longer escalate the result to Affected. The result stays Probably
Affected, the deterministic basis names the unverified mitigation, and the
audit view records the contradiction as a concern with review status. An
unverified mitigation still cannot clear the assessment; YES/LIKELY and
UNCERTAIN handling are unchanged. The decision and its live benchmark
context are recorded in the analysis reference.

phbaer added 1 commit 2026-11-02 13:52:44 +01:00

fix(workflow): move from node to debian-ci e20edce2df

phbaer added 1 commit 2026-11-02 13:55:13 +01:00

build: pin the uv interpreter to Python 3.14 ... b48d718da5

CI started failing when the runner offered Python 3.15: uv selected it for
the ">=3.14" requirement, tree-sitter has no 3.15 wheels, and the runner has
no C compiler for a source build. Pin the interpreter uv uses with a
.python-version file so test, SBOM, and version steps resolve the same
interpreter as the container image.

phbaer added 1 commit 2026-11-02 13:58:51 +01:00

ci: install git on the debian-ci runner ... 031deabfa5

The debian-ci runner does not ship a git binary. GitPython fails at import
("Failed to initialize: Bad git executable"), which broke every test module
that imports the dependency scanner or version analyzer, and the version and
tag jobs cannot query history without it. Install git before checkout in
each debian-ci job so actions/checkout keeps a real repository.

phbaer added 1 commit 2026-11-02 14:00:48 +01:00

ci: install curl and gnupg before adding the Docker apt repository ... fa9962e169

The debian-ci runner ships neither curl nor gpg, so the Docker CLI step
failed before it could fetch the repository key.

phbaer added 1 commit 2026-11-02 14:41:19 +01:00

ci: install the Docker CLI from the static release archive ... a0b5e46a0f

Adding Docker's apt repository depends on the runner's Debian codename
being published there and failed on the debian-ci image. The static CLI
archive only needs curl and works on any glibc Debian release; buildx is
still provided by setup-buildx-action.

phbaer added 1 commit 2026-11-02 14:44:16 +01:00

ci: build and push the image on the node runner ... 19e25d3a9a

The image job needs a Docker daemon for registry login, buildx, and push.
The debian-ci image has no daemon, so the job failed at login after the
CLI was installed. The node runner built and pushed the image before the
runner move and keeps doing so; tests stay on debian-ci.

phbaer added 1 commit 2026-11-02 14:59:08 +01:00

ci: build images with daemonless buildah 0a135cdf69

phbaer added 1 commit 2026-11-02 15:02:33 +01:00

fix(ci): initialize Buildah auth file eed5504f59

phbaer added 1 commit 2026-11-02 15:05:06 +01:00

fix(ci): use VFS for image pushes 7a9a2181ef

phbaer force-pushed feat/improve-reliability from 7a9a2181ef to 18b9accafe 2026-11-02 15:12:15 +01:00 Compare

phbaer added 2 commits 2026-11-03 06:08:48 +01:00

docs: explain the analysis approach compactly ... f25647ce03

Rewrite the README "How it works" and "Understanding results" sections
around the one question an assessment answers, the four facts it
establishes and who decides each, how the model investigates a checkout,
and the four verdicts with what each needs. Add an "approach at a glance"
section to the analysis reference with the investigation loop, the verdict
table, and what the model cannot do, and move the SBOM special case below
the decision algorithm. Existing headings and anchors are unchanged.

feat(runtime): assess infrastructure components without a repository ... d5acb242df

Components such as a language runtime have no source to review. Until now
the registry's defaults template turned their name into a repository URL
and the assessment failed on the clone.

- Declare them as `kind: runtime` with a CPE in the registry, or send
  `component_kind: runtime`, `component_version` (from the SBOM), and
  `cpe` in the request; the CLI gains --kind, --component-version, --cpe.
  Runtime entries never resolve through the defaults template and are never
  cloned; a missing version is reported as Inconclusive before the pipeline.
- Parse NVD CPE version bounds and exact CPE versions into product-scoped
  constraints kept in their own advisory keys (cpe_ranges,
  cpe_version_entries, cpe_fixed_entries). Repository assessments keep
  using OSV/GHSA package ranges and never see them as package constraints.
- Add the assess_runtime stage after the relevance filter: match the
  advisory's CPE entries, compare the installed version with the bounds,
  compute remediation, record source, deep, and transitive reviews as not
  applicable, and continue directly with the verdict.
- Verdict: outside the bounds is Not Affected through a runtime-specific
  version exclusion covering the supplied version only; inside is Probably
  Affected with exposure "runtime" and a note that trigger conditions need
  confirmation; no identity or no bounds is Inconclusive; an advisory naming
  only other products is Not Affected as a filtered advisory, which now
  also counts as an audit strength.
- Progress counts the six-step runtime flow instead of the repository flow.

phbaer added 1 commit 2026-11-03 06:28:19 +01:00

feat(report): show how the verdict was reached with a decision trace ... 2f34a94623

The report's "Reachability and audit evidence" section listed each review
stage's reasoning as one long sentence per stage, so readers could not
see which fact produced the verdict or where an exploitability claim came
from.

Add build_decision_trace, derived from the same typed claims as the
deterministic verdict, exposed as assessment.decision_trace:

- inputs: one row per fact (advisory applies, dependency present, version
  vs. range, independent version exclusion, first-pass reachability, deep
  review, transitive path, confirmed vulnerability-specific path) with its
  finding, evidence pointer, and role: decides, supports, blocks
  escalation, gate, or n/a
- the rule that fired and what would change the verdict
- the audit status, consistency, supports, concerns, and notes
- each review finding split into its labeled Path / Evidence / Mitigation /
  Risk / Fix parts

The report renders it as "How the verdict was reached" and the CLI as
"Decision rationale"; executive_summary.why is unchanged and older results
without a trace fall back to it. The trace never changes a verdict.

phbaer added 1 commit 2026-11-03 09:33:29 +01:00

feat(verdict): keep affected versions with unknown reachability as Probably Affected ... 58a63b0a49

Most live assessments ended Inconclusive with the basis "current runtime
reachability or vulnerability-specific exploitability remains unresolved"
although the lock file already proved an affected version. The prompts
require affirmative evidence for YES and NO, so UNKNOWN and UNCERTAIN are
common answers, and the policy ranked a model that could not answer below
a model that answered and found nothing.

Decision (2026-11-03, recorded in the analysis reference): with the
dependency present at an affected version in the current workspace,
unknown reachability or uncertain exploitability now yields Probably
Affected with Low confidence, the dependency's exposure, and no
eligibility for automatic application or rescoring. A verified affected
shipped release keeps Medium confidence; without either version fact the
result stays Inconclusive. Affected still requires a confirmed path. The
decision trace states the rule for this case; existing tests keep their
property that unknown reachability never yields Affected or eligibility.

phbaer added 1 commit 2026-11-03 10:38:25 +01:00

feat(exclusion): scope the source-exclusion gate to the claimed path ... f34f53287d

The exclusion gate required a complete syntax index and no dynamic
dispatch anywhere in the repository. On a Node checkout that never holds
(unreadable vendored binaries, more than a thousand vendored files with
dynamic-dispatch limitations), so no verified claim could clear an
assessment and the proof machinery was dead code.

- Add a bounded repository-wide component import scan that reads every
  source file on disk, loads importing files on demand, and enumerates
  their call sites before any claim is checked; a truncated scan, an
  unreadable source file, or an importing file without a clean parse
  leaves the exclusion unverified.
- Replace the repository-wide gate with SourceIndex.exclusion_coverage:
  verified claims, a complete import scan, clean parsing of the cited
  files, no dynamic dispatch inside the enclosing definitions of cited
  lines, and no inventory gap in application source. Vendored gaps
  elsewhere, incomplete initial excerpt selection, and other stages'
  prompt compaction no longer block a proof that does not depend on them.
- Add the literal_option_at_sink claim condition for Python, JavaScript,
  and TypeScript: every call at the sink passes the named option as a
  literal, inside the argument itself or set on the argument variable at
  the guard line with no later spread, unpacking, assignment, duplicate or
  computed key, or other use before the call. Helper merges, callee
  defaults, and configuration hooks stay unverifiable by design.
- Define CONFIRMED in the deep prompt as a complete unmitigated path so a
  mitigated case is consistent with the verification gate.

Decision and rejected alternatives are recorded in the analysis reference.

phbaer added 1 commit 2026-11-04 05:08:25 +01:00

fix(research): finalize instead of failing when a research deadline expires ... 199a1f41b9

Stages without a response schema, such as the transitive path review, had
no finalization reserve: research could consume the whole stage deadline
and the loop re-raised the timeout as "LLM research stage deadline
exceeded", failing the job after the full budget was spent.

Every research stage now reserves 25% of its deadline (at most 120 s) for
the final answer. When the research phase expires, both the text and the
native tool-call loops record an incomplete research_deadline entry and
finalize the completed evidence, so the policy turns the run into
Inconclusive instead of a stage error. The transitive review passes its
parser and response contract as validator so its final answer is checked
like the other reviews.

phbaer force-pushed feat/improve-reliability from 199a1f41b9 to 8bd42e2bbd 2026-11-04 05:13:38 +01:00 Compare

All checks were successful

Hide checks

Build and Publish Agentyzer / delete-pr-image (pull_request) Has been skipped

Details

Build and Publish Agentyzer / test (pull_request) Successful in 1m1s

Details

Build and Publish Agentyzer / check-version (pull_request) Has been skipped

Details

Build and Publish Agentyzer / create-tag (pull_request) Has been skipped

Details

Build and Publish Agentyzer / build-push-image (pull_request) Successful in 1m4s

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin "feat/improve-reliability:feat/improve-reliability"

git switch "feat/improve-reliability"

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch "main"

git merge --no-ff "feat/improve-reliability"

git switch "feat/improve-reliability"

git rebase "main"

git switch "main"

git merge --ff-only "feat/improve-reliability"

git switch "feat/improve-reliability"

git rebase "main"

git switch "main"

git merge --no-ff "feat/improve-reliability"

git switch "main"

git merge --squash "feat/improve-reliability"

git switch "main"

git merge --ff-only "feat/improve-reliability"

git switch "main"

git merge "feat/improve-reliability"

git push origin "main"

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

phbaer/agentyzer!8

No description provided.