Skip to content

Verification evidence — 2026-10-10

Historical snapshot only. Counts, commands and limitations describe that snapshot and do not establish current acceptance. Use current verification.

Provider finish-reason and tool-feedback fixes (2026-10-10)

Behavioral regressions reproduced acceptance of incomplete model steps and a whitespace-only reply throwing at presentation. Six initial regression cases failed on the old behavior; the corrected Core suite subsequently passed. The final full matrix includes ten additional Core cases and two Protocol cases. A Length finish now rejects the whole step before tool proposal/dispatch and gets up to two same-model recovery notes requesting smaller output. Exhaustion reports model_output_truncated. ContentFilter and Error finishes fail immediately as model_content_filtered and model_reply_error; whitespace-only text uses existing empty-reply recovery. Suppressed steps retain typed finish metadata for internal audit and never enter the conversation outbox. This is not a heuristic that requires every legitimate reply to edit a file.

Tool descriptions/examples now clarify Windows workspace-relative paths, op=write plus overwrite=true for replacing existing files, exact fresh text for edits, and argument vectors excluding the executable. The file_exists repair hint includes the required operation. Schema digests, versions, grants, presets and approval behavior did not change. Stale recovery-count claims in architecture and tool guidance were corrected to the existing four unverified, four model-repairable and two empty notes, plus two new truncation notes, under the same cumulative twelve-note cap.

Verification:

  • scripts/test_dotnet.ps1 -Configuration Debug: 5,849 passed, no failures or skips across 12 projects (artifacts/benchmark-fixes-dotnet.log).
  • .venv\Scripts\python.exe -m pytest tests -q: 211 passed, 1 skipped (artifacts/benchmark-fixes-pytest.log).
  • build.bat: 11/11 stages passed, refreshed flat Release distribution (artifacts/benchmark-fixes-release.log).
  • Behavioral RED evidence: artifacts/model-finish-red.log (six failures); the first Core GREEN run passed 474 before four additional cap/text-only cases brought the final Core count to 478.

A small C# shipping diagnostic reused the earlier prompt and pristine baseline on the new flat Release binaries with the existing Nemotron/OpenRouter selection, standard and a 600-second deadline. A first local setup was excluded: run creation returned invalid_workspace before model execution while Host data and workspace were colocated under one artifact parent. A second setup with a separate TEMP data root created a run, but parked after 43.41 seconds with provider_rate_limited, before any tool call or source edit. The runner cancelled it; the persisted state is terminal, the owned Host shut down, and the independent verifier still failed the exact-threshold check. This is provider interruption, not a measured coding improvement or a new corpus score. Artifacts are under artifacts/ru-small-csharp-fixes-20261010 (excluded setup) and artifacts/ru-small-csharp-fixes-20261010-run2 (live diagnostic). Provider/model selection was not changed and no approval/Continue was supplied. The new TEMP data directory remains locally with the diagnostic evidence; no recursive cleanup was retried after the earlier automatic review rejection. Native UI, crash-orphan collection and another complete live corpus run remain unverified.

Post-refactor live coding rerun (2026-10-10)

The owner requested a live rerun after sandbox/client simplification. All nine custom Terminal/SWE-style tasks ran sequentially, once each, using the existing selection models.dev:openrouter / nvidia/nemotron-3.5-lightning:free / chat_completions. No provider/model switch, approval grant, Continue, or task patch intervention was performed. The runner used standard, fresh trusted workspaces/sessions, the task's 1,800-second agent deadline and 120-second external verifier deadline. .NET SDK was 10.0.303, Node v24.13.0, and Python 3.13.14 on native Windows. Grading remains native-local, outside an OS sandbox.

The flat Release CLI and Host from the preceding verified build.bat were used. Host execution authority runs in process; there is no Broker executable. The checkout is 1f383ab0; binaries were built from the changes before that commit, so the Host's embedded version still names base revision 4d925e4d. The manifest retains executable and eleven assembly SHA-256 fingerprints. A separate Host used a free loopback port and fresh TEMP data/log roots; only non-secret provider settings and catalog data were copied in. Credentials were read through the existing Windows Credential Manager. The user's normal data root was not migrated or modified by this run.

Artifacts are under ignored artifacts/ru-live-post-refactor-20261010: manifest.json, report.json, analysis.json, public tool-payload analysis, terminal-state evidence, per-task trajectories, verifier results and patches. The local driver artifacts/run-live-20261010.py invokes the existing benchmark.run_live one task at a time and checkpoints the aggregate report. Two setup probes failed before any scored run (legacy settings had no SQLite business table; a doubled URL separator prevented readiness). Both were fixed in this local driver. No product or benchmark-runner source changed.

Wall time was 41.00 minutes, summed agent time 40.77 minutes. Strict reward was 0/9, Terminal 0/6 and SWE 0/3. Three runs reported completed, but all three left source unchanged and failed the verifier. Six stopped for approval and were cancelled; persisted final states confirm all nine runs are terminal (three completed, six cancelled). Only one candidate changed source, and none passed every verifier group. Partial passes on unchanged starters are not improvements. All event trajectories caught up to the authoritative cursor. Token totals and actual billing remain unknown.

Task Agent minutes Stop Candidate groups passed Source changed
cs-atomic-ledger 0.17 External directory approval (list /) 0/7 No
cs-single-flight-cache 6.66 Completed without patch 5/8 No
py-dependency-scheduler 0.22 External directory approval (/workspace/...) 3/6 No
py-stream-parser 4.58 Completed without patch 2/6 No
ts-generation-pager 14.76 Unknown executable approval (node) 3/6 src/pager.ts
ts-transaction-store 7.97 Completed without patch 4/7 No
cs-lease-renewal 2.52 Unknown executable approval (cat) 2/7 No
py-migration-rollback 2.79 Terminal external-mutation approval 2/6 No
ts-session-replay 1.10 Unknown executable approval 2/7 No

Observed failures include invented Linux paths, attempts to read directories as files, create-only writes to existing files (file_exists), mismatched edit text (edit_no_match), and missing required tool arguments (malformed_call). Pager eventually used a valid overwrite, but its patch still failed shared promise identity, stale-generation isolation and refresh behavior. Its terminal call redundantly put node in the argument list as well as the executable. Cache and parser returned long public final text (33,203 and 32,715 characters) without edits. Their exact provider finish reasons were not retained in the public final-text payload, so output truncation is a hypothesis, not an isolated cause. Source inspection separately shows the Coordinator routes a non-empty text-only step to final handling without distinguishing Length; this warrants a focused regression before changing completion behavior.

These results identify model/tool-use, unattended approval and completion problems under this selection. They do not prove a regression from Step 5: that pilot used another provider/model and older runtime. One attempt per public custom task cannot establish general coding quality or parity with other agents. Crash-orphan sandbox cleanup and interactive native GUI/TUI acceptance remain open. Both the trial Host and evidence-replay Host shut down cleanly. Automatic approval review rejected deletion of the three owned trial TEMP directories (blocked by policy); those data/log directories remain locally. The documentation/count guards passed 19 tests and git diff --check passed after this evidence update. The full .NET/Python matrix and Release build were not repeated for this documentation-only change; their preceding results remain recorded below.

Sandbox lifecycle and client simplification (2026-10-10)

Session close, directory-grant revocation and async Host shutdown cancel and drain terminal pumps before sandbox release. Closing a foreign run leaves its lease and execution untouched. Disposed authority/runtime instances reject new work. Recreated sandbox entries use fresh profile names/SIDs; failed extensions roll back new grants, and failed creation removes its temporary directory. Crash orphan profiles/ACEs still need eventual cleanup, but are not revived by a new entry. A failed drain remains blocked and can be retried.

WinUI and CLI share the event-feed reconnect loop and durable cursor handling. Callback failures propagate instead of being retried as transport failures. CLI parking retains durable text recovery and leaves the server run available for Continue. Preset and durable provider calling-mode conversions use shared codecs. Managed Git preserves sandbox start failures through its readers; workspace discovery uses the current checkout marker instead of retired src.

The final full invocation passed all 5,837 tests:

Project Passed
Broker 928
CLI 1,279
Client 211
Harness Core 468
Harness Persistence 176
Hosting 39
Protocol 86
Runtime Adapters 195
Runtime Worker 69
Session Interchange 43
Backend 1,040
GUI 1,303
Total 5,837

artifacts/simplification-full-dotnet.log retains the first invocation: Broker 927 passed, then CLI 1 failed / 1,278 passed because cancellation ended the newly shared subscription before parked-run payload recovery. The fix preserves the existing assertion; seven focused park/resume checks and artifacts/simplification-full-dotnet-final.log passed. Focused RED experiments and corrected fixture failures remain in artifacts/simplification-*.log. The final GUI compile has neither the prior nullable highlighter warning nor the filtered Assert.Single analyzer warning.

The first Python run was 1 failed / 210 passed / 1 skipped: a partial docs update changed the README count before the verification table, because the update script used the Windows default text encoding. The UTF-8 update now keeps both counts aligned; artifacts/simplification-pytest-final.log records 211 passed / 1 skipped, including route, tool, docs and benchmark guards.

Commands:

.\scripts\test_dotnet.ps1 -Configuration Debug
.venv\Scripts\python.exe -m pytest tests -q
.\build.bat
.\scripts\publish_smoke.ps1

Release packaging passed 11/11 stages. Both the standalone publish smoke and the flat distributable smoke passed authenticated /status and clean /host/shutdown, using isolated temporary data/log roots and synthetic bearer authority. The flat smoke used a retained local copy of the canonical smoke script with publishing replaced by the existing flat output; it is validation evidence, not an additional production launch path. Logs: artifacts/simplification-release-build.log, artifacts/simplification-publish-smoke.log, artifacts/simplification-flat-smoke.log.

Real-provider task quality and interactive native UI acceptance are separate and were not exercised by the deterministic matrices.