Verification evidence — 2026-10-10¶
Historical snapshot only. Counts, commands and limitations describe that snapshot and do not establish current acceptance. Use current verification.
Provider finish-reason and tool-feedback fixes (2026-10-10)¶
Behavioral regressions reproduced acceptance of incomplete model steps and a
whitespace-only reply throwing at presentation. Six initial regression cases
failed on the old behavior; the corrected Core suite subsequently passed. The
final full matrix includes ten additional Core cases and two Protocol cases.
A Length finish now rejects the whole step before tool proposal/dispatch and
gets up to two same-model recovery notes requesting smaller output. Exhaustion
reports model_output_truncated. ContentFilter and Error finishes fail
immediately as model_content_filtered and model_reply_error; whitespace-only
text uses existing empty-reply recovery. Suppressed steps retain typed finish
metadata for internal audit and never enter the conversation outbox. This is
not a heuristic that requires every legitimate reply to edit a file.
Tool descriptions/examples now clarify Windows workspace-relative paths,
op=write plus overwrite=true for replacing existing files, exact fresh text
for edits, and argument vectors excluding the executable. The file_exists
repair hint includes the required operation. Schema digests, versions, grants,
presets and approval behavior did not change. Stale recovery-count claims in
architecture and tool guidance were corrected to the existing four unverified,
four model-repairable and two empty notes, plus two new truncation notes, under
the same cumulative twelve-note cap.
Verification:
scripts/test_dotnet.ps1 -Configuration Debug: 5,849 passed, no failures or skips across 12 projects (artifacts/benchmark-fixes-dotnet.log)..venv\Scripts\python.exe -m pytest tests -q: 211 passed, 1 skipped (artifacts/benchmark-fixes-pytest.log).build.bat: 11/11 stages passed, refreshed flat Release distribution (artifacts/benchmark-fixes-release.log).- Behavioral RED evidence:
artifacts/model-finish-red.log(six failures); the first Core GREEN run passed 474 before four additional cap/text-only cases brought the final Core count to 478.
A small C# shipping diagnostic reused the earlier prompt and pristine baseline
on the new flat Release binaries with the existing Nemotron/OpenRouter
selection, standard and a 600-second deadline. A first local setup was excluded:
run creation returned invalid_workspace before model execution while Host data
and workspace were colocated under one artifact parent. A second setup with a
separate TEMP data root created a run, but parked after 43.41 seconds with
provider_rate_limited, before any tool call or source edit. The runner cancelled
it; the persisted state is terminal, the owned Host shut down, and the independent
verifier still failed the exact-threshold check. This is provider interruption,
not a measured coding improvement or a new corpus score. Artifacts are under
artifacts/ru-small-csharp-fixes-20261010 (excluded setup) and
artifacts/ru-small-csharp-fixes-20261010-run2 (live diagnostic). Provider/model
selection was not changed and no approval/Continue was supplied. The new TEMP
data directory remains locally with the diagnostic evidence; no recursive
cleanup was retried after the earlier automatic review rejection. Native UI,
crash-orphan collection and another complete live corpus run remain unverified.
Post-refactor live coding rerun (2026-10-10)¶
The owner requested a live rerun after sandbox/client simplification. All nine
custom Terminal/SWE-style tasks ran sequentially, once each, using the existing
selection models.dev:openrouter / nvidia/nemotron-3.5-lightning:free /
chat_completions. No provider/model switch, approval grant, Continue, or task
patch intervention was performed. The runner used standard, fresh trusted
workspaces/sessions, the task's 1,800-second agent deadline and 120-second
external verifier deadline. .NET SDK was 10.0.303, Node v24.13.0, and Python
3.13.14 on native Windows. Grading remains native-local, outside an OS sandbox.
The flat Release CLI and Host from the preceding verified build.bat were used.
Host execution authority runs in process; there is no Broker executable. The
checkout is 1f383ab0; binaries were built from the changes before that commit,
so the Host's embedded version still names base revision 4d925e4d. The manifest
retains executable and eleven assembly SHA-256 fingerprints. A separate Host
used a free loopback port and fresh TEMP data/log roots; only non-secret provider
settings and catalog data were copied in. Credentials were read through the
existing Windows Credential Manager. The user's normal data root was not
migrated or modified by this run.
Artifacts are under ignored artifacts/ru-live-post-refactor-20261010:
manifest.json, report.json, analysis.json, public tool-payload analysis,
terminal-state evidence, per-task trajectories, verifier results and patches.
The local driver artifacts/run-live-20261010.py invokes the existing
benchmark.run_live one task at a time and checkpoints the aggregate report.
Two setup probes failed before any scored run (legacy settings had no SQLite
business table; a doubled URL separator prevented readiness). Both were fixed
in this local driver. No product or benchmark-runner source changed.
Wall time was 41.00 minutes, summed agent time 40.77 minutes. Strict
reward was 0/9, Terminal 0/6 and SWE 0/3. Three runs reported
completed, but all three left source unchanged and failed the verifier. Six
stopped for approval and were cancelled; persisted final states confirm all
nine runs are terminal (three completed, six cancelled). Only one candidate
changed source, and none passed every verifier group. Partial passes on
unchanged starters are not improvements. All event trajectories caught up to
the authoritative cursor. Token totals and actual billing remain unknown.
| Task | Agent minutes | Stop | Candidate groups passed | Source changed |
|---|---|---|---|---|
cs-atomic-ledger |
0.17 | External directory approval (list /) |
0/7 | No |
cs-single-flight-cache |
6.66 | Completed without patch | 5/8 | No |
py-dependency-scheduler |
0.22 | External directory approval (/workspace/...) |
3/6 | No |
py-stream-parser |
4.58 | Completed without patch | 2/6 | No |
ts-generation-pager |
14.76 | Unknown executable approval (node) |
3/6 | src/pager.ts |
ts-transaction-store |
7.97 | Completed without patch | 4/7 | No |
cs-lease-renewal |
2.52 | Unknown executable approval (cat) |
2/7 | No |
py-migration-rollback |
2.79 | Terminal external-mutation approval | 2/6 | No |
ts-session-replay |
1.10 | Unknown executable approval | 2/7 | No |
Observed failures include invented Linux paths, attempts to read directories
as files, create-only writes to existing files (file_exists), mismatched edit
text (edit_no_match), and missing required tool arguments (malformed_call).
Pager eventually used a valid overwrite, but its patch still failed shared
promise identity, stale-generation isolation and refresh behavior. Its terminal
call redundantly put node in the argument list as well as the executable.
Cache and parser returned long public final text (33,203 and 32,715 characters)
without edits. Their exact provider finish reasons were not retained in the
public final-text payload, so output truncation is a hypothesis, not an isolated
cause. Source inspection separately shows the Coordinator routes a non-empty
text-only step to final handling without distinguishing Length; this warrants
a focused regression before changing completion behavior.
These results identify model/tool-use, unattended approval and completion
problems under this selection. They do not prove a regression from Step 5:
that pilot used another provider/model and older runtime. One attempt per public
custom task cannot establish general coding quality or parity with other agents.
Crash-orphan sandbox cleanup and interactive native GUI/TUI acceptance remain
open. Both the trial Host and evidence-replay Host shut down cleanly.
Automatic approval review rejected deletion of the three owned trial TEMP
directories (blocked by policy); those data/log directories remain locally.
The documentation/count guards passed 19 tests and git diff --check
passed after this evidence update. The full .NET/Python matrix and Release
build were not repeated for this documentation-only change; their preceding
results remain recorded below.
Sandbox lifecycle and client simplification (2026-10-10)¶
Session close, directory-grant revocation and async Host shutdown cancel and drain terminal pumps before sandbox release. Closing a foreign run leaves its lease and execution untouched. Disposed authority/runtime instances reject new work. Recreated sandbox entries use fresh profile names/SIDs; failed extensions roll back new grants, and failed creation removes its temporary directory. Crash orphan profiles/ACEs still need eventual cleanup, but are not revived by a new entry. A failed drain remains blocked and can be retried.
WinUI and CLI share the event-feed reconnect loop and durable cursor handling.
Callback failures propagate instead of being retried as transport failures.
CLI parking retains durable text recovery and leaves the server run available
for Continue. Preset and durable provider calling-mode conversions use shared
codecs. Managed Git preserves sandbox start failures through its readers;
workspace discovery uses the current checkout marker instead of retired src.
The final full invocation passed all 5,837 tests:
| Project | Passed |
|---|---|
| Broker | 928 |
| CLI | 1,279 |
| Client | 211 |
| Harness Core | 468 |
| Harness Persistence | 176 |
| Hosting | 39 |
| Protocol | 86 |
| Runtime Adapters | 195 |
| Runtime Worker | 69 |
| Session Interchange | 43 |
| Backend | 1,040 |
| GUI | 1,303 |
| Total | 5,837 |
artifacts/simplification-full-dotnet.log retains the first invocation:
Broker 927 passed, then CLI 1 failed / 1,278 passed because cancellation
ended the newly shared subscription before parked-run payload recovery.
The fix preserves the existing assertion; seven focused park/resume checks and
artifacts/simplification-full-dotnet-final.log passed. Focused RED experiments
and corrected fixture failures remain in artifacts/simplification-*.log.
The final GUI compile has neither the prior nullable highlighter warning nor
the filtered Assert.Single analyzer warning.
The first Python run was 1 failed / 210 passed / 1 skipped: a partial docs
update changed the README count before the verification table, because the
update script used the Windows default text encoding. The UTF-8 update now
keeps both counts aligned; artifacts/simplification-pytest-final.log records
211 passed / 1 skipped, including route, tool, docs and benchmark guards.
Commands:
.\scripts\test_dotnet.ps1 -Configuration Debug
.venv\Scripts\python.exe -m pytest tests -q
.\build.bat
.\scripts\publish_smoke.ps1
Release packaging passed 11/11 stages. Both the standalone publish smoke
and the flat distributable smoke passed authenticated /status and clean
/host/shutdown, using isolated temporary data/log roots and synthetic bearer
authority. The flat smoke used a retained local copy of the canonical smoke
script with publishing replaced by the existing flat output; it is validation
evidence, not an additional production launch path. Logs:
artifacts/simplification-release-build.log,
artifacts/simplification-publish-smoke.log,
artifacts/simplification-flat-smoke.log.
Real-provider task quality and interactive native UI acceptance are separate and were not exercised by the deterministic matrices.