Skip to content

Verification evidence — 2026-11-17

Historical snapshot only. Counts, commands and limitations describe that snapshot and do not establish current acceptance. Use current verification.

Live coding cohorts on hy4-flash-free (2026-11-17)

The owner asked for a live rerun of the coding benchmark on the current checkout, first with the October selection and then with models.dev:opencode-go / hy4-flash-free. Every scored trial below used that selection, standard, coding_v3, fresh trusted workspaces and sessions, the tasks' own agent and verifier deadlines, and no approval, Continue or task intervention. .NET SDK 10.0.303, Node v24.13.0 and Python 3.13.14 on native Windows; grading is native-local, outside an OS sandbox.

Setup. build.bat produced the flat Release CLI and Host twice: first from 79068c35 (first nine-task cohort), then from the tree committed as 004b357b (baseline, variant, advisor and Everyday cohorts; the baseline ran before the prompt change and its binaries were the first build). Each trial set used a separate Host with fresh TEMP data/log roots. Only models-dev.json and a non-secret settings.json were copied in; keys were read from Windows Credential Manager. The user's data root was not modified. Artifacts are under ignored artifacts/ru-live-20261117.

Setup failures, not scored. The October selection (nvidia/nemotron-3.5-lightning:free on OpenRouter) failed every run within seconds with provider_catalog_unauthorized: the owner had removed that key. The next attempt failed the same way because the copied settings.json was written by PowerShell 5.1 with a UTF-8 BOM, which the Host's legacy settings import skips, leaving the default OpenAI backend. Writing the file without a BOM fixed it; the imported selection was checked in the trial database before the scored runs. One variant py-stream-parser attempt failed with provider_catalog_timeout after 17 seconds and no model step; it is excluded.

Terminal and SWE, one attempt each: 6/9

Task Reward Groups Minutes Steps / tool calls
cs-atomic-ledger 1 7/7 7.0 32 / 30
cs-single-flight-cache 1 8/8 3.8 13 / 17
py-dependency-scheduler 1 6/6 5.3 23 / 30
py-stream-parser 1 6/6 2.7 10 / 15
ts-generation-pager 1 6/6 4.9 18 / 22
ts-transaction-store 0 6/7 5.2 19 / 23
cs-lease-renewal (SWE) 0 6/7 3.8 15 / 21
py-migration-rollback (SWE) 1 6/6 2.0 12 / 16
ts-session-replay (SWE) 0 6/7 2.0 7 / 15

All nine runs completed with source changes, no approval stop and no compaction; every event trajectory caught up to the terminal cursor. About 37 agent minutes and 3.5M tokens in total. Each miss is one group whose rule the task text states: snapshot values not frozen (ts-transaction-store), an expired lease still accepting old mutations (cs-lease-renewal), and a delta event without text accepted instead of rejected (ts-session-replay). The October 10 record (0/9) used another model and is not a controlled comparison.

Tool-call evidence and the A/B check

Trajectories now record each call's tool id, declared purpose, executable base name and outcome (006ce13b). A baseline of the three failing tasks and two controls, three attempts each, gave: failing tasks 1/9 (cs-lease-renewal 0/3, ts-session-replay 0/3, ts-transaction-store 1/3), controls 6/6. Every one of the 15 runs ran 2–6 purpose=verification checks and its last check passed: the checks ran but did not exercise the rule that was missed. In the cs-lease-renewal runs a declared check failed earlier, so the existing completion_review note fired, and the model still confirmed its work.

The prompt change in 004b357b (the final check must exercise every stated rule) gave failing tasks 2/9 (cs-lease-renewal 1/3) and controls 5/5 valid. The difference is within noise; check counts per run did not change materially, and the misses stayed on the same groups. The owner kept the guidance.

Advisor, one attempt each: 1/3

With the owner's advisor pair (deepseek-v4.1-flash on OpenCode Go) pinned, each failing task drew one advisor_recommended hold and one successful advisor call. ts-transaction-store passed; cs-lease-renewal and ts-session-replay failed on the same groups as before. Runs took longer (cs-lease-renewal 9.0 minutes). One attempt per task cannot attribute the pass to the advisor; it does show the advisor did not catch the other two misses.

Everyday, one attempt each: 11/11

All eleven Everyday tasks passed in about 18 agent minutes (5–19 model steps, 1.8M tokens). Every run declared 1–4 verification checks; three recovered from a failed check after completion_review. Edits stayed in src/ plus build output and one task-requested file. Non-ok tool outcomes were write_file file_exists (3), process_exit_nonzero (3) and executable_not_found (1). Across all of today's cohorts file_exists recurred: the model used the create-only write_file on an existing file and recovered on the next step.

Reading

The harness was stable on this selection: no approval stops, compactions, repetition guards or incomplete trajectories in any scored run. The remaining misses are stated edge rules that the same model, a prompt rule, the existing completion review and a second-model advisor did not catch, consistent with the ts-transaction-store snapshot misses recorded on 2026-10-20. Single and three-attempt cohorts show large effects only; they do not establish rates.