Verification evidence — 2026-11-17¶
Historical snapshot only. Counts, commands and limitations describe that snapshot and do not establish current acceptance. Use current verification.
Live coding cohorts on hy4-flash-free (2026-11-17)¶
The owner asked for a live rerun of the coding benchmark on the current checkout,
first with the October selection and then with models.dev:opencode-go /
hy4-flash-free. Every scored trial below used that selection, standard,
coding_v3, fresh trusted workspaces and sessions, the tasks' own agent and
verifier deadlines, and no approval, Continue or task intervention. .NET SDK
10.0.303, Node v24.13.0 and Python 3.13.14 on native Windows; grading is
native-local, outside an OS sandbox.
Setup. build.bat produced the flat Release CLI and Host twice: first from
79068c35 (first nine-task cohort), then from the tree committed as 004b357b
(baseline, variant, advisor and Everyday cohorts; the baseline ran before the
prompt change and its binaries were the first build). Each trial set used a
separate Host with fresh TEMP data/log roots. Only models-dev.json and a
non-secret settings.json were copied in; keys were read from Windows Credential
Manager. The user's data root was not modified. Artifacts are under ignored
artifacts/ru-live-20261117.
Setup failures, not scored. The October selection
(nvidia/nemotron-3.5-lightning:free on OpenRouter) failed every run within
seconds with provider_catalog_unauthorized: the owner had removed that key.
The next attempt failed the same way because the copied settings.json was
written by PowerShell 5.1 with a UTF-8 BOM, which the Host's legacy settings
import skips, leaving the default OpenAI backend. Writing the file without a BOM
fixed it; the imported selection was checked in the trial database before the
scored runs. One variant py-stream-parser attempt failed with
provider_catalog_timeout after 17 seconds and no model step; it is excluded.
Terminal and SWE, one attempt each: 6/9¶
| Task | Reward | Groups | Minutes | Steps / tool calls |
|---|---|---|---|---|
cs-atomic-ledger |
1 | 7/7 | 7.0 | 32 / 30 |
cs-single-flight-cache |
1 | 8/8 | 3.8 | 13 / 17 |
py-dependency-scheduler |
1 | 6/6 | 5.3 | 23 / 30 |
py-stream-parser |
1 | 6/6 | 2.7 | 10 / 15 |
ts-generation-pager |
1 | 6/6 | 4.9 | 18 / 22 |
ts-transaction-store |
0 | 6/7 | 5.2 | 19 / 23 |
cs-lease-renewal (SWE) |
0 | 6/7 | 3.8 | 15 / 21 |
py-migration-rollback (SWE) |
1 | 6/6 | 2.0 | 12 / 16 |
ts-session-replay (SWE) |
0 | 6/7 | 2.0 | 7 / 15 |
All nine runs completed with source changes, no approval stop and no
compaction; every event trajectory caught up to the terminal cursor. About 37
agent minutes and 3.5M tokens in total. Each miss is one group whose rule the
task text states: snapshot values not frozen (ts-transaction-store), an
expired lease still accepting old mutations (cs-lease-renewal), and a delta
event without text accepted instead of rejected (ts-session-replay). The
October 10 record (0/9) used another model and is not a controlled comparison.
Tool-call evidence and the A/B check¶
Trajectories now record each call's tool id, declared purpose, executable base
name and outcome (006ce13b). A baseline of the three failing tasks and two
controls, three attempts each, gave: failing tasks 1/9 (cs-lease-renewal
0/3, ts-session-replay 0/3, ts-transaction-store 1/3), controls 6/6.
Every one of the 15 runs ran 2–6 purpose=verification checks and its last
check passed: the checks ran but did not exercise the rule that was missed. In
the cs-lease-renewal runs a declared check failed earlier, so the existing
completion_review note fired, and the model still confirmed its work.
The prompt change in 004b357b (the final check must exercise every stated
rule) gave failing tasks 2/9 (cs-lease-renewal 1/3) and controls 5/5
valid. The difference is within noise; check counts per run did not change
materially, and the misses stayed on the same groups. The owner kept the
guidance.
Advisor, one attempt each: 1/3¶
With the owner's advisor pair (deepseek-v4.1-flash on OpenCode Go) pinned,
each failing task drew one advisor_recommended hold and one successful
advisor call. ts-transaction-store passed; cs-lease-renewal and
ts-session-replay failed on the same groups as before. Runs took longer
(cs-lease-renewal 9.0 minutes). One attempt per task cannot attribute the pass
to the advisor; it does show the advisor did not catch the other two misses.
Everyday, one attempt each: 11/11¶
All eleven Everyday tasks passed in about 18 agent minutes (5–19 model steps,
1.8M tokens). Every run declared 1–4 verification checks; three recovered from a
failed check after completion_review. Edits stayed in src/ plus build output
and one task-requested file. Non-ok tool outcomes were write_file
file_exists (3), process_exit_nonzero (3) and executable_not_found (1).
Across all of today's cohorts file_exists recurred: the model used the
create-only write_file on an existing file and recovered on the next step.
Reading¶
The harness was stable on this selection: no approval stops, compactions,
repetition guards or incomplete trajectories in any scored run. The remaining
misses are stated edge rules that the same model, a prompt rule, the existing
completion review and a second-model advisor did not catch, consistent with the
ts-transaction-store snapshot misses recorded on 2026-10-20. Single and
three-attempt cohorts show large effects only; they do not establish rates.