Verification evidence - 2026-10-29¶
Historical snapshot only. These results describe the recorded binaries and conditions, not current acceptance. See current verification.
Coding benchmark stopped by provider rate limiting¶
Checkout 2675ab68b1678e2a88f6df515a8900bb91da1782; separately published Release
Host and CLI. The user's saved selection was snapshotted and verified on a
dedicated Host: OpenRouter nvidia/nemotron-3-ultra-550b-a55b:free, effort high,
protocol chat_completions, standard permissions and coding_v2 tools.
The experiment used a separate data root and temporary workspaces outside the
Ru checkout. No GUI interaction or provider/model substitution occurred.
The plan was one attempt on each of the 20 coding tasks, followed by two more attempts on the six original Everyday tasks: 32 attempts total. Offline oracle checks passed for all 20 tasks: faulty starters failed and references passed.
The first six Everyday attempts completed and passed external grading:
| Task | Wall seconds | Result |
|---|---|---|
py-config-isolation |
128.185 | Passed |
cs-search-pagination |
84.483 | Passed |
cs-todo-summary |
47.356 | Passed |
py-stable-dedupe |
40.332 | Passed |
ts-emitter-lifecycle |
88.253 | Passed |
ts-query-builder |
27.307 | Passed |
Their mean was approximately 69.3 seconds. This is a single attempt per task, not evidence of repeated-attempt reliability.
Four subsequent runs entered waiting_user with the durable reason
continue.requested: provider_rate_limited. The batch-reconcile workspace also
retained test_implementation.py, so patch admission rejected it before running
test groups. A separate source-only patch replay passed its 10 groups; that
diagnostic does not change the original run's zero acceptance reward.
The batch was stopped: 10 result records, one additional attempt explicitly cancelled, and 21 attempts not started. The dedicated Host was shut down and its database showed zero active benchmark runs. Terminal/SWE and the repeat rounds remain unmeasured. This is an incomplete cohort, not a full benchmark score.
The older Everyday baseline used Kilo Step 5 Preview Free and resolved 16/18.
Four original fixture fingerprints match; py-config-isolation and
py-stable-dedupe differ. Model/provider changes and differing sample counts
prevent attributing a difference solely to backend upgrades.
Local evidence: artifacts/coding-benchmark-20261029/manifest.json, oracle.json,
report.json, summary.json, RESULT.md, per-attempt trajectories and patches,
diagnostics/batch-source-only.json, stop.json and cleanup.json. Provider-only
latency, malformed-call taxonomy and cost remain unknown where the RPC evidence
does not expose them. Observed event counts retain their completeness flags.
Kilo continuation: completed coding benchmark¶
After the user changed provider, the continuation pinned models.dev:kilo,
nvidia/nemotron-3-ultra-550b-a55b:free, effort high, coding_v2 and
standard. It reused the exact published binaries and 20 task fingerprints.
Twenty-six fresh attempts completed: four prior provider-blocked attempts,
one prior cancelled attempt and 21 previously unstarted attempts. Six earlier
OpenRouter successes remain a separate cohort. No desktop control or model
substitution was used. The path was CLI stdio RPC to a dedicated Public Host
and Broker, not native GUI acceptance.
Raw acceptance requires external grading success and unattended run completion:
| Suite | Accepted / attempts | Candidate passed grader | Agent completed | Mean wall seconds |
|---|---|---|---|---|
| Everyday | 12/17 | 14/17 | 15/17 | 131.73 |
| Terminal | 0/6 | 0/6 | 5/6 | 562.74 |
| SWE | 0/3 | 1/3 | 0/3 | 534.49 |
| Total | 12/26 | 15/26 | 20/26 | 277.67 |
Median wall time was 180.69 seconds; the slowest trial took 1160.75 seconds. Paired durable model-step intervals cover 97.03% of aggregate agent wall time, with no unmatched model/tool IDs. They include prompt preparation, provider, retries and finalization, so they are not a provider-only latency metric. Billing cost remains unknown.
Batch reconciliation, checkpoint journal and migration rollback passed external grading but failed run completion. Activity traces and source inspection show a verification lifecycle trap: deleting a temporary verification script invalidates its retained check, and later passing smoke checks do not supersede that stale command identity. Migration explicitly used purpose=verification on the later passing checks. This needs a runtime fix preserving freshness guards; no runtime change was made during measurement. Other failures include real concurrency, cancellation, ordering, capacity, parsing and event-validation bugs.
Offline audits on copied fixtures (unchanged candidate patches, failing seeds and passing references) found three grader defects. Atomic Ledger rejects a valid non-castable immutable snapshot; correcting that permits its original groups, but a separate repeated-transfer contract check still fails the candidate. Scheduler records zero checks when a case throws; correcting its counter yields genuine failed groups rather than invalid evidence. Todo tries to mutate a read-only snapshot: correcting that leaves attempt 2 failing ordinal ordering and makes attempt 3 pass. Raw scores remain 12/26; separately, diagnostic acceptance with these grader corrections is 13/26.
Together with the six prior OpenRouter successes, all 32 planned slots have observations, with 18 raw successes. This mixed-provider coverage is not a single-provider controlled score. Different historical models, changed fixtures and sample composition prevent attributing differences solely to backend work. These are repository custom tasks, not official SWE-bench or Terminal-Bench scores.
Evidence: artifacts/coding-benchmark-20261029-kilo/RESULT.md, manifest.json,
report.json, summary.json, timing.json, per-attempt patches/trajectories,
diagnostics/verification-freshness-findings.md, and the three
diagnostics/*-grader-audit/summary.json reports. The isolated database has zero
active runs; runtime binaries and all fixture hashes still match the manifest.
The benchmark Host/CLI processes have exited. No product code was changed.
Repairs following the completed benchmark¶
The scoped working-tree implementation adds explicit supersedes metadata to
terminal verification calls. It satisfies only a retired previously passing
stale check after an appropriate replacement passes fresh in the same canonical
cwd. It cannot retire failed, pending, negative or unknown-freshness checks.
Original evidence is retained, and later changes reopen the obligation. Core
regressions and a SQLite warm-cache/cold-replay/compaction test cover this lifecycle.
Python repair-family evidence also recognizes complete empty pytest discovery (exit 5 plus the no-tests diagnostic), allowing a meaningful passing correction in the same resolved Python interpreter/cwd. Direct inline parse errors remain repairable; actual test failures, collection errors, truncated output and negative-test mismatches do not gain this exception. File-tool guidance now explicitly distinguishes creation, partial editing and deliberate replacement; long Python code is directed to file/argv execution rather than malformed strings.
Todo, Atomic Ledger and Scheduler graders incorporate the three audited repairs. The ledger now additionally tests valid repeated transfer pairs. Behavioral tests accept valid alternatives and reject mutable snapshots/rejected repeated pairs. All three revised fixtures still fail their starter and pass the reference. Historical cohort scores and fingerprints were not rewritten.
Verification commands and evidence:
scripts/test_dotnet.ps1 -Configuration Debug: first run stopped after a CLI wordmark allocation assertion (expected 0, observed 5144 bytes). The individual test passed unchanged in isolation. A complete rerun passed all 13 projects, 9,094 tests, no skips. Both the first failure and rerun logs are retained.- A final Harness run after canonical-cwd alias coverage passed 778 Core and 370 Persistence tests (the full-matrix Core count was 777 before that extra case).
- Focused Python benchmark/process/RPC/cohort/evaluation, tool metrics, document and source-encoding checks: 78 passed.
- Three revised grader oracle self-tests: starter failed, reference passed.
- Release Host and CLI publish succeeded in a dedicated artifact directory; no installed application files were replaced.
Evidence: artifacts/benchmark-fixes-dotnet.log,
artifacts/benchmark-fixes-dotnet-rerun.log,
artifacts/benchmark-fixes-harness-final.log,
artifacts/benchmark-fixes-python.log, and
artifacts/benchmark-fixes-20261029/corrected-graders-oracle.json.
Targeted real-provider follow-up¶
The dedicated Release cohort pinned Kilo, Nemotron Ultra Free, effort high, standard permissions and coding_v2. Three fresh workspaces used unchanged Batch/Checkpoint/Migration fixture fingerprints, with failing seeds and passing references. All three candidates passed the external grader and their runs completed unattended:
| Task | Agent wall seconds | Model steps | Tool calls | Acceptance |
|---|---|---|---|---|
| py-batch-reconcile | 375.080 | 17 | 19 | passed |
| py-checkpoint-journal | 317.823 | 14 | 15 | passed |
| py-migration-rollback | 455.507 | 22 | 24 | passed |
This is a three-case follow-up, not a repeat of the complete 26-run benchmark. It does not establish a general coding-quality or latency improvement. Across 58 tool completions, 51 were ok; two file_exists, two invalid_arguments and three malformed_call receipts remained failures and were subsequently recovered by the model. They were not silently coerced into successful tool executions. The audited Batch call intents omitted purpose=verification, so its success does not directly prove the explicit supersession mechanism was used. Those semantics have separate deterministic and SQLite reconstruction coverage.
After the frozen live cohort, final bookkeeping hardening added stable check IDs, unambiguous provider-ID aliases and read-only replacement bindings. It cannot retire multiple checks by guessing a reused provider call ID. This final boundary was tested separately: 780 Core tests, 370 Persistence tests, 200 Protocol tests, and two Broker runtime cases covering valid replacement execution and invalid metadata rejection all passed. Latest Host and CLI Release builds also passed. The model cohort was not rerun after this bounded hardening; its original manifest, binaries and source hash evidence remain unchanged.
Evidence: artifacts/benchmark-fixes-20261029/RESULT.md, report.json,
manifest.json, cleanup.json, per-task results/patches/trajectories,
batch-call-intents.json, artifacts/benchmark-fixes-harness-hardened.log,
artifacts/benchmark-fixes-broker-positive.log,
artifacts/benchmark-fixes-protocol-final.log and
artifacts/benchmark-fixes-release-final.log.
The isolated database has zero active runs and no dedicated benchmark Host/CLI
process remains. This exercised CLI RPC to Host/Broker, not native GUI acceptance.
Grader corrections: coalescer and pager loader timing¶
The ts-abortable-coalescer groups shared_inflight and clear_generation, and
the ts-generation-pager group same-generation-dedup, asserted calls === 0
right after run() or loadNext(). That required the loader not to run
synchronously. The task text only requires concurrent calls to share one loader
call and does not say when the loader runs, so a synchronous implementation failed
on a requirement the task does not state. The pager assertion also bundled this
timing claim with one === two, which the task does require.
The pre-settlement calls === 0 assertions were removed. The spec claims stay:
one === two in the pager group and the one-loader count checks. The two dedup
groups now also assert exactly one loader call after the shared request settles
(shared_inflight, and same-generation-dedup after Promise.all).
clear_generation checks generation separation rather than sharing, so it gained
no dedup assertion. Group names, group counts and other expectations are unchanged.
The retained one-microtask checks (calls === 1 after await Promise.resolve())
still assume the loader starts within one microtask. An implementation that defers
the loader by two or more ticks would fail them. The task does not require that
timing, so this is a known limit of those groups.
Oracle results. The starter and reference were graded by the repository self-test. The variants differ from the reference only in loader timing or dedup, and were graded in a scratch directory outside the checkout.
| Task | Starter | Reference | Synchronous loader | Extra loader call for a shared request |
|---|---|---|---|---|
ts-abortable-coalescer |
Fails all 5 fail-to-pass groups; passes the 3 pass-to-pass groups | Passes 8/8 | Passes 8/8 | Fails shared_inflight (and abort_isolated) |
ts-generation-pager |
Fails 4 groups | Passes 6/6 | Passes 6/6 | Fails same-generation-dedup on its loader-count checks |
Under the previous grader, the synchronous variant failed shared_inflight and
clear_generation for the coalescer and same-generation-dedup for the pager,
while the reference still passed. This is the over-strict behavior removed here.
Historical candidate diagnostics. The historical patches were applied to scratch copies and graded with the corrected graders. The artifacts were not modified.
ts-abortable-coalescer-1:shared_inflightnow passes, since the candidate shares one loader call.clear_generationstill fails: an older settlement deletes the newer generation's entry, sosize()is 0 instead of 1.retry_after_rejectionfails withTypeError, because the candidate calls.thenon a plain loader value. The other five groups pass.ts-generation-pager-1:same-generation-dedupfails only onone === two, because itsasync loadNext()returns a new promise for each call. Its loader-count checks pass, and the other five groups pass.
Both candidates still fail under the corrected graders, so their historical outcomes are unchanged. Under the previous graders both also failed on the timing assertion.
Historical cohort scores, reports, trajectories, patches and fixture records are unchanged. The 2026-10-29 cohorts used the previous fingerprints, so these two tasks are no longer matched against those cohorts. Earlier results for these tasks were measured under the previous graders. The 2026-10-20 observation that the pager grader required a synchronous-call restriction no longer applies after this correction.
Fingerprints changed as follows:
ts-abortable-coalescer:cf481b7373597ec45cc93dfe8517e9dde4a2d9b43bdfb3515b420bd339da0d29to671811635c338d411fe1e0531993d39efa9cb2cc08106df20855483b52df6718.ts-generation-pager:d880cb834307b49141b67aca25b881f30b7701799663551128d1d7ccb865fc4dtoe8215ce76475429a7174d264f9f1aaf792fecf20a7ac685d729c4cfe8288b3ac.
No test or tool pins either value, so none was updated. The
provenance paragraph in tools/coding-benchmark/README.md named only the
three earlier changed fixtures when this correction was recorded. That README was
outside this correction's file scope; it was later updated to list all five
changed fixtures.
Verification: both tasks validated through benchmark.py self-test (starter
failed, reference passed). The benchmark Python tests ran with five C#-graded
tests deselected because they invoke dotnet, which was not run for this change.
The documentation guard tests/test_current_documentation.py passed.