Skip to content

Verification evidence - 2026-10-29

Historical snapshot only. These results describe the recorded binaries and conditions, not current acceptance. See current verification.

Coding benchmark stopped by provider rate limiting

Checkout 2675ab68b1678e2a88f6df515a8900bb91da1782; separately published Release Host and CLI. The user's saved selection was snapshotted and verified on a dedicated Host: OpenRouter nvidia/nemotron-3-ultra-550b-a55b:free, effort high, protocol chat_completions, standard permissions and coding_v2 tools. The experiment used a separate data root and temporary workspaces outside the Ru checkout. No GUI interaction or provider/model substitution occurred.

The plan was one attempt on each of the 20 coding tasks, followed by two more attempts on the six original Everyday tasks: 32 attempts total. Offline oracle checks passed for all 20 tasks: faulty starters failed and references passed.

The first six Everyday attempts completed and passed external grading:

Task Wall seconds Result
py-config-isolation 128.185 Passed
cs-search-pagination 84.483 Passed
cs-todo-summary 47.356 Passed
py-stable-dedupe 40.332 Passed
ts-emitter-lifecycle 88.253 Passed
ts-query-builder 27.307 Passed

Their mean was approximately 69.3 seconds. This is a single attempt per task, not evidence of repeated-attempt reliability.

Four subsequent runs entered waiting_user with the durable reason continue.requested: provider_rate_limited. The batch-reconcile workspace also retained test_implementation.py, so patch admission rejected it before running test groups. A separate source-only patch replay passed its 10 groups; that diagnostic does not change the original run's zero acceptance reward.

The batch was stopped: 10 result records, one additional attempt explicitly cancelled, and 21 attempts not started. The dedicated Host was shut down and its database showed zero active benchmark runs. Terminal/SWE and the repeat rounds remain unmeasured. This is an incomplete cohort, not a full benchmark score.

The older Everyday baseline used Kilo Step 5 Preview Free and resolved 16/18. Four original fixture fingerprints match; py-config-isolation and py-stable-dedupe differ. Model/provider changes and differing sample counts prevent attributing a difference solely to backend upgrades.

Local evidence: artifacts/coding-benchmark-20261029/manifest.json, oracle.json, report.json, summary.json, RESULT.md, per-attempt trajectories and patches, diagnostics/batch-source-only.json, stop.json and cleanup.json. Provider-only latency, malformed-call taxonomy and cost remain unknown where the RPC evidence does not expose them. Observed event counts retain their completeness flags.

Kilo continuation: completed coding benchmark

After the user changed provider, the continuation pinned models.dev:kilo, nvidia/nemotron-3-ultra-550b-a55b:free, effort high, coding_v2 and standard. It reused the exact published binaries and 20 task fingerprints. Twenty-six fresh attempts completed: four prior provider-blocked attempts, one prior cancelled attempt and 21 previously unstarted attempts. Six earlier OpenRouter successes remain a separate cohort. No desktop control or model substitution was used. The path was CLI stdio RPC to a dedicated Public Host and Broker, not native GUI acceptance.

Raw acceptance requires external grading success and unattended run completion:

Suite Accepted / attempts Candidate passed grader Agent completed Mean wall seconds
Everyday 12/17 14/17 15/17 131.73
Terminal 0/6 0/6 5/6 562.74
SWE 0/3 1/3 0/3 534.49
Total 12/26 15/26 20/26 277.67

Median wall time was 180.69 seconds; the slowest trial took 1160.75 seconds. Paired durable model-step intervals cover 97.03% of aggregate agent wall time, with no unmatched model/tool IDs. They include prompt preparation, provider, retries and finalization, so they are not a provider-only latency metric. Billing cost remains unknown.

Batch reconciliation, checkpoint journal and migration rollback passed external grading but failed run completion. Activity traces and source inspection show a verification lifecycle trap: deleting a temporary verification script invalidates its retained check, and later passing smoke checks do not supersede that stale command identity. Migration explicitly used purpose=verification on the later passing checks. This needs a runtime fix preserving freshness guards; no runtime change was made during measurement. Other failures include real concurrency, cancellation, ordering, capacity, parsing and event-validation bugs.

Offline audits on copied fixtures (unchanged candidate patches, failing seeds and passing references) found three grader defects. Atomic Ledger rejects a valid non-castable immutable snapshot; correcting that permits its original groups, but a separate repeated-transfer contract check still fails the candidate. Scheduler records zero checks when a case throws; correcting its counter yields genuine failed groups rather than invalid evidence. Todo tries to mutate a read-only snapshot: correcting that leaves attempt 2 failing ordinal ordering and makes attempt 3 pass. Raw scores remain 12/26; separately, diagnostic acceptance with these grader corrections is 13/26.

Together with the six prior OpenRouter successes, all 32 planned slots have observations, with 18 raw successes. This mixed-provider coverage is not a single-provider controlled score. Different historical models, changed fixtures and sample composition prevent attributing differences solely to backend work. These are repository custom tasks, not official SWE-bench or Terminal-Bench scores.

Evidence: artifacts/coding-benchmark-20261029-kilo/RESULT.md, manifest.json, report.json, summary.json, timing.json, per-attempt patches/trajectories, diagnostics/verification-freshness-findings.md, and the three diagnostics/*-grader-audit/summary.json reports. The isolated database has zero active runs; runtime binaries and all fixture hashes still match the manifest. The benchmark Host/CLI processes have exited. No product code was changed.

Repairs following the completed benchmark

The scoped working-tree implementation adds explicit supersedes metadata to terminal verification calls. It satisfies only a retired previously passing stale check after an appropriate replacement passes fresh in the same canonical cwd. It cannot retire failed, pending, negative or unknown-freshness checks. Original evidence is retained, and later changes reopen the obligation. Core regressions and a SQLite warm-cache/cold-replay/compaction test cover this lifecycle.

Python repair-family evidence also recognizes complete empty pytest discovery (exit 5 plus the no-tests diagnostic), allowing a meaningful passing correction in the same resolved Python interpreter/cwd. Direct inline parse errors remain repairable; actual test failures, collection errors, truncated output and negative-test mismatches do not gain this exception. File-tool guidance now explicitly distinguishes creation, partial editing and deliberate replacement; long Python code is directed to file/argv execution rather than malformed strings.

Todo, Atomic Ledger and Scheduler graders incorporate the three audited repairs. The ledger now additionally tests valid repeated transfer pairs. Behavioral tests accept valid alternatives and reject mutable snapshots/rejected repeated pairs. All three revised fixtures still fail their starter and pass the reference. Historical cohort scores and fingerprints were not rewritten.

Verification commands and evidence:

  • scripts/test_dotnet.ps1 -Configuration Debug: first run stopped after a CLI wordmark allocation assertion (expected 0, observed 5144 bytes). The individual test passed unchanged in isolation. A complete rerun passed all 13 projects, 9,094 tests, no skips. Both the first failure and rerun logs are retained.
  • A final Harness run after canonical-cwd alias coverage passed 778 Core and 370 Persistence tests (the full-matrix Core count was 777 before that extra case).
  • Focused Python benchmark/process/RPC/cohort/evaluation, tool metrics, document and source-encoding checks: 78 passed.
  • Three revised grader oracle self-tests: starter failed, reference passed.
  • Release Host and CLI publish succeeded in a dedicated artifact directory; no installed application files were replaced.

Evidence: artifacts/benchmark-fixes-dotnet.log, artifacts/benchmark-fixes-dotnet-rerun.log, artifacts/benchmark-fixes-harness-final.log, artifacts/benchmark-fixes-python.log, and artifacts/benchmark-fixes-20261029/corrected-graders-oracle.json.

Targeted real-provider follow-up

The dedicated Release cohort pinned Kilo, Nemotron Ultra Free, effort high, standard permissions and coding_v2. Three fresh workspaces used unchanged Batch/Checkpoint/Migration fixture fingerprints, with failing seeds and passing references. All three candidates passed the external grader and their runs completed unattended:

Task Agent wall seconds Model steps Tool calls Acceptance
py-batch-reconcile 375.080 17 19 passed
py-checkpoint-journal 317.823 14 15 passed
py-migration-rollback 455.507 22 24 passed

This is a three-case follow-up, not a repeat of the complete 26-run benchmark. It does not establish a general coding-quality or latency improvement. Across 58 tool completions, 51 were ok; two file_exists, two invalid_arguments and three malformed_call receipts remained failures and were subsequently recovered by the model. They were not silently coerced into successful tool executions. The audited Batch call intents omitted purpose=verification, so its success does not directly prove the explicit supersession mechanism was used. Those semantics have separate deterministic and SQLite reconstruction coverage.

After the frozen live cohort, final bookkeeping hardening added stable check IDs, unambiguous provider-ID aliases and read-only replacement bindings. It cannot retire multiple checks by guessing a reused provider call ID. This final boundary was tested separately: 780 Core tests, 370 Persistence tests, 200 Protocol tests, and two Broker runtime cases covering valid replacement execution and invalid metadata rejection all passed. Latest Host and CLI Release builds also passed. The model cohort was not rerun after this bounded hardening; its original manifest, binaries and source hash evidence remain unchanged.

Evidence: artifacts/benchmark-fixes-20261029/RESULT.md, report.json, manifest.json, cleanup.json, per-task results/patches/trajectories, batch-call-intents.json, artifacts/benchmark-fixes-harness-hardened.log, artifacts/benchmark-fixes-broker-positive.log, artifacts/benchmark-fixes-protocol-final.log and artifacts/benchmark-fixes-release-final.log. The isolated database has zero active runs and no dedicated benchmark Host/CLI process remains. This exercised CLI RPC to Host/Broker, not native GUI acceptance.

Grader corrections: coalescer and pager loader timing

The ts-abortable-coalescer groups shared_inflight and clear_generation, and the ts-generation-pager group same-generation-dedup, asserted calls === 0 right after run() or loadNext(). That required the loader not to run synchronously. The task text only requires concurrent calls to share one loader call and does not say when the loader runs, so a synchronous implementation failed on a requirement the task does not state. The pager assertion also bundled this timing claim with one === two, which the task does require.

The pre-settlement calls === 0 assertions were removed. The spec claims stay: one === two in the pager group and the one-loader count checks. The two dedup groups now also assert exactly one loader call after the shared request settles (shared_inflight, and same-generation-dedup after Promise.all). clear_generation checks generation separation rather than sharing, so it gained no dedup assertion. Group names, group counts and other expectations are unchanged.

The retained one-microtask checks (calls === 1 after await Promise.resolve()) still assume the loader starts within one microtask. An implementation that defers the loader by two or more ticks would fail them. The task does not require that timing, so this is a known limit of those groups.

Oracle results. The starter and reference were graded by the repository self-test. The variants differ from the reference only in loader timing or dedup, and were graded in a scratch directory outside the checkout.

Task Starter Reference Synchronous loader Extra loader call for a shared request
ts-abortable-coalescer Fails all 5 fail-to-pass groups; passes the 3 pass-to-pass groups Passes 8/8 Passes 8/8 Fails shared_inflight (and abort_isolated)
ts-generation-pager Fails 4 groups Passes 6/6 Passes 6/6 Fails same-generation-dedup on its loader-count checks

Under the previous grader, the synchronous variant failed shared_inflight and clear_generation for the coalescer and same-generation-dedup for the pager, while the reference still passed. This is the over-strict behavior removed here.

Historical candidate diagnostics. The historical patches were applied to scratch copies and graded with the corrected graders. The artifacts were not modified.

  • ts-abortable-coalescer-1: shared_inflight now passes, since the candidate shares one loader call. clear_generation still fails: an older settlement deletes the newer generation's entry, so size() is 0 instead of 1. retry_after_rejection fails with TypeError, because the candidate calls .then on a plain loader value. The other five groups pass.
  • ts-generation-pager-1: same-generation-dedup fails only on one === two, because its async loadNext() returns a new promise for each call. Its loader-count checks pass, and the other five groups pass.

Both candidates still fail under the corrected graders, so their historical outcomes are unchanged. Under the previous graders both also failed on the timing assertion.

Historical cohort scores, reports, trajectories, patches and fixture records are unchanged. The 2026-10-29 cohorts used the previous fingerprints, so these two tasks are no longer matched against those cohorts. Earlier results for these tasks were measured under the previous graders. The 2026-10-20 observation that the pager grader required a synchronous-call restriction no longer applies after this correction.

Fingerprints changed as follows:

  • ts-abortable-coalescer: cf481b7373597ec45cc93dfe8517e9dde4a2d9b43bdfb3515b420bd339da0d29 to 671811635c338d411fe1e0531993d39efa9cb2cc08106df20855483b52df6718.
  • ts-generation-pager: d880cb834307b49141b67aca25b881f30b7701799663551128d1d7ccb865fc4d to e8215ce76475429a7174d264f9f1aaf792fecf20a7ac685d729c4cfe8288b3ac.

No test or tool pins either value, so none was updated. The provenance paragraph in tools/coding-benchmark/README.md named only the three earlier changed fixtures when this correction was recorded. That README was outside this correction's file scope; it was later updated to list all five changed fixtures.

Verification: both tasks validated through benchmark.py self-test (starter failed, reference passed). The benchmark Python tests ran with five C#-graded tests deselected because they invoke dotnet, which was not run for this change. The documentation guard tests/test_current_documentation.py passed.