Skip to content

Verification evidence — 2026-10-11

Historical snapshot only. Counts, commands and limitations describe that snapshot and do not establish current acceptance. Use current verification.

GUI/backend review fixes (2026-10-11)

All seven findings in that review have implementation changes and regression coverage: owned run workers, stale stream/history settlement, deleted-run SSE cleanup, provider key probes, latest Files refresh and export workspace resolution. That full Debug matrix passed 5,957 tests across 12 projects (Backend 1,049; GUI 1,352; artifacts/gui-backend-fixes-dotnet.log). Its initial export test requested unsupported writable formats; the test setup was corrected to the catalog's CanWrite set (artifacts/gui-backend-fixes-dotnet-initial.log). documentation guards passed 19 tests. Native race acceptance, live providers, Release publish and the full Python suite were not rerun in this follow-up. Prior Release/native/Python evidence in this document remains dated evidence.

Transcript scroll regression (2026-10-11)

The user reported upward jumps both on session open and while reading history. Pure viewport-policy tests reproduced the prior calculation errors: 8 failed, 2 passed (artifacts/gui-scroll-red.log). The fix rejects inconsistent extents even with follow enabled, preserves follow intent during layout changes, realizes the latest item on open and restores a visible-item anchor on bulk reload. Deferred callbacks check session, transcript and reader-input generations.

The first compilation failed because importing Windows.System made DispatcherQueuePriority ambiguous; a VirtualKey alias corrected it. The next focused run passed 40 tests and failed one source assertion: it prohibited scrolling even in the existing branch that does not reset the list. That assertion now covers the post-reset branch, while behavioral tests cover the viewport policy. Logs: artifacts/gui-scroll-focused.log, artifacts/gui-scroll-focused-green.log.

The scoped command scripts/test_dotnet.ps1 -Configuration Debug -ProjectFilter gui-winui3 passed 1,339 / 0 failed / 0 skipped (artifacts/gui-scroll-final-gui.log). These tests alone did not establish native acceptance; the next user-authorized GUI probe reproduced an additional virtualized-panel failure.

build.bat passed 11/11 stages (artifacts/gui-scroll-release.log). The documentation checks (test_current_documentation.py, test_guidance_route_counts.py, test_readme_counts.py) passed 19 tests (artifacts/gui-scroll-docs.log).

Native GUI follow-up

With explicit user authorization, the Release GUI opened the latest stored session (12 messages). A native UI Automation request to scroll to 100% ended at 0% after transient measurements around 34%; physical wheel input also exercised history scrolling. The first calculation-only repair was insufficient. Evidence: artifacts/gui-scroll-native-bottom-samples.json and artifacts/gui-scroll-uia-baseline.json. Temporary layout diagnostics showed a constant message count with changing extents and visible-item indices, rather than a history reset.

Setting ItemsUpdatingScrollMode="KeepScrollOffset" alone did not settle the latest position. The final change also schedules a guarded follow attempt after a completed view change when follow mode is active and realized row heights have moved the bottom. Reader input still cancels following and deferred work. Temporary diagnostics were removed from the shipped code.

On the final Release, opening the same session without another scroll request reached 100% during the observation window and remained there (artifacts/gui-scroll-uia-final-open.json: 25 samples, 200 ms apart). Wheel input moved up into history to 79.21%, then back down to 100%; screenshots before, after and 1.5 seconds after each gesture are under artifacts/gui-scroll-native-final-history-{up,down}-*.png. This is acceptance for that stored, idle session; it does not cover every session, active streaming or a turn-end reload. Initial virtualization still needs layout settling.

Final checks: 1,340 GUI tests / 0 failures / 0 skips (artifacts/gui-scroll-native-final-gui.log), 11/11 Release stages (artifacts/gui-scroll-native-final-release.log). The full backend/process matrix was not rerun for this local GUI change. A diagnostic attempt to publish only the GUI over the flattened directory made Host startup fail; rebuilding through build.bat restored the complete distributable before final acceptance.

Review remediation (2026-10-11)

An external review pass over the whole tree produced 11 high, 16 medium and a set of low findings; all high findings and the large majority of the rest are repaired in this pass. The highlights, by area:

  • CI/tooling. scripts/check_python_runtime_boundary.py treated the GUI's vendored browser assets (xterm.js, addon-fit.js) as a shipped interpreter and exited 1 on the repository itself while CI runs it bare. The two paths are now the pinned, exact-path exception; the docs record the enforcement.
  • Provider runtime. A 120-second idle watchdog in the SSE reader turns a silent provider into the retryable provider_timeout on all three protocols; stream-level error events keep their HTTP retryability (rate_limit_error/api_error, server_error/overloaded); a .done-only Responses arguments stream is accepted; empty model_tools omits the key on every protocol; extra sibling fields on a valid Anthropic block are ignored.
  • Harness. Batch aborts revoke the approvals they registered; cleanup never replaces the original exception; a retried tool execution publishes a reset frame (and the client model now parses the always-emitted reset: null field that had made every live tool-output frame unparsable); the two native-history projectors share one matcher with one identity guard.
  • Persistence. Infrastructure SQLite failures in a work-state update abort the commit; recovery consume for a missing run fails closed; a failed migration rollback no longer masks the error; request-path schema initialization is memoized per database path; the conversation outbox spends a bounded retry budget before failing an intent.
  • CLI/RPC/SDK. TUI wind-downs cancel the in-flight run and report discarded queue entries; Esc between turns is remembered; panel requests answer a wind-down with a cancel; run/attach report a failed cancel; bridge backpressure drains through RunAsync with exit code 3 instead of killing the process from the pump thread; filtered or unreadable durable frames end the subscription with run.detached; run.detach accepts a run id for attach cleanup; SDK handler exceptions become warnings; stdin backpressure is respected; cmd metacharacters in .cmd spawn arguments fail closed.
  • GUI. Session/workspace/diagnostics async-void handlers catch their own failures; the last session click wins a fast switch; settings loading always clears its flag; trace lines go through one bounded background writer.

Evidence: the full 12-project matrix passed 5,909 tests / 0 failures / 0 skips (preset adoption + mode hardening; this run), the Python suite passed 214 with one Windows/POSIX skip (artifacts/review-remediation-full-python.log), the SDK typecheck/test/build passed 22 tests, and the scripted RPC smoke passed end to end (PASS, terminal run.event at sequence 7). One CLI matrix run flaked once on a timing-sensitive test (1 failure; two reruns green, including the final full matrix) and is recorded here rather than retried away. Deferred with reasons: the reply-effect failure keeps its existing durable best-effort activity marker; a Length-finish step deliberately does not release held-back text (the whole step is suppressed); InteractiveLineReader's key loop and the reconciliation-target scan keep their existing shapes, recorded here rather than rewritten in this pass.

Dialog collision fix (2026-10-11)

A GUI bug report showed "Coding Agent could not start: An async operation was not properly started. Only a single ContentDialog can be open at any time." after sending a message into a workspace that still needed trust. WinUI allows one ContentDialog per XamlRoot, so the trust prompt's ShowAsync collided with another dialog (open or still closing), the trust path projected that as a start failure, and the run was marked handled — leaving the parked run with no approve affordance. Every app dialog now goes through ContentDialogGate, which serializes them and retries the single-dialog condition briefly; a prompt that still could not open no longer marks the run handled (the monitor refresh retries it after a short cooldown) and reports a bounded status instead of a start failure. Source-contract tests pin that every new ContentDialog site goes through the gate and that no site calls ShowAsync directly. GUI tests passed 1,303 including the two new gate contracts; the full matrix above is the verification.

Tool boundary repairs (2026-10-11)

The four outstanding findings at 50fd2e28 are repaired: web fetch pages no longer split UTF-16 surrogate pairs, read windows cannot return a non-EOF zero-progress page, grep strips the UTF-8 BOM before searching, and glob traversal observes deadline/caller cancellation after its worker starts. Small windows and offsets inside a character return named, repairable errors; malformed UTF-8 at EOF is rejected instead of generating an endless cursor. Protocol descriptions and the tool guide document these recovery actions.

All 12 initial regression cases failed before repair (artifacts/tool-boundary-red.log). The affected focused set passed 51 tests (artifacts/tool-boundary-green.log); adding caller-cancellation coverage produced 13 passing boundary cases (artifacts/tool-boundary-final-focused.log). The final 12-project .NET matrix passed 5,879 tests / 0 failures / 0 skips, including 941 Broker tests (artifacts/tool-boundary-full-dotnet.log). The original synthetic probe was repeated: BOM/no-BOM searches now agree, small file windows fail with the repair code, web pages survive strict UTF-8 encoding and JSON roundtrip, and three 1 ms glob calls return timed_out (artifacts/tool-boundary-smoke-green.log). Cancellation is cooperative; these timings do not establish a hard real-time deadline. The Python suite passed 213 tests with one Windows/POSIX skip (artifacts/tool-boundary-full-python.log). Final bundled Help checks after README updates passed 12 tests (artifacts/tool-boundary-final-help.log). build.bat passed all 11 Release stages and refreshed the self-contained GUI, CLI and Host package (artifacts/tool-boundary-release.log). The prior discovery/web evidence below remains historical. These are tool correctness checks, not a new real-provider coding benchmark or native UI trial.

Discovery and output-projection upgrades (2026-10-11)

Commit b00282f0 records the preceding web repairs. Seven new discovery regressions failed before this upgrade (artifacts/tools-discovery-red.log): recursive globs excluded root files, a literal wildcard placeholder name was misread, long-line previews omitted their match, exact final pages claimed more results, no explicit next cursor was returned, and small queries read all candidate files. The focused discovery run then passed all 41 tests (artifacts/tools-discovery-green.log). Glob/grep now share a literal-aware nonbacktracking matcher. Grep consumes ordinal bounded batches, collects only up to the offset/page/lookahead bound, and exposes actual scanned counts, preview columns and separate scan-incompleteness/next-page metadata.

A Host projection regression also failed before repair (artifacts/tools-discovery-projection-red.log): a scan/time bound with no known next page invented a nonadvancing model cursor. Modern receipts now use next_offset for continuation; legacy receipts retain the old truncated compatibility path. Two web projection regressions failed before repair (artifacts/tools-web-projection-red.log): generic clipping lost structured JSON/cursors, and a full fetched page with more pages supplied no model cursor. Structured web projection preserves its wrapper/marker and counts only the extracted text for a UTF-16 cursor. The final focused Host run passed 32 tests (artifacts/tools-web-projection-green.log).

The first full discovery matrix stopped on one Help inventory failure: 927 Broker tests passed, but longer documentation displaced op=write from its bounded excerpt (artifacts/tools-discovery-full-dotnet.log). Splitting the details into a dedicated documentation section fixed the bundled Help output without weakening the inventory assertion (artifacts/tools-discovery-help-green.log). A subsequent 12-project discovery matrix passed 5864 tests before the two web projection cases were added (artifacts/tools-discovery-full-dotnet-final.log). The final 12-project matrix passed 5,866 tests / 0 failures / 0 skips, including 928 Broker and 1043 Host tests (artifacts/tools-upgrade-full-dotnet.log).

An actual-service read-only probe searched the real Protocol checkout through ToolExecutionService, using a synthetic session with only workspace/system read scopes, no provider requests or user credentials (artifacts/tools-upgrade-smoke/report.json). glob **/*.cs returned 37 paths, including 15 root files. Three grep web_fetch pages followed cursors 0, 1, 2 and ended with no next cursor; the first page searched 8 of 37 discovered files, while the final page scanned the remainder to establish completion. This is functional and file-count evidence, not a timing benchmark or live-model/native-UI acceptance. Free upstream search relevance is unchanged. The full Python suite passed 213 tests with one POSIX-only skip on Windows (artifacts/tools-upgrade-full-python.log), and build.bat passed all 11 Release stages (artifacts/tools-upgrade-release.log).

Web-tool repairs and remaining upstream limits (2026-10-11)

Eight new replay cases failed on the original implementation (artifacts/web-quality-red.log): known-engine navigation anchors were accepted, Bing used a breadcrumb rather than the result heading/destination, JSON scalar responses escaped as an unbounded parser exception, plain code lost whitespace and generic syntax, and ordinary documentation exceeded the download bound. The final repair uses AngleSharp 1.8.4 for local DOM parsing, preserves code, text and links, and separates the 4 MiB download ceiling from 32000-character output pages. An initial restore with AngleSharp 1.4.0 was rejected by the existing NuGet audit policy (artifacts/web-anglesharp-restore.log); the final version restores and builds without that warning. No audit rule was weakened.

The complete sequential Debug matrix passed 5,856 tests / 0 failures / 0 skips, across all 12 projects (artifacts/web-repair-full-dotnet.log). Broker contributed 921 passing tests, including 19 new adapter/executor cases. Paging is also checked through the real service approval path. Documentation guards passed 15 tests (artifacts/web-repair-docs.log). The first full Python run had 1 failure, 212 passes and 1 skip because README's older 5809-test badge did not match the already-recorded 5837-test run (artifacts/web-repair-full-python.log); README and the current verification row now reflect the new 5856-test run. The final full Python rerun passed 213 tests with one Windows-inapplicable POSIX skip (artifacts/web-repair-full-python-final.log). build.bat then passed all 11 stages and published the updated self-contained distribution, including AngleSharp (artifacts/web-repair-release.log). Historical figures below remain intact.

The final direct-service public-HTTP rerun made 12 calls, with synthetic sessions/approvals and the production connection policy, no keys or model requests (artifacts/web-live-check/report.json, artifacts/web-live-check/run-final.log). All five text URLs succeeded: Example Domain, Microsoft Learn C# async, Python sqlite3, TypeScript narrowing, and the raw TypeScript README. Python sqlite3 was read to completion over four pages (offsets 0, 32000, 64000, 96000), including transaction control. The PDF probe still correctly failed as web_content_type_not_supported; PDF extraction and browser/JavaScript execution are outside this HTTP text tool.

Search no longer accepts DuckDuckGo's homepage or Mojeek navigation as results. The isolated engine rerun rejected those layouts and Google as web_search_parse_failed; Bing supplied actual titles, snippets and decoded destination URLs (artifacts/web-engine-check/report.json). However, all three live coding queries still returned weak or unrelated upstream Bing results. The query echoed in Bing's response was complete, and ordinary English-market and RSS diagnostic variants did not fix relevance. Additional public Brave and Yahoo probes returned HTTP 429 and HTTP 500 respectively (artifacts/web-engine-check/extra-sources.log); they were not added to the engine set. No CAPTCHA was solved and no browser session was used.

This repairs the parser and fetch defects; it does not establish reliable live search relevance. Free engines remain dependent on their upstream availability and ranking. Deterministic tests, live HTTP text retrieval, live-model behavior and interactive UI acceptance are separate evidence.

Live web-tool diagnostic (2026-10-11)

On checkout 18180d7c, 47 focused Broker web adapter, executor, capability and connection-policy tests passed (artifacts/web-tools-focused-tests.log). A separate diagnostic then called the real ToolExecutionService, with synthetic session scopes, one-use approvals, the production HTTP connection policy and default enabled web options (Google plus the fixed fallback race, 256 KiB response bound). This used public HTTP requests, no model generation, no user session and no API keys. Report: artifacts/web-live-check/report-before-fix.json.

All three coding-oriented search queries reported success but returned only the DuckDuckGo homepage with title here, rather than query results. An engine-by-engine check of the Python query confirmed Google failed to parse, DuckDuckGo returned that same false result, Bing returned tracking URLs and placeholder snippets, and Mojeek returned navigation links. DuckDuckGo's saved HTTP 202 response was a bot-challenge page. Known-engine parsers fall through to generic absolute-anchor extraction when their expected result blocks are absent; this allows a challenge/navigation link to win the race and cancel other engines. Evidence: artifacts/web-engine-check/report-before-fix.json and artifacts/web-engine-check/duckduckgo-response.html. No challenge was solved.

Of six fetch URLs, Example Domain, the Microsoft Learn C# async page and the raw TypeScript README succeeded. Python's sqlite3 documentation and the TypeScript narrowing handbook failed with web_response_too_large. The PDF probe failed with the expected web_content_type_not_supported. Fetch is HTTP text extraction without JavaScript execution; its regex sanitizer flattens whitespace and does not preserve code-block layout or links as Markdown.

This is one small live sample from this machine, not a stability or relevance benchmark. It demonstrates a false-success search defect and default fetch limits that exclude ordinary documentation pages. Production code and web configuration were not changed in this assessment.

Indexed Chat Completions call repairs (2026-10-11)

Commit bb27c06f records the preceding terminal-argument and truncation repairs. The subsequent SSE replay pass reproduced four indexed-call failures through the real Chat Completions adapter and model runner: interleaved arguments were attached to the wrong call, a repeated id started a new call, a split function name was rejected, and out-of-order opening chunks reversed call order. All four failed before the repair (artifacts/indexed-chat-tools-red.log) and passed after it (artifacts/indexed-chat-tools-green.log).

The adapter now assembles indexed calls in a separate bounded component and emits them in index order at completion. Wholly unindexed streams retain the existing compatibility path; mixed indexed/unindexed metadata, conflicting ids and oversized arguments are rejected. A replay with two indexed calls lacking provider ids also reproduced an ambiguous-call failure (artifacts/indexed-chat-tools-missing-ids-red.log); distinct generated ids now keep those calls separate. During iteration a regression was caught where an argument-only chunk's usage was dropped (artifacts/indexed-chat-tools-usage-red.log) and corrected before final verification. Indexed length output still returns no executable calls through the preceding truncation repair.

Final verification: scripts/test_dotnet.ps1 -Configuration Debug passed all 12 projects, 5,837 passed / 0 failed / 0 skipped (artifacts/quality-indexed-chat-full-dotnet.log), including the 15 new indexed SSE replay cases. Current documentation and route-guidance checks passed 15 tests (artifacts/quality-indexed-chat-docs.log), and git diff --check passed.

These are deterministic SSE replay results, with synthetic credentials and no network provider calls. Another live-model benchmark, Release package build and interactive UI acceptance have not been run for this repair.

Tool argument and truncation repairs (2026-10-11)

An omitted bash run-mode argument vector now means an empty vector, matching the advertised schema. Mode-specific argument parsing retains its named error instead of flattening it to adapter_error. The model runner preserves a provider's length finish before parsing native calls or text selections and returns no executable calls or continuation for that incomplete step. The Harness's existing bounded truncation recovery stays in charge.

Three terminal regression cases failed before the fix (artifacts/terminal-arguments-red.log): omitted arguments, missing executable and missing shell script. All eight launch-feedback cases passed after the fix (artifacts/terminal-arguments-green.log), and the Broker suite passed 902 tests (artifacts/terminal-arguments-broker-suite.log). Three model-step cases also failed before the fix: a complete native call was promoted despite length, an incomplete native call threw a parse error, and a text selection was promoted. All three pass after the fix (artifacts/truncated-step-red.log, artifacts/truncated-step-green.log).

Final verification: scripts/test_dotnet.ps1 -Configuration Debug passed all 12 projects, 5,822 passed / 0 failed / 0 skipped (artifacts/quality-tool-arguments-truncation-full.log). The focused current documentation and route-guidance guards passed 15 cases (artifacts/quality-tool-arguments-truncation-docs.log), and git diff --check passed. The Release package, full Python suite and interactive native UI were not revalidated in this repair pass.

Trace inspection of run 3 found that Python's empty write arguments occurred at 8,192 output tokens, while the provider reported tool_calls, not length. The truncation repair above does not change that upstream finish marker or raise the pinned output limit. Python and TypeScript also supplied shell scripts with mode=run; their existing malformed-call feedback correctly requested shell mode. These repairs have not yet been evaluated by another real-provider benchmark, so the latest measured resolution remains 0/3 below.

Post-terminal-fix SWE rerun (2026-10-11)

For the subsequent ten-task Standard baseline and its matched follow-up probes, see the single-agent readiness diagnostic (2026-10-13, retained in git history). The runtime, preset and difficulty mix differ from this older three-task run.

The three custom SWE tasks were rerun on committed 96f658c2, using the rebuilt Release Host/CLI, Kilo stepfun/step-5-preview-free, chat_completions, confirmed full_access, one fresh workspace/session per task and one attempt with a 300-second agent deadline. Each task's fixture, effective instruction hash and Git baseline match run 2. Graders and reference solutions were not supplied as model input. This rerun does not cover the six Terminal-style tasks.

Task Run outcome Agent seconds Passing groups Reward
cs-lease-renewal timeout 300.016 6/7 0
py-migration-rollback timeout 300.013 2/6 0
ts-session-replay timeout 300.013 2/7 0

C# changed only src/LeaseRegistry.cs and still failed expiry/takeover behavior. Python and TypeScript left source unchanged and exported empty patches; their passing groups are baseline preservation behavior. Durable receipts record two malformed Python calls, one malformed TypeScript call and one nonzero C# command. All three timed-out runs were durably cancelled, with no nonterminal run left in the isolated database, and the diagnostic Host was stopped. Runtime binary hashes remained unchanged throughout the run.

Overall resolution remains 0/3. Run 2's Python candidate passed 6/6 groups; this attempt produced no Python repair. One stochastic attempt per task cannot attribute that difference to the terminal fixes or establish stable improvement. Reports, trajectories and patches are under artifacts/quality-swe-20261011-run3/; comparison-and-runtime-audit.json records matching fixtures and final cancellation evidence. This is live-model evidence on a small custom corpus, not an official SWE-bench score or native UI acceptance.

Terminal launch and exit-code repairs (2026-10-11)

Windows process creation now opens and retains the Process handle while the child is suspended. A child that exits before the caller waits still retains its real exit code. Executable and shell resolution run before task registration, retaining executable_not_found and executable_not_trusted instead of collapsing them into process_start_failed or reporting an unresolved background task as started. Execution permissions are unchanged.

The two exit-code regression cases failed on the old implementation (artifacts/quality-process-exit-red.log) and all six suspended-process tests passed after the repair (artifacts/quality-process-exit-green.log). The valid missing-executable regression failed for foreground and background requests (artifacts/quality-launch-feedback-valid-red.log); the first fixture attempt omitted the optional argument array and exposed a separate executor/schema mismatch, now covered by the subsequent argument-vector repair; that first log is retained separately. Focused launch/streaming checks passed 20 cases (artifacts/quality-terminal-green.log). A local diagnostic then ran Python directly and through cmd, retaining exit code zero in both cases (artifacts/quality-terminal-diagnostic.log). This does not change which bare executable directories are trusted.

Environment failures are retained: the first full matrix was interrupted after a 512 MiB fixture could not be allocated on C (artifacts/quality-terminal-full-dotnet.log). A temporary root inside the checkout invalidated a no-repository fixture (artifacts/quality-terminal-full-dotnet-retry.log). Moving that root outside the checkout on D passed Broker/CLI/client/core checks, then failed five Windows key-store ACL tests (artifacts/quality-terminal-full-dotnet-final.log). Both volumes are NTFS; the D-root ACL setup was incompatible with those tests. The final invocation uses the normal Windows temporary directory after C free space recovered. No permission tests or key-store implementation were weakened.

Final verification: the normal-temp full .NET matrix passed 5,815 tests in all 12 projects, with no failures/skips (artifacts/quality-terminal-full-dotnet-complete.log). Documentation guards passed 9 tests (artifacts/quality-terminal-docs.log); git diff --check passed. The full Python suite was not rerun for these C# changes; its preceding 213-pass result remains recorded separately. Release publishing completed all 11/11 stages (artifacts/quality-terminal-release.log).

No new live-model benchmark or interactive native UI acceptance is claimed for these launch fixes. The earlier custom SWE results remain unchanged.

Coding recovery and feedback repairs (2026-10-11)

Suppressed repeated-tool proposals are retained for internal audit, without orphan native tool calls in model history. Repetition advice still counts toward its existing bound and cannot hide a failed receipt from completion checks. Terminal start/exit/timeout failures now preserve an exit code and bounded, already-redacted diagnostics for model repair. Windows shell guidance names PowerShell explicitly. No provider fallback or permission-policy change was made.

Regression evidence:

  • Release distributable rebuilt successfully: 11/11 stages (artifacts/quality-release-final.log).
  • Final full Python tooling suite: 213 passed, 1 skipped (artifacts/quality-pytest-complete.log), including the cache-patch regression. The intermediate run passed 212 (artifacts/quality-pytest-final.log); the Windows skip remains the POSIX process-group case.
  • Final full .NET invocation: 5,809 passed / 0 failed / 0 skipped, all 12 projects (artifacts/quality-dotnet-complete.log).
  • The first full matrix stalled in CLI and was interrupted; its log is artifacts/quality-dotnet-final.log. The isolated CLI diagnostic with --blame-hang-timeout 90s --blame-hang-dump-type none completed all 1,278 tests (artifacts/quality-cli-diagnostic.log). The hang did not reproduce; its cause remains unconfirmed and is not claimed fixed.
  • Durable SQLite/native-provider history: two causal RED failures in artifacts/quality-repetition-red3.log, then two passing recovery cases in artifacts/quality-repetition-green2.log. Earlier red/red2 logs record corrected test setup/compilation failures, not the reproduced production bug.
  • Advice-hidden repetition: two RED cases in artifacts/quality-guard-red.log. Advice-hidden verification failure: one RED case in artifacts/quality-false-success-red.log. Core subsequently passed 481 tests (artifacts/quality-core-green.log).
  • Missing compiler diagnostics: receipt mapping and a real failed PowerShell child both failed before the fix (artifacts/quality-terminal-feedback-red.log). All seven related mapping/history/process checks passed afterward (artifacts/quality-terminal-feedback-green.log). Existing terminal redaction coverage is retained; these diagnostics are private receipt content, not public SSE.
  • Benchmark instructions previously omitted immutable-file and scratch-file restrictions enforced by grading. A failing prompt regression reproduced this omission; all nine benchmark tooling tests passed after disclosing the constraints. Logs: artifacts/quality-benchmark-prompt-red.log, artifacts/quality-benchmark-tests.log. Grading criteria remain unchanged; prepared trials now record the effective instruction hash.
  • Live Python exposed cache files entering patch export despite tree grading excluding them. The staged-cache round-trip regression failed before the fix (artifacts/quality-benchmark-patch-red.log); all 10 benchmark tests passed afterward (artifacts/quality-benchmark-tests-final.log). The actual Python candidate source patch was replayed into a fresh baseline, passed every group, and retained the same candidate hash (artifacts/quality-swe-20261011-run2/py-migration-rollback/patch-validation.json).

Live evidence uses the same configured Kilo stepfun/step-5-preview-free, chat_completions, one attempt per task, isolated ephemeral Host/data roots, and disposable workspaces. It does not send reference solutions to the model.

  • Small shipping-boundary C#: standard parked for the existing terminal approval (artifacts/quality-small-csharp-20261011-run1/report.json). A separate explicitly confirmed full_access trial completed in 70.906 s, changed only src/Shipping.cs, and passed the independent grader (artifacts/quality-small-csharp-20261011-full-access-run1/report.json). This used intermediate Debug binaries; it is not a controlled before/after score.
  • Initial hard custom SWE diagnostics, 300 s/task, full_access: C# and Python timed out; TypeScript failed model_output_truncated after 284.860 s and failed five of seven groups. C# changed a project file; Python left a scratch file. The original prompts did not disclose these integrity restrictions, so the resulting 0/3 is diagnostic evidence with a prompt defect, not a fair quality score. Reports: artifacts/quality-swe-20261011-run1/.

  • Final hard diagnostic used rebuilt flat Release binaries, disclosed final-workspace constraints and 300 s/task; binary and instruction hashes are retained in artifacts/quality-swe-20261011-run2/manifest.json and each trial.json.

Task Agent outcome Seconds Grader groups passed Live reward
C# lease renewal completed; expiry semantics still incorrect 294.992 6/7 0
Python migration rollback timed out; candidate passed every group 300.017 6/6 0
TypeScript session replay failed model_output_truncated 257.804 2/7 0

Live reward requires both a completed agent run and a fully passing candidate: 0/3 resolved, while 1/3 candidate patches passed all grading groups. The corrected Python source-only patch also passed fresh-baseline replay; its original prediction.patch retains the pre-fix cache-export failure as historical evidence. No model-generated source was repaired by the evaluator. Receipt metadata from three real failed commands confirms compiler feedback reached private payloads (artifacts/quality-swe-20261011-run2/failure-feedback-evidence.json). Reports and trajectories are under artifacts/quality-swe-20261011-run2/. This run does not establish a controlled improvement rate: several harness and prompt corrections were applied, there is one attempt, and the deadline is short.

These custom tasks and short single attempts are not official Terminal-Bench or SWE-bench results, a model-independent measure, or native UI acceptance.

Terminal OS sandbox retirement (2026-10-11)

ADR 0015 retires AppContainer profiles, ACL grants, capability attributes and sandbox escape UI/arguments. Terminal and managed Git launch as the current Windows user. Ordinary preset and approval policies, process-tree cleanup, cancellation, timeout and output bounds remain. A real standard-preset approved PowerShell child writes outside workspace/TEMP and connects an ephemeral loopback listener without a network capability, proving the former OS boundary is absent.

Credential storage and structured protected-path denials remain. Child processes do not inherit secret environment variables; all stripped values now seed terminal output redaction. Regression tests reproduced the old redactor leaking a key across its emission boundary (three RED cases), then verified chunk splits, longest matches and complete overlapping matches. A real process writes a synthetic key in separate chunks; both frames and receipts redact it. This does not claim that unrestricted current-user commands cannot retrieve credentials independently.

Verification:

  • The first full .NET run failed an obsolete sandbox-only approval-ledger test; that case was removed while ordinary approval-memory coverage was retained (artifacts/remove-sandbox-dotnet.log).
  • The second full run passed the first eleven projects, then failed one GUI assertion expecting obsolete sandbox copy (artifacts/remove-sandbox-dotnet-green.log). The corrected GUI suite passed 1,299; the final Broker suite passed 894, including the fourth redactor regression (artifacts/remove-sandbox-gui-final.log, artifacts/remove-sandbox-broker-final.log). Combined with the unchanged projects from the full run, the total is 5,802 passing tests across 12 projects. A final complete invocation subsequently passed all 5,802 / 0 failed / 0 skipped (artifacts/remove-sandbox-dotnet-final.log). No failed assertion was weakened to permit an execution-policy regression.
  • Python tooling suite: 211 passed, 1 skipped (artifacts/remove-sandbox-pytest-final.log). The first run exposed stale documentation vocabulary and a README badge; both were corrected without changing the guards (artifacts/remove-sandbox-pytest.log).
  • build.bat: 11/11 stages passed (artifacts/remove-sandbox-release.log), refreshing the flat distributable.
  • Redactor RED and initial GREEN evidence: artifacts/remove-sandbox-redactor-red.log and artifacts/remove-sandbox-redactor-green.log.

The scripted CLI RPC smoke completed its session/run/attach/event workflow (artifacts/remove-sandbox-rpc-smoke.log). It used an isolated ephemeral Host, synthetic model and temporary data root; its ignored runner copy retained the temporary directory instead of deleting it. This proves transport wiring.

A separate small C# shipping diagnostic used the new flat Release binaries and the current user-selected Kilo stepfun/step-5-preview-free, chat_completions, standard, one attempt and a 180-second bound. The pristine baseline failed the exact-threshold assertion. The live run read files and issued tool calls, then failed after 15.058 seconds with model_history_invalid, following a malformed_call receipt and repeated_tool_call recovery note. No source file changed and independent grading remained red. This trace does not establish the root cause; sandbox retirement has not demonstrated improved coding-task completion. Evidence: artifacts/ru-small-csharp-no-sandbox-20261011/manifest.json, trajectory.json, report.json and prediction.patch. The run preserved the model selection, shut down its own isolated Host and retained its data root. No interactive desktop UI was operated.