Ru coding benchmark¶
Three established suites contain 20 task fixtures (six Terminal, three SWE and eleven Everyday) across C#, Python and TypeScript, with five opt-in skill workflow fixtures and a separate three-task opt-in repository suite. They use Terminal-Bench and SWE-bench evaluation patterns; they are not official datasets or leaderboard scores. The corpus is public, so runs can be contaminated by prior exposure to its reference solutions.
The opt-in skills suite contains five dependency-free Python end tasks for
explicit invocation with arguments, implicit workflow relevance, unrelated and
manual-only negative controls, and supporting-resource use. Tasks use the same
baseline-fails/reference-passes grader contract, immutable fixture checks and
content fingerprints as other nonterminal suites. skill_scenario and
expected_skill_selection are cohort labels for later analysis;
observed_skill_selection remains unknown because current durable RPC
summaries do not expose authoritative skill identity. End-task acceptance does
not prove skill selection or adherence. Skill tasks are excluded from
--suite all; select --suite skills explicitly. Offline self-test never
calls a provider.
The opt-in repository suite adds three multi-file fixtures across C#, Python
and TypeScript. Per-task editable_paths can include explicitly named project
files; graders and solutions stay outside the candidate workspace. Each task
has independent behavior and preservation groups, and the offline oracle
requires the starter to fail and the reference to pass. The TypeScript
repository task runs tsc --noEmit as a required grader group. If tsc is not
installed, grading returns unavailable with
typescript_compiler_unavailable; runtime checks are not substituted for type
checking. This requirement applies only to the repository task. Existing
TypeScript tasks continue to use Node type stripping and runtime checks without
tsc.
Grader correction provenance (2026-10-29)¶
Five fixture fingerprints change after the recorded Kilo cohort: Todo accepts
independent read-only dictionaries; Atomic Ledger accepts non-castable immutable
snapshots and additionally checks repeated valid transfer pairs; Scheduler counts
caught exceptions as failed checks instead of emitting invalid zero-check groups;
ts-abortable-coalescer and ts-generation-pager drop the pre-settlement
calls === 0 loader-timing assertions and assert exactly one loader call for
shared requests.
Behavioral regressions cover valid alternatives and invalid implementations.
Historical reports/patches retain their original fingerprints and raw scores;
offline corrected-grader diagnostics are labelled separately. Do not silently
compare or merge scores across these fixture revisions.
Tasks¶
| Suite | Task | Main difficulty |
|---|---|---|
| Terminal | cs-atomic-ledger |
Ordered atomic batches, idempotent replay, overflow, simultaneous writers |
| Terminal | cs-single-flight-cache |
Shared loads, independent cancellation, invalidation generations, TTL and LRU |
| Terminal | py-stream-parser |
Arbitrary byte fragmentation, strict UTF-8, bounded frames and error recovery |
| Terminal | py-dependency-scheduler |
Dependency validation, state transitions, retries and transactional rejection |
| Terminal | ts-generation-pager |
Deduplicated promises, refresh races, stale completions and isolated snapshots |
| Terminal | ts-transaction-store |
Optimistic conflicts, atomic validation, reentrant notifications and snapshots |
| SWE | cs-lease-renewal |
Repair versioned lease renewal, expiry, stale ownership and concurrent mutations |
| SWE | py-migration-rollback |
Repair SQLite schema/version rollback and migration transaction ownership |
| SWE | ts-session-replay |
Repair duplicate deliveries, gaps, conflicting replay and generation replacement |
| Everyday | cs-search-pagination |
Repair filtering/paging order while preserving validation |
| Everyday | cs-todo-summary |
Add pending counts by normalized owner with independent snapshots |
| Everyday | py-config-isolation |
Repair configuration leakage between calls |
| Everyday | py-stable-dedupe |
Add stable record deduplication and preserve missing/falsy IDs |
| Everyday | ts-emitter-lifecycle |
Repair unsubscribe idempotence and mutation during dispatch |
| Everyday | ts-query-builder |
Add deterministic query encoding without losing falsy values |
| Everyday | py-tool-workflow |
Create a new Unicode/spaced path, repair existing functions and verify grouped output |
| Everyday | py-batch-reconcile |
Parse a multi-source export, normalize identities, resolve conflicts and preserve first-seen order |
| Everyday | py-checkpoint-journal |
Validate ordered event delivery and restore checkpoints atomically |
| Everyday | py-log-redaction |
Recursively redact nested secrets while preserving safe values and input containers |
| Everyday | ts-abortable-coalescer |
Share asynchronous requests while isolating caller cancellation and stale generations |
| Skills | explicit-currency-rounding |
Resolve /skill currency-rounding precision=2 and produce a correctly rounded invoice total |
| Skills | implicit-calendar-slots |
Select a relevant scheduling workflow for conflict-free intervals |
| Skills | negative-unrelated-slug |
Complete standalone normalization with an unrelated skill in the catalog |
| Skills | negative-manual-only-csv |
Complete CSV cleanup without auto-invoking a manual-only workflow |
| Skills | supporting-resource-json-summary |
Apply aliases defined in a workflow supporting resource |
| Repository | cs-atomic-transfer |
Validate and apply cross-file transfer batches atomically |
| Repository | py-dependency-plan |
Produce a stable dependency order while rejecting invalid graphs |
| Repository | ts-event-policy |
Preserve strict cross-file event policy typing and filtering |
These are compact, dependency-free repositories. Terminal/SWE tasks have difficult correctness constraints; Everyday tasks are focused bug repairs and small features with explicit requirements and runnable public regression checks. The small corpus measures those constraints, not arbitrary large-repository maintenance or a statistically established difficulty level. Terminal tasks supply a public smoke program. External graders test additional cases, using fixed random seeds, explicit gates and manual clocks where needed.
Offline use¶
Run from the Ru checkout. Python 3.11+, .NET SDK 10 and Node 22.18+ must be
available. No packages are downloaded by this tooling. TypeScript verification
uses Node's type stripping and checks runtime behavior; it does not run tsc.
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py list
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py validate
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py self-test --attempts 3 --report artifacts/coding-oracle.json
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py self-test --suite everyday --report artifacts/everyday-oracle.json
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py self-test --suite skills --report artifacts/skills-oracle.json
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py list --suite repository
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py self-test --suite repository --report artifacts/repository-oracle.json
The skills corpus only needs Python 3.11+ and the standard library. Its files are fixtures for future controlled cohorts; the suite does not install skill roots, call a provider, or infer a selection from end-task correctness.
self-test prepares a fresh Git repository, grades its starter, overlays the
reference sources, and grades again. Every starter must fail; every reference
must pass. SWE starters must fail every FAIL_TO_PASS group and pass every
PASS_TO_PASS group. A passing self-test validates fixtures and graders. It
does not execute Ru or measure model quality.
Repository-wide Ruff excludes the deliberately faulty candidate repositories
(repository/ and environment/workspace/); graders and reference solutions
remain in the lint gate. Style cleanup must preserve the starter-fails and
reference-passes oracle.
Prepare a candidate workspace for any coding agent, then grade its changes:
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py prepare --suite terminal --task cs-single-flight-cache --destination artifacts/cache-trial
# Give the agent artifacts/cache-trial/instruction.md and workspace/ only.
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py grade --suite terminal --task cs-single-flight-cache --workspace artifacts/cache-trial/workspace --report artifacts/cache-trial/result.json
A destination must not already exist. Only the task's src/**/*.cs,
src/**/*.py or src/**/*.ts may change. Modified project files, smoke checks,
README files, extra non-source files, symlinks and junctions invalidate a
candidate. Build/cache directories are excluded from grading. Grading copies
admitted files into a fresh directory and runs the external verifier there.
The effective instruction given to every agent includes these final-workspace
constraints; scratch checks must use temporary locations or be removed before
completion. trial.json records the effective instruction's SHA-256 separately
from the fixture fingerprint so prompt revisions can be distinguished.
Patch export applies the same build/cache exclusions as workspace grading,
including already-staged caches; they cannot invalidate an otherwise admitted
source patch when it is replayed into a fresh baseline.
Each complete named group must report positive check counts and consistent
failure evidence. Reward is 1 only if every group passes and the verifier
exits successfully, otherwise 0. Missing toolchains report unavailable
with null reward. Process time and output are bounded; C# compilation consumes
the same verifier deadline as execution. Syntax/build failures fail a task.
Windows child processes start suspended, enter a kill-on-close Job Object, then resume. Assignment failure stops launch; no uncontained retry is used. Tree cleanup also applies after the original process exits. POSIX execution uses a process group. These process lifetime bounds do not restrict filesystem or network access.
SWE-style patches¶
Each issue instance has its own deterministic Git baseline SHA and separate regression and preservation groups. Grade a patch against a fresh baseline:
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py prepare --suite swe --task py-migration-rollback --destination artifacts/migration-author
# After the agent makes its source changes:
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py export-patch --suite swe --task py-migration-rollback --workspace artifacts/migration-author/workspace --patch fix.patch
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py grade-patch --suite swe --task py-migration-rollback --patch fix.patch --destination artifacts/migration-evaluation --report artifacts/migration-result.json
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py export-swe --report artifacts/instances.jsonl
export-swe writes repo, instance_id, base_commit, problem_statement,
FAIL_TO_PASS, PASS_TO_PASS, version and fixture fingerprint. Reference
patch and test_patch are omitted as empty strings unless --include-oracle
is explicitly passed. Keep oracle exports away from agent input.
These synthetic local repository names are not registered with the official SWE-bench Docker harness. The export uses its record conventions; running the official harness needs separately supported repositories/images and dataset integration. Use this runner to evaluate these custom instances.
Native Ru trials¶
Live execution requires an existing Ru executable or DLL, configured provider
credentials, and two explicit flags. --allow-live permits provider calls;
--trust-workspace consents to Workspace Trust for each generated trial
directory. The adapter uses the existing stdio RPC bridge and standard
preset. It never approves tool requests or continues parked runs.
Optional skill behavior cohort¶
Run self-test --suite skills first; it only validates grader fixtures. A live
skill cohort is a separate, manually prepared experiment: prepare one task,
use the attempt's workspace/.agents/skills, discovered automatically.
Enable the intended skill there before starting Ru. Use the same provider,
model, effort, protocol, toolchain and deadlines across matched attempts, and
use fresh workspaces. For the explicit case the task prompt starts with the
normal /skill currency-rounding precision=2 chat turn. The implicit and
supporting-resource cases rely on the registered catalog and its resource
base. Negative cases expose an unrelated or manual-only skill and expect none.
The current trajectory can report end-task grader reward and general tool
dispatch counts only. skill_eval.observed_skill_selection therefore remains
unknown; do not convert expected selection labels, tool counts or a passing
task into a selection/adherence score. Provider trials require explicit
--allow-live --trust-workspace, for example:
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py run-live --suite skills --task explicit-currency-rounding --destination artifacts/skill-live-explicit --ru-executable gui-winui3\bin\Release\net10.0-windows10.0.26100.0\win-x64\publish\RuAgent.Cli.exe --allow-live --trust-workspace --tool-profile coding_v3 --report artifacts/skill-live-explicit/report.json
This command is documentation for a future controlled cohort; no live skill quality result is implied by the offline oracle.
Cold initialize/run-start RPCs get up to 30 seconds inside the task deadline; ordinary RPC calls keep their 10-second bound. Preserve startup failures with no model steps as infrastructure evidence rather than model-quality outcomes.
Use --tool-profile coding_v2|coding_v3 to pin a trial's
new Host process. A requested/actual pin mismatch fails explicitly. For matched
tool comparisons, run the same task, model, effort, protocol and deadline with
fresh destinations for each profile; compare externally graded success before
latency. The trajectory records observed model steps, tool dispatches/receipts,
approvals and completeness. Nested PTC call counts, malformed-call taxonomy and
provider-only latency are null when RPC does not expose them; outer dispatches
cannot be compared as though they were total nested tool usage. coding_v3
advertising run_code does not prove that the model used it.
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py run-live --suite terminal --task cs-single-flight-cache --attempts 3 --destination artifacts/ru-cache-live --ru-executable gui-winui3\bin\Release\net10.0-windows10.0.26100.0\win-x64\publish\RuAgent.Cli.exe --allow-live --trust-workspace --tool-profile coding_v3 --report artifacts/ru-cache-live/report.json
Each attempt starts in a fresh workspace and session. An approval/user wait
records an intervention outcome and requests cancellation. Timeout also
requests cancellation before terminating the bridge. Cancellation failure is
reported explicitly. Closing a bridge alone is not proof the Host run stopped.
History persistence waits (waiting_history, waiting_history_user, and
waiting_history_final) are acknowledged automatically by the Host. The
adapter polls through them within the same task deadline.
Existing Ru start/attach behavior and configured provider selection apply;
there is no alternate provider or execution path.
Reports retain fixture fingerprints, base commit, provider metadata, elapsed
time, available usage, outcomes and source patches. Each prediction uses
instance_id, model_name_or_path, model_patch. Unknown usage and cost stay
null. Repeated-attempt success rate is the fraction of attempts resolved; it
is not pass@k. A failed/intervened agent run cannot earn a passing reward from
a coincidentally correct final workspace.
trajectory.json records bounded safe events and run/session IDs. After a run
reaches a terminal state, the adapter fetches each tool.proposed argument
payload with the documented run.output method (at most 400 calls, within 60
seconds outside the agent deadline) and keeps only tool_id, the declared
check purpose and, for run_process/powershell, the executable's base name.
Argument text, paths, commands and results are not retained, and the safe
events still carry no payload_ref. tool_calls.outcomes_paired is true only
when every payload was read and proposal and completion counts match; outcomes
then pair in durable order, as the TUI settles tools, and
tool_calls.summary counts calls per tool, purpose=verification checks and
failed checks. Otherwise those outcomes stay null rather than guessed. The
tool_profile field records the actual run pin when the Host supplies it; older
Hosts return null. To compare coding_v2 and coding_v3, start a dedicated Host with RU_AGENT_TOOL_PROFILE set before its
new runs, use fresh identical fixture copies, and hold the runtime, model,
effort, task deadlines and toolchains fixed. Do not infer the active profile
from the bridge's environment when it attached to an already-running Host.
The
events_complete and usage_complete fields compare the received sequence
with the terminal run's authoritative cursor: true means it caught up, false
means partial evidence, and null means the cursor was unavailable. Available
token totals must not be presented as exact final usage when incomplete.
Timeouts and intervention outcomes leave completeness unknown when only a
pre-cancellation, nonterminal cursor was observed; catching up to that older
cursor does not establish complete terminal evidence.
The dated Step 5 Preview Free pilot records actual runtime fingerprints and separates candidate correctness, unattended completion, scope violations and provider/policy interruptions. The post-refactor Nemotron rerun records another nine-task native trial with an isolated Host and the existing selected model. Its 0/9 result includes three completed runs without patches; run completion alone is not task acceptance. The two dated trials use different provider/model and runtime conditions and are not a controlled comparison.
The post-terminal-fix SWE rerun repeats the three SWE tasks with matching instructions/baselines and the same Step 5 model, preset and 300-second deadlines. All three timed out, resolving 0/3; Python and TypeScript produced empty patches. One attempt per task does not establish a causal effect of the fixes.
Running live trials correctly¶
run-live starts the CLI, which resolves the Host exactly like the app: a flat
distribution uses the sibling RuBackend.exe; a repository CLI uses the
published Host under backend-dotnet/RuBackend/bin/Release/net10.0/win-x64/publish.
Use the flat publish produced by build.bat
(gui-winui3\bin\Release\net10.0-windows10.0.26100.0\win-x64\publish\RuAgent.Cli.exe);
the current build's RuBackend.exe sits beside it. A repository CLI needs a fresh
Host publish after Host/Broker changes.
The report and every trial.json record runtime.cli and runtime.host
(path, sha256, bytes, modified_utc), host_layout (flat or
repository) and host_stale. A missing Host fails with
host_executable_missing; a Host older than the CLI fails with
host_publish_stale unless --allow-stale-host is passed for a deliberately
historical comparison. A Debug DLL pointed at an old Host publish is therefore
refused instead of silently measuring old code.
The CLI still keeps its start-or-attach behavior: a Host already running for the same profile is reused, and that process may not be the file recorded above. Stop it or use a dedicated data root when the trial must prove the current build.
Set RU_REASONING_EFFORT (for example low) in the environment of the benchmark command to pick the reasoning effort for the whole trial without editing saved settings; request_options_json in the run's pinned provider selection records what was applied. Set RU_DRIVE_FAILURE_DIAGNOSTIC to a file path to capture the full exception when a run ends with runtime_drive_failed.
Held-out external-agent comparisons¶
compare imports JSONL predictions and externally produced acceptance results;
it does not call an agent or verifier. Keep task instructions, repositories and
verifiers in a private evaluation store. Publish a manifest containing only a
task ID and SHA-256 fingerprint for each task, plus a cohort pin and fresh
attempt count. Fingerprint the task bundle (instructions, baseline, verifier,
acceptance rules) with a canonical file-content hash; never put a reference
patch or solution in the manifest. A private task can span any number of files
and can use a long instruction. The comparator verifies each definition against --task-store; definition hashes
exclude the solution directory and generated build directories.
{
"schema_version": 1,
"cohort": {
"agent": "Ru",
"agent_version": "release-2026.10.24",
"model": "provider/model-id",
"effort": "high",
"protocol": "responses",
"toolchain": "windows-x64-dotnet-10-node-22.18",
"deadline_seconds": 1800
},
"attempts_per_task": 3,
"tasks": [
{"task_id": "heldout/long-migration-01", "task_sha256": "<64 lowercase hex characters>"}
]
}
Each JSONL line must include task_id, task_sha256, one-based attempt, all
cohort pins (agent, agent_version, model, effort, protocol,
toolchain, deadline_seconds), status, externally graded boolean correct,
boolean intervention, and wall_seconds. status is completed, timeout,
interrupted, infrastructure_error or failed. Optional
model_attempts, tool_attempts, input_tokens, and output_tokens are
nonnegative integers or null. Missing records remain missing; the report
lists them and marks the cohort incomplete. Duplicate, unexpected, mismatched
fingerprint or incompatible provenance records are rejected. Import an
independent record set for each product/configuration; only compare cohorts
whose task manifest, acceptance verifier, toolchain and deadline match. This
summary does not imply cross-product comparability when configurations differ.
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py compare --manifest heldout-manifest.json --predictions predictions.jsonl --task-store D:\private-evaluation --report comparison.json
Correctness includes any externally accepted prediction, divided by expected trials;
observed_correctness_rate separately describes received trials. Autonomous completion
counts only correct, completed, non-intervened trials, divided by the full
expected matrix, so missing trials cannot inflate the rate. Timeout and
infrastructure interruption stay separate. Latency p50/p95 includes every
received trial. Usage totals remain null unless present for every received
record; the report's completeness flags make partial telemetry visible. No
external-agent result exists until prediction files and verifier outcomes are
actually collected.
Reproducible native and GUI cohorts¶
Schema-v2 cohorts pin the source commit, source-tree SHA-256 and artifact manifest SHA-256 as well as agent version, model, effort, protocol, tool profile, toolchain and deadline. Hash the canonical published payload manifest (sorted relative paths and file digests), since a .NET apphost alone may remain unchanged across builds. Each task entry pins its task definition, fixture and effective prompt SHA-256; the prompt and reference solution are never copied into the manifest. Create a manifest from an explicit pin file and selected fixtures:
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py make-cohort --suite everyday --attempts 2 --cohort-pins artifacts\cohort-pins.json --report artifacts\cohort-manifest.json
The pin file contains those exact schema-v2 cohort fields. Candidate and
baseline runs must use separate manifests and fresh workspaces. For a
matched-model comparison (--track matched-model, alias harness), keep task,
fixture and effective-prompt hashes, model, effort, protocol, tool profile,
toolchain, deadline and attempt matrix equal. If provider identity is included
in cohort pins, it must match too;
the source commit, source-tree and artifact hashes identify the differing
implementation under evaluation, including uncommitted source changes.
compare-cohorts rejects pin or fixture mismatches instead of silently merging
them. The native-product track (alias product) allows differences in agent,
provider, model, effort, protocol and tool profile, and lists which fields
differ. Both tracks require the same task, fixture, prompt, toolchain, deadline
and attempt matrix.
An incomplete observation matrix is reported with comparable: false and no
paired outcome summary; it is not treated as a matched comparison.
Each observation JSONL line repeats all cohort pins and task/prompt hashes,
then records attempt, optional per-task base_commit, status, trusted
grader_accepted (boolean or null when grading evidence is unavailable),
unattended_completed, intervention, and wall_seconds.
Statuses distinguish parked, interrupted, infrastructure_error, timeout
and failed from completed. CLI, GUI and external adapters can all write the
same format. The Python API cohort.record_attempt creates a validated row;
cohort.record_ru_attempt adapts the existing native RPC result and trusted
grader result, rejecting mismatched observed model, protocol or tool profile.
GUI results can be imported with cohort.load_observations.
trajectory and usage fields carry counts only when authoritative; missing or
partial event streams remain null in aggregate totals. Current RPC records can
count model steps, outer tool dispatches/receipts and complete-stream
compactions. Multi-step fixture acceptance does not prove behavior across a
forced compaction or restart, so no continuity score is inferred. Nested tool
errors, repeated reads, recovery behavior, cached
tokens, search usage, cost and skill selection remain unknown unless a future
adapter supplies direct evidence. A parked attempt may pass the external
grader, but it does not count as unattended completion. Aggregate and compare:
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py aggregate-cohort --manifest artifacts\cohort-manifest.json --predictions artifacts\observations.jsonl --report artifacts\cohort-report.json
.venv\Scripts\python.exe tools/coding-benchmark/benchmark.py compare-cohorts --cohorts artifacts\cohorts.json --track matched-model --report artifacts\comparison.json
For complete schema-v2 comparisons, paired_outcomes compares grader
acceptance and unattended completion separately for each paired attempt. It
also counts correct-but-not-completed attempts, so an interrupted correct patch
does not become an unattended success. Unknown usage remains null in each
aggregate and is never filled from a paired cohort.
The aggregate reports grader acceptance (null until all expected attempts
have authoritative grader outcomes), unattended completion and
unattended accepted completion separately. It also lists missing attempts,
per-status counts, latency and completeness for trajectory/usage totals. An
unknown cost or skill-selection result never becomes zero or a passing score.
For a schema-v1 held-out comparison, compare-cohorts --cohorts cohorts.json --track harness
--task-store D:\private-evaluation --report comparison.json reads an array of
{"label":"Ru","manifest":"ru.json","predictions":"ru.jsonl"} entries, resolving
paths relative to cohorts.json. The schema-v1 harness track requires identical model,
effort, protocol, toolchain, deadline, task hashes and trial matrix. The product
track permits different model/effort/protocol configurations and labels that
limitation; tasks, toolchain and deadline must still match.
Private tasks can use the existing task format and trusted grading workflow:
prepare --suite swe --task-root D:\private-evaluation --task task-id --destination
D:\trials\ru-01, then run the selected agent only in the emitted workspace with
the emitted instruction, and grade --suite swe --task-root D:\private-evaluation
--task task-id --workspace D:\trials\ru-01\workspace. Keep the private definition,
verifier and reference solution outside the agent workspace. Collect process
status, intervention, duration and actual usage independently; import the trusted
grader's acceptance as correct. The importer does not attest self-reported
outcomes or run competitors automatically. Long multi-file task definitions use
the same explicit editable-source constraints and existing verifier limits.
Environment and comparison limits¶
The single-agent readiness diagnostic (2026-10-13, retained in git history) records a ten-task Standard baseline, separates candidate correctness from unattended completion, and documents repairs to effort metadata and cancelled trial completeness. The Everyday evaluation (2026-10-13, retained in git history) uses three fresh attempts per focused task with the current model, separately records infrastructure interruptions and keeps matched runtime conditions.
Terminal tasks follow Harbor's instruction.md, task.toml, environment/,
tests/test.sh, solution/solve.sh layout. Docker build contexts contain only
starter workspaces. Verifiers write /logs/verifier/reward.txt and
groups.json; test and solution files are supplied separately. Images have
versioned tags but are not pinned by digest. Containers are configured with
network_mode = "no-network".
The delivered runner labels its execution native-local. It runs trusted candidate code on the current machine, outside an OS sandbox. Keeping grader and reference files out of the candidate workspace is evaluation separation, not protection from a program that reads arbitrary machine files or spoofs grader output. Native mode does not enforce the container network setting. Use controlled machines and separate, equally configured trials for agent comparisons. Ru's Windows-only native executable has no Harbor container adapter in this delivery. Docker/Harbor runs and real-provider quality are separate acceptance checks.
The format references are Harbor task layout, Harbor configuration, SWE-bench dataset fields and SWE-bench evaluation. When comparing agents, record their exact executable/model/protocol, policy, toolchain, OS, attempts and corpus fingerprint. Do not compare these custom native results directly with published official benchmark percentages.