Benchmark methodology

Published measurements belong to their stated test date, workload and environment. The public evidence summaries preserve functional outcomes and material failures; internal operating records remain private. A revised summary has its own digest and does not inherit a signature over its source.

How a gate runs

Each test states a falsifiable acceptance criterion and reports its observed result. Negative controls check that it detects a defeated property. Infrastructure assertions and model/task results are separate: a passing test can include unsuccessful task attempts. An aborted trial is reported as not held, never as a pass. Public summaries identify the test’s scope and limitations without supplying the private harness or operating procedure.

The three measurement families

A number on this site belongs to exactly one measurement family, and the family is the environment its row names on the availability matrix. Nothing generalizes across families: a bench number stays a bench number, a canary result is one run against the deployed service, and a customer-trial result is settled with a customer or it is not published.

Bench

Two labs carry the bench family. They keep separate pin sets, and a checkpoint does not travel between them (Limits, section 3).

The aarch64 evidence bench is a private Apple Silicon laboratory host. The x86_64 fleet series uses a separate laboratory host and supported runtime. The founding tests ran on both architectures; later fleet measurements were recorded on x86_64 (az-0). Restoration requires a matching supported runtime and architecture, as described in the public version scope.

The laboratory is not the customer installation path. The quickstart installs the released client and starts a hosted session; the evidence bench page explains the historical laboratory scope.

Bench series published on this site: the soak and its dedup ratio, the resume-latency range, the Wilson interval, and the fleet resume percentiles below.

Production canary

production-standing rows are gates that ran against the deployed control plane and gateway rather than a lab, on a real account: cross-provider continuation through the production gateway (pv-3-prod), the hosted attach relay, the paid fork, the browser console, the credits ledger and its statements, the key path for a customer’s own provider key, and the deletion paths for checkpoints, credentials and ledger rows.

These are existence-and-correctness results on the deployed service, not throughput or latency measurements. Every one of them carries the measurement state unavailable on the availability matrix, and this site publishes no service-level number, uptime figure or recovery-time objective from them.

Customer trial

No settled customer trial result is published. The only measurement whose environment includes a machine we do not operate is the install path: bench-arm64+customer-env, the public client gate (record), which timed install and first session on the released client and refused a tampered mirror. Its stated limitation travels with it: a true fresh macOS VM is not automatable on the bench, and no Windows binary exists yet.

When a customer trial does settle, it is reported under its own template with the agreed baseline, the allocation policy and the complete costs of every attempt, and it is never folded into the bench series.

The soak, on the aarch64 bench

The durability and dedup numbers come from one long run, recorded in the phase-6 record: a soak of about seven hours (24,932 s elapsed) with a five-minute checkpoint cadence, judged afterward by gate 6 against clauses pre-registered before the run.

• 82 of 82 scheduled checkpoints present, zero failed, zero skipped, zero cadence gaps.
• 25 of 25 fresh kill-and-resume cycles succeeded, each verified by a witness nonce: same nonce with a larger count proves the same memory woke; a new nonce would mean a mere reboot.
• Storage across the 82 checkpoints: 4,233 MiB actual against 80,952 MiB naive, which is the 19.1x dedup figure. One workload, one soak; your ratio will differ.
• Daemon memory: +0 MiB peak RSS growth on both daemons over the run.
• A 15.2-hour-old preserved checkpoint was woken live in 14.3 s.

The resume-latency range quoted sitewide, 13.7 to 25.2 s for machines running an agent on the bench, spans three phase records: phase-7 (13.7 s, aider), phase-6 (15.3 to 15.8 s, Claude Code), and phase-4 (25.2 s, the full demo where the agent and its daemon are killed together). All three ran on bench-arm64. The recovery scope and measurements are on the architecture page.

The fleet series, on x86_64

Two series were measured on fleet-x86. They are bench-family numbers about a lab host, not measurements of the deployed service.

• Resume percentiles (the fleet resume gate, 2026-09-05; record). A real 2 GiB agent guest on the fleet host, 20 interleaved resumes per condition: warm p95 25.1 s (median 18.1 s) with the chunks already local, cold p95 47.5 s (median 33.4 s) with the chunks fetched during the resume. These measurements do not establish sub-second recovery (rs-1).
• Failover downtime (the failover gate’s second run, 2026-09-06; record). 47 s from a real deallocation of the first fleet host to reconciliation on the replacement, one run, same region, shared durable store. Recorded as data, never as a pass condition, and it is not a recovery-time objective (fo-1b).

Why the interval, not the point

25 successes in 25 trials is 100% observed, and 100% is not the claim, because 25 trials cannot carry it. The Wilson score interval answers the honest question: given 25 of 25, what success rate is supportable at 95% confidence? The answer is at least 86.7%. So the published claim is the interval, [86.7%, 100%], with its denominator named: fresh cycles run by that gate on bench-arm64. The claim moves only when the interval does, which means more trials, not more adjectives (gate 6, Limits, section 7).

Known failures

Measurement systems fail, and a methodology page that omits its failures is marketing. The record so far:

• The meter defect (found in engine release v0.1.1, fixed in v0.1.2). The gateway was found writing its billing meter into a database table it no longer had, silently: a whole demo served ten calls and the billing ledger recorded none, with zero errors logged. This undermined the evidence behind the billed-at-most-once claim. The release records report fixes to gateway accounting in v0.1.2 and session recording in v0.1.3. Those are historical fixes, not independent proof of the current deployment. The current status of every finding is in the findings record.
• The connection leak. The first seven-hour soak found a leak that killed any session past its ninth checkpoint, invisible to all 25 kill-cycles because kill-cycles reset the very counters long-lived sessions accumulate. The soak exists as a method because point trials structurally cannot see this class of defect (phase-6).
• The parser that could not read its own evidence. Gate 6 v1 produced six false alarms and one false comfort from a single hand-written parser defect while the system under test was healthy. Test correctness is itself an assurance requirement, separate from the system being tested.
• The aider 3-of-4. Across four gate 7 runs the gateway held an interrupted call’s response 4 of 4 times; the client walked back through the held door 3 of 4. Reported as data, never promised (phase-7).
• The sealed-reproduction flake. This is the bench reproduction document that gate 7 extracts and runs, not the hosted quickstart on this site. A stranger’s machine completed it from a clean clone 3 times of 4; the fourth was a host VM-creation flake, recorded as an abort rather than a falsification, because a trial that could not be held is neither green nor red (phase-7).

Raw records

The public evidence index provides reviewed summaries and their exact-byte derivatives. Original internal transcripts are retained privately. A public summary preserves the relevant outcome, failure, date and scope; its hash identifies that summary’s bytes.

• Bench records: phase-4, phase-6 and phase-7.
• Fleet records: phase-az-0, phase-rs1, phase-fo-1b.
• Production check: phase-pv-3-prod.
• Client installation: phase-cli-1.
• The claims, limitations and public version scope.