Models guess optimizations well and prove them badly, so gotorque does the proving

gotorque is a harness I wrote for making Go CLIs faster with a language model in the loop, and the model is the least interesting part of it. Each cycle a model proposes one small patch; deterministic Go applies it in a throwaway worktree, builds it, runs the target’s own test suite, measures old and new binaries in interleaved pairs, and a 407-line policy that no model output can reach returns accepted, rejected or inconclusive. It has no opinion about a program’s best shape, only about the one patch in front of it. A good guess looks like this: on gron, the optimizer’s patch to the top-ranked target was accepted at -15.1% wall time, -22.2% on flatten-users (ADR 0013). The README’s summary is the one I’d keep: “Models are good at guessing optimizations and bad at proving them. This harness keeps the guessing and mechanizes the proof.” This post is about the proof half, and about the ways the evidence was wrong before the code to catch it existed.

What a model is allowed to do

I gave the model five roles, each a single turn that answers in JSON: a coordinator picks the next experiment, an explorer proposes workloads, an analyst reads the profile, an optimizer writes one patch, a reviewer argues against it. The target reaches them as a manifest, a command and its arguments:

target.command   a fixed prefix, empty for a root CLI
args             one array per workload
stdin            optional

Nothing the manifest can’t describe gets measured, so this is a harness for Go CLIs and nothing else. The five prompts fit in one 199-line file, internal/agents/agents.go. The rest of the repository is about seventeen thousand lines of non-test Go (wc -l at commit 925251f), and most of it sits between a model’s answer and a verdict.

The reviewer’s instruction says what the design says: “Your proceed recommendation is advisory; policy owns the verdict — your job is to surface anything the metrics would hide.” apply_policy decodes the reviewer’s answer and calls policy.Evaluate with the behaviour flags and the metric comparisons. The concerns are copied into the candidate’s record for the next optimizer call, and that is as far as they go. The evidence a verdict is judged on is the struct policy.Evidence, at internal/policy/policy.go:34, and it has exactly five fields, few enough that I can list them here:

BehaviorMatches
FailureSummary
SafetyChecksPassed
RepresentativeEvidence
Comparisons

There is no field for a concern, and nothing in internal/policy mentions one. The package touches no filesystem, process or network. The repo states the invariant as agents advise, code decides; the rest of this post is what it costs to keep that true.

The patch can’t touch what judges it

The first way a patch could pass is to edit the test. The optimizer is shown which tests its last attempt broke, so rewriting the assertion is the shortcut it would be steered toward. These paths are off-limits to it:

*_test.go   testdata/   go.mod   go.sum   go.work   default.pgo   vendor/   .git

I check them twice. Once on every name in the diff headers, as written and as -p1 strips them. Then again on what Git reports changed, after the patch is applied in a fresh worktree at the base revision.

The second pass exists because the first was fooled. Handed --- a/go.mod and +++ b/other.go, GNU patch edits go.mod. It also follows a tracked symlink into testdata/ that git apply refuses, which is why the worktree runs git apply --check first and falls back to patch --fuzz=5. A protected path, or a new symlink in Git’s list, rejects the candidate before anything is built.

A third check holds the patch to the remedy that was asked for: a buffered writer for unbuffered I/O, a preallocation call for a missing one. It is substring matching on added lines, so a patch that merely mentions the mechanism still reaches the build, the tests and the measurement (ADR 0017).

The test gate runs go test -json on the patched tree and used to demand a clean pass. On Go 1.27, gojq’s suite already fails on the unpatched revision. The release notes say encoding/json is now backed by the v2 implementation and that “the exact text of error messages may differ”; the pair that broke gojq, recorded in ADR 0005, is “unexpected end of JSON input” where a test expects “unexpected EOF”. Every candidate was rejected for a failure no patch could have caused. The gate now runs the suite once on the baseline, records what fails and what passes, and charges a candidate only for new failures. A test the patch disables can’t fail, so a baseline-passing test that is skipped or never runs also rejects, by name.

Twenty-five pairs, and the p-value that was never there

Baseline and candidate run in alternating pairs, baseline first, so CPU contention lands on both sides. Stdout must match byte for byte, or by sorted-lines digest when two runs of the baseline against itself already differ, and exit codes must match.

The pair count is 25 per workload, and it used to be seven. Seven can’t see a 3% effect through the 6–8% per-run spread of a 10–25 ms CLI run: a gron candidate recorded as inconclusive at -3.51% measured -3.75% at p<1e-4 over thirty pairs on the same binaries (ADR 0004). Support has two sources, and neither uses fewer than four samples a side: a Welch t-test that detects a difference, or a 95% confidence interval on the relative delta lying entirely below the 2% guardrail limit, which lets a flat reading count as evidence when the data rules out a material regression:

func metricSupport(a, b []float64, baselineMean, deltaPercent float64) bool {
  if len(a) < 4 || len(b) < 4 {
    return false
  }
  // ...
  se := math.Sqrt(va/float64(len(a)) + vb/float64(len(b)))
  if se == 0 {
    return ma == mb // exact identical measurements
  }
  t := math.Abs(ma-mb) / se
  if t > 2.2 {
    return true
  }
  if baselineMean <= 0 || deltaPercent >= 2.0 {
    return false
  }
  ciUpper := ((mb - ma) + 2.2*se) / baselineMean * 100
  return ciUpper < 2.0
}

When benchstat is installed, a p-value below 0.05 grants support and an insignificant one withdraws it. Support is therefore wider than significance: significance is the t-test or the p-value alone, and the interval path supports a reading that is not significant.

The benchstat lane is the wrong turn I’d keep in: for a long time it had never contributed a single p-value. The sample files were one bare number per line, which is not a benchmark report, so benchstat skipped every line, exited 0 and printed nothing, and every verdict fell back to the t-test over means without saying so. A test now runs the real binary over fresh samples and fails if it reads nothing. A supported reading now shows the pair count and the p-value the lane produced (ADR 0004):

-6.70% (p=0.000 n=25)

The arithmetic worked against real wins in two more places. Concatenating workloads’ samples into one pool inflated the spread with between-workload variance: a 20.8% win on gron’s large workload produced a pooled t of 1.06, against 10.69 on that workload alone, so samples are now folded per repetition. And judging the pooled figure alone let a 41-byte seed, whose run is mostly process startup, drag a supported 3.35% win (p=0.006) to a pooled -2.53% against a 3% bar (ADR 0002). A verdict may now rest on any representative workload, and it names the one it rests on. That 3% is a floor on targets as much as on patches. An accepted patch has cleared it on a representative workload without the 2% guardrail rejecting it, and scc, whose run is dominated by its own startup, is a poor target for this.

A regression has to prove itself too

Acceptance always required support, and rejection didn’t: any eligible reading whose mean moved more than 2% the wrong way ended the candidate. Short CLI runs carry bursts of slow outliers that move a mean while the median stays put, so the rule was tested A/A: the same binary against itself, which nothing should reject. Over 30 trials the point-estimate rule rejected 12, on readings from +2.00% to +9.10% (ADR 0016). Those trials ran on darwin/arm64; the ADRs record no machine for any of these campaigns. It cost real wins too: a gron candidate that improved flatten-users by 20.41% with support was rejected because small-doc read +4.01%, which benchstat did not call significant.

Now a reading over the limit rejects only when it is significant. Support could not be the test, because it is also granted to a flat reading whose interval rules out a 2% regression, so significance became a second field on every comparison: benchstat’s p below 0.05 when it ran, otherwise |t| above 2.2. When a reading is over the limit without significance, every representative workload is measured over a second series of 25 pairs and every comparison is derived again from the 50.

Re-run against the same 30 A/A trials, the rule rejected one. The confirmation series brought ten of the twelve back toward zero, a +2.04% reading resolving to -0.01% and a +2.79% one to +1.22%, which is the shape you want from a second look at noise. The rejection left was a small-doc reading of +7.54% that benchstat itself called significant: the false positive a 0.05 test accepts by design, and I would rather the harness keep it than tune the threshold until it disappears. No trial was accepted under either rule, which is the half of an A/A control that is easy to forget — a regression rule that rejects nothing looks perfect right up until you notice it is waving everything through. A reading that stays over the limit without significance does not reject, and the verdict says so in words; that is the rule’s cost, since a real regression smaller than 50 pairs can resolve passes with its reading named. A manifest with statistical_support_required: false keeps the point estimate as the whole rule.

The classifier is advice too

There is a classifier in this project, opt-in. With --analyst jev, the analyst model is replaced by Jev, a classifier from TypeSafe that the harness reaches through the Vercel AI Gateway; it answers typed yes/no questions about a state with calibrated probabilities and generates no text. For each of up to twelve hot functions discovery found, code sends the whole function and seven questions, one per cause: avoidable allocation, unbuffered I/O, string building, a missing fast path, superlinear work, missing preallocation, repeated work. Each answer is compared against Jev’s usual answer to that question over 93 real single-function performance fixes, and a cause is flagged only when it clears two gates: at least half a standard deviation above that usual answer, and a probability of at least 0.5, Jev saying yes outright (ADR 0012, ADR 0025). --reviewer jev asks eight behaviour-hazard questions of a diff at a stricter gate: two standard deviations above the usual answer, and the same probability floor of 0.5.

It gets the same trust as the language models, which is none at the verdict. Its flags choose what the optimizer attacks, in an order code computes, and code overrules it where it knows more. It flagged gojq’s (*encoder).writeByte for unbuffered I/O at +2.9 sd. Every write in that function goes to a bytes.Buffer field, so a veto drops the flag: an unbuffered-I/O flag needs a read or write that reaches a real file or stream (ADR 0025, which generalises and replaces the in-memory override of ADR 0019). Its hazard concerns are recorded for the next cycle and, like the model reviewer’s, never reach apply_policy. The case that tests the rule: the gron patches the harness accepted wrap stdout in a bufio.Writer and defer the flush, discarding its error, and the Jev reviewer raises exactly that concern on the accepted patch, at probability 0.96 (ADR 0014). It was accepted anyway, because the behaviour gate and the measurement both passed. The design’s position is that a model output ending a candidate is the failure the harness exists to prevent.

The guessing half had a hole the proof half could not see, and what closed it was losing access to the provider. Jev used to arrive through a gateway that served an unversioned alias and named no version in its answers, so a release could move every answer the baselines describe with nothing in the harness noticing. Then that account stopped reaching Jev at all, and the endpoint that replaced it takes an exact build id, so the client now refuses any answer whose model or provider is not the one it asked for: a hard failure with no override, because nothing recorded is valid for a build it was never measured against (ADR 0030).

Re-measuring the baselines against that pinned build gave me the figure I would keep from the whole project. Across 186 functions every question’s mean moved less than two points, a paired mean |Δ| of 0.010. The top-ranked cause changed for 9 of those functions and the flagged cause set for 21. A statistic that holds to a hundredth while 11.3% of the decisions it feeds flip is this post’s own argument arriving one layer up, in the half of the harness that was never supposed to need it, and I saw it only because a migration forced a measurement nobody had asked for. What is left is bounded rather than closed: the canary guards seven cause questions, so a release that moves only the hazard or fix-kind answers behind the same id still passes preflight unseen (ADR 0030).