Lab / / failure

Struck as UNMATCHED, then falsified at 1.24× slower

A "1.29× faster than Meta" claim was produced from an unmatched comparison, quoted as a prize, struck, and then inverted to 1.24× slower on a matched protocol.

Measurement methodology, Negative results, BenchmarkingProteus
Machine
UNKNOWN — not stated
Commit
UNKNOWN
Original claim
1.29× faster than Meta
Matched-protocol result
1.24× slower

This is the most instructive benchmark failure I have, and it is instructive because the sequence is documented rather than narrated.

  • A ratio was produced from a comparison whose sides were not matched.
  • It was quoted as a prize.
  • It was struck within the source material.
  • On a matched protocol it inverted: 1.24× slower.

The doctrine document names both failure modes present here — *"arithmetic standing in for a measurement"* and *"a comparison quoted as a prize"*. This is the second one.

Direction matters more here than anywhere

The original claim was a speedup: candidate ÷ baseline, higher is better, and 1.29× meant the candidate was ahead. The matched-protocol result is 1.24× slower, which is the same ratio convention reading below 1.00. Anyone who takes the two numbers as comparable without the words attached will read the inversion backwards.

I am not publishing it as a ratio. It is published as a cautionary entry, which is what the brief asked for — record rather than reconcile.

StepFigureProtocol
original claim1.29× fasterunmatched — struck
after matching1.24× slowermatched protocol
Direction: stated in words, because the two figures were produced under different protocols and the sign is the whole finding.