Local benchmarks look hilariously official for something I ran on a laptop next to a browser with too many tabs open. One command is enough to produce a very serious table:
wrk -t12 -c400 -d30s http://localhost:3000/pingThe output makes me want to say something confident, then I unplug the charger, run it again, and watch the number move. Sometimes the difference is small. Sometimes it is big enough to make the first comparison look a bit silly. The benchmark happened on my desk while macOS was doing whatever macOS does in the background, Chrome was open, music may have been playing, and Spotlight probably woke up at the worst possible moment. Very controlled. Extremely scientific.
Laptops care about power and heat in ways a benchmark table does not mention.
The CPU may boost differently on battery, the fans may already be busy, and the
machine may still be warm from the previous run. If framework A gets 90,400 req/sec and framework B gets 89,900 req/sec, I have learned almost nothing.
The charger, room temperature, or run order could easily have cast the deciding
vote. A large difference that survives repeated runs gets my attention; a tiny
difference from one run mostly confirms that computers are annoying in completely
normal ways.
Run order is another easy way to fool myself. The first process starts with cold caches and a runtime that may still be settling. The second run improves, then a few runs later the laptop is warm enough to slow down. I change one line and test again as if the machine has remained physically identical through all of this. Repeated runs at least make the wobble visible:
go test -bench=. -benchmem -count=5Five runs around 91k, 93k, 92k, 91k, and 92k tell me much more
than a single 93,482.11 req/sec. The precise number has better posture, but it
can still describe one lucky thirty-second window. I care about the cluster and
the spread, especially when the difference I am comparing is smaller than the
spread between my own runs.
The wrk -t12 -c400 -d30s command also carries a bunch of opinions. Duration
changes how much startup effects matter, concurrency can move the bottleneck, and
running the client beside the server makes them compete for the same CPU. When I
publish numbers, I need to publish the command, machine, run count, power mode,
and any weird thing I noticed. That feels like boring diary work until somebody
tries to reproduce the test, at which point the boring details become the only
useful part.
I also try to resist calling a slow outlier “noise” because it ruined a nice average. Garbage collection might have started, a lock might have got hot, or one allocation change may have shown up only under pressure. Latency makes this even more obvious. Users never experience an average request; they experience the one request they happened to send, including the ugly p99 hiding behind a cheerful mean.
I still love local benchmarks because they are cheap and very good at answering whether something is obviously bad. I just trust them more when the result comes with a little account of where it ran: charger or battery, warm machine or cold, one run or five, clean terminal or my usual chaos. The extra context makes the table less pretty and the conclusion much harder to accidentally invent.