Reading your results

BenchKit is at its most accurate when you compare two runs from the same machine. Run it, change one thing, run it again, and the difference between the two is what that change was worth. Everything else on the machine stayed the same. Nothing else can explain the gap.

Comparing your machine against somebody else's in the gallery works too, and it is the reason the gallery exists. Just hold the comparison more loosely, because their host, their tuning, and their neighbors on that hardware are all different from yours.

Known limits

Every benchmark tool has limits. The ones below apply to BenchKit, and each comes with what to do about it.

A self-test generates its load on the same machine

In a self-test, oha runs on the machine it is testing, alongside the application and the database. All three compete for the same CPU. Keeping everything on one box is what makes an A and B comparison clean, because the request path is identical between your two runs. It also means the absolute requests per second is lower than the machine could serve if the traffic came from somewhere else — which is why it is the opt-in, not the default.

The default is the External load test: when you start a run, BenchKit shows you one command to run on a second machine. That machine handshakes, drives the standard measurement the moment the run starts, and its results are recorded into the run like any other. Every result records which mode produced it, and the gallery never compares the two on one axis.

Where that second machine sits matters as much as having one, which is the next limit.

A distant load generator measures the network

oha is closed-loop: each connection sends its next request only when the previous reply arrives. So the concurrency needed to keep a server's workers busy grows with the distance to the machine driving the load.

That is not a small effect. A server answering in a fraction of a millisecond behind a 13ms round trip spends 98% of every connection's life in transit — reaching the same pool from there takes roughly fifty times the connections it would from next door. BenchKit measures the round trip at the handshake, tells you the most it could measure from there before you start, sizes the sweep accordingly, and flags the result when even that was not enough. Use a machine in the same datacenter; a laptop across the internet answers a different question. Testing from another machine works through the trade-off with numbers.

PHP-FPM's pool size caps concurrency

Under PHP-FPM a request holds a worker for its whole duration, including any time it spends waiting. The pool size therefore sets a hard ceiling on concurrency that no amount of CPU can raise.

This is not hidden — it is drawn. The /bench/io curve on your result climbs in a straight line and then goes flat, and the line it flattens onto is workers ÷ delay, worked out from arithmetic before the test ran. The height of that line is your pool, not the concurrency it arrives at: a connection holds a worker only while the server holds the request, so a generator further away needs proportionally more connections to fill the same pool. The shipped pool size is fixed rather than sized to your hardware, so on a large machine the curve flattens well below what the machine could do, and Common tuning strategies covers what to do about it.

SQLite is the default database

SQLite runs inside the PHP process. It has no connection cost and none of the indexing and contention behavior of a database server, and it is there to let BenchKit run with no setup at all.

If your application talks to MySQL or Postgres, point BenchKit at one. See Adding a real database. Every result records which engine produced it.

What a clean run is

A run is marked clean in the gallery when nothing about how it was measured undermines the numbers:

• the load came from a second machine, not from the server itself
• that machine kept up, so the figure describes the server rather than the path to it
• throughput was pushed until it stopped speeding up, so the figure is a maximum rather than a floor
• every request was answered
• the application was set up the way you would deploy it — cached config, routes and events, OPcache on, debug off
• the database was on a disk rather than in memory

It says nothing about whether the host is fast. A modest VPS measured carefully is clean; a fast one measured through a laptop on hotel wifi is not. When a run misses, the gallery says which of these it missed, so you can see what to change before running again.

Filter to clean runs when you want numbers you can put side by side.

Comparing two runs fairly

Change one thing at a time, and keep everything else identical. When you compare image variations, tune both sides before you decide, because a default FPM pool against a default FrankenPHP tells you about the defaults rather than about the two servers.

Read the curve, not just the headline. A server that wins at low concurrency can lose at high concurrency, and the shape of the two lines tells you which one you are looking at. BenchKit measures several concurrency levels on every run so you do not have to run it more than once to find out.