Reading your results
BenchKit is at its most accurate when you compare two runs from the same machine. Run it, change one thing, run it again, and the difference between the two is what that change was worth. Everything else on the machine stayed the same. Nothing else can explain the gap.
Comparing your machine against somebody else's in the gallery works too, and it is the reason the gallery exists. Just hold the comparison more loosely, because their host, their tuning, and their neighbours on that hardware are all different from yours.
Known limits
Every benchmark tool has limits. The ones below apply to BenchKit, and each comes with what to do about it.
The load generator runs on the same machine
By default oha runs on the machine it is testing, alongside the application and the database. All three compete for the same CPU. Keeping everything on one box is what makes an A and B comparison clean, because the request path is identical between your two runs. It also means the absolute requests per second is lower than the machine could serve if the traffic came from somewhere else.
When you want the absolute number, open Test from your own machine in the app. It gives you the same oha commands to copy, pointed at your server. Drive the load from your own laptop or from another server, and the generator comes off the box entirely. See Testing from your own machine.
PHP-FPM's pool size caps concurrency
Under PHP-FPM a request holds a worker for its whole duration, including any time it spends waiting. The pool size therefore sets a hard ceiling on concurrency that no amount of CPU can raise, and the ceiling is most obvious on the route that waits.
The shipped pool size is fixed rather than sized to your hardware. A large machine can end up measuring a fraction of itself. BenchKit works out how many of your cores could have been busy and tells you on the result when workers were the limit, along with the environment variable to raise. Raise it and run again before you read anything into an FPM against worker mode comparison.
The built-in test measures throughput, not tail latency
The load test holds a fixed number of connections open and measures how much the server can get through. When a fixed-connection client meets a server that slows down, the client waits rather than sending more. The reported p95 and p99 then come out better than reality. Gil Tene named this coordinated omission.
BenchKit labels its own tail percentiles as indicative for that reason. If you care about latency targets, use the corrected command in Test from your own machine, which drives a constant request rate and passes --latency-correction to oha. Find your maximum throughput with a normal run first, then measure latency at a rate below it. See Testing from your own machine.
SQLite is the default database
SQLite runs inside the PHP process. It has no connection cost and none of the indexing and contention behavior of a database server, and it is there to let BenchKit run with no setup at all.
If your application talks to MySQL or Postgres, point BenchKit at one. See Adding a real database. Every result records which engine produced it.
Comparing two runs fairly
Change one thing at a time, and keep everything else identical. When you compare image variations, tune both sides before you decide, because a default FPM pool against a default FrankenPHP tells you about the defaults rather than about the two servers.
One connection count is one point on a curve. A server that wins at low concurrency can lose at high concurrency. Run the comparison at more than one connection count before you decide.