Methodology
A BenchKit run measures four things about a machine, using an established tool for each one rather than something we wrote ourselves.
| Stage | Tool | What it measures |
|---|---|---|
| Hardware | YABS | CPU and disk, through Geekbench and fio |
| Network | cfspeedtest | Latency and bandwidth to Cloudflare |
| Web server load | oha | Max throughput and response time, per route |
| PHP and database | phpbench | Timing for individual PHP and database operations |
The web server load test
The load test drives four routes in your own application, one after another. Each one is served without session and CSRF middleware. What you measure is the framework request path rather than session bookkeeping.
| Route | What it represents |
|---|---|
/bench/static | A plain string. The ceiling for the framework, with no database involved |
/bench/json | A JSON payload built in PHP, like an API response |
/bench/db-read | A small query, like a page that reads from the database |
/bench/io | A request that waits, described below |
Before each route is measured, BenchKit runs a short warmup pass and throws the numbers away. A cold OPcache, an unprimed route cache, or a worker that has just booted would otherwise land in your results.
Two questions, two measurements
"How much can this server take?" and "how fast does a page feel?" are different questions, and one test cannot answer both. Push a server until it stops going faster and every request is queuing, so the response times you record describe the queue rather than the server. Send traffic gently enough that nothing queues and you never find out where the server tops out.
So each route is measured three times.
One connection
A single request at a time, with nothing queued. The response time here is the server's own work plus the round trip to wherever the load is coming from — and because the round trip was measured before the run started, subtracting it gives how long the server actually takes. That number decides the rest.
A sweep
The same route at rising concurrency, until throughput stops improving. That gives the maximum, the concurrency it took to reach it, and a curve showing how the server behaves in between.
The levels are not a setting. They are worked out from your machine — its cores, its worker count, and how far away the load is coming from. A fixed connection count cannot be right for both a one-core box and a thirty-two-core one, and behind a slow link a server answering in a fraction of a millisecond needs far more connections to reach the same worker pool than it would from next door.
A steady rate
Finally, requests at about 70% of the maximum the sweep found, held at that rate rather than as fast as the server will take them. This is where the response times on your result come from. Staying below the maximum is the point: at saturation everything queues, and the percentiles describe the backlog.
Because the rate is held rather than chased, this last pass is not subject to coordinated omission — the effect where a client that waits for a slow server stops sending, and quietly under-reports its own tail.
Where the load comes from
The load can be generated in one of two modes:
- External load test — the default. A second machine you pair runs the same three measurements and uploads the results into the run. This is the honest absolute number: when you start the run, BenchKit shows one command to run on the other machine, measures the round trip when it connects, and tells you the most it could measure from there. Pair a machine near the server. This stage runs first, so the part that needs you is over in a minute. See Testing from another machine for how to pick and set up that machine.
- Self-test — the server drives its own load. Zero setup, and the right instrument for comparing configurations on the same machine, but the generator shares the CPU with what it measures, so absolute throughput reads as a floor. Chosen in the settings, or from the pairing dialog when you have no second machine handy.
Which mode produced a result travels with the result — see Reading your results.
Why one route just waits
The /bench/io route sleeps for a delay you choose, standing in for a single outbound call such as a database over a socket, a cache, or an HTTP API.
It exists because framework overhead is a fixed cost per request. On a route that does nothing else, that cost is most of the request. On a route that spends most of its time waiting for something else to answer, the same fixed cost barely registers. Raising the delay and watching what happens to the gap between two runs is how you find out which of those your own application looks like.
The delay is simulated rather than a real network call. A real dependency would add variance BenchKit cannot control, and two people running the same benchmark would not be measuring the same thing.
One standard method
Every run measures the same way: probe, sweep to the top, then hold a steady rate below it. That shared method is what makes it meaningful to put your run next to somebody else's.
What is not shared is the concurrency, and that is deliberate. Holding fifty connections open is twelve times more than a four-worker server can use and barely warm for a large one — the same number describes two different tests. "Raise it until throughput stops improving" describes the same test everywhere.
The one load setting left is the simulated I/O delay, which changes what a single route measures rather than how hard the load pushes. Change it and the value is recorded and shown with the result, so a custom run cannot be mistaken for a standard one.
What gets recorded with a run
Every result carries the environment that produced it, which covers the PHP version and SAPI, whether the application was serving in worker mode, the performance-related php.ini settings, the FPM pool size where BenchKit can detect it, the database engine, the hardware, the concurrency levels the sweep used, and where the load came from.
The environment comes from the process that served the requests, not from the command line process that assembled the results. The two processes disagree about OPcache, memory limits, and the SAPI. Reporting the wrong one would describe a machine nobody benchmarked.
Once you have a result, Reading your results covers what it can and cannot tell you.