Quick mode
Quick mode is enabled with the --quick
flag and tells criterion to stop benchmarks early once the significance level is below a certail value (default 5%, see the --significance-level
flag).
Quick mode in criterion works exactly like tasty-bench
which has a wealth of details: https://github.com/Bodigrim/tasty-bench
Statistical model
- Set n ← 1.
- Measure execution time tₙ of n iterations and execution time t₂ₙ of 2n iterations.
- Find t which minimizes deviation of (nt, 2nt) from (tₙ, t₂ₙ), namely t ← (tₙ + 2t₂ₙ) / 5n.
- If deviation is small enough (see
--significance-level
) or time has run out (see--measurement-time
), return t as a mean execution time. - Otherwise set n ← 2n and jump back to Step 2.
Disclaimer
Statistics is a tricky matter, there is no one-size-fits-all approach. In the absence of a good theory simplistic approaches are as (un)sound as obscure ones. Those who seek statistical soundness should rather collect raw data and process it themselves using a proper statistical toolbox. Data reported by criterion in quick mode is only of indicative and comparative significance.