Performance benchmarking#

The benchmarks in tests/benchmarks measure Z-Rad execution time and memory use. Use individual operation benchmarks to identify where performance changed, and IBSI workflows to assess the effect on complete processing workflows. Setup and validation run outside the timed region. For memory measurements, see Memory profiling; for workload boundaries and adding benchmarks, see Benchmark methodology and maintenance.

Quickstart#

Run these commands from the repository root. Install the test dependencies (also included in the dev extra), then run the standard suite:

python -m pip install -e '.[test]'
python tests/benchmarks/run.py --suite standard

The launcher sets thread limits before importing numerical libraries and runs pytest with parallel workers and coverage disabled. Ordinary test runs skip benchmarks before their fixtures run; see Testing for test commands.

Choose a suite#

Timing and memory measurements use the same suite definitions and workload IDs.

Benchmark suites#
Suite	Cases	Coverage
`standard` (default)	100	Synthetic operation benchmarks and all 20 IBSI I workflows
`exhaustive`	151	All standard cases, plus 33 IBSI II phase-I filter response maps and 18 IBSI II phase-II feature workflows
`ibsi`	71	Only the published IBSI cases: `ibsi1`, `ibsi2_phase1`, and `ibsi2_phase2`

For broader coverage or a focused IBSI run:

python tests/benchmarks/run.py --suite exhaustive
python tests/benchmarks/run.py --suite ibsi

Save and compare results#

Choose a comparison workflow#
What you want to measure	Approach
Uncommitted production changes	Run `run.py` in the working tree and save the result.
Two committed revisions	Use `compare_revisions.py` to install and measure each revision.
Previously saved measurements	Use `compare_results.py` with explicit reference files.

Compare committed revisions#

Use the revision runner to compare committed changes with master on the same machine:

git fetch origin master --tags
python tests/benchmarks/compare_revisions.py --master origin/master --current HEAD --output reports/benchmarks/comparison-001

The output directory must be new. The runner resolves each ref once, installs it non-editably in a separate temporary virtual environment, and runs revisions sequentially with the same Python interpreter and thread controls. It uses the current working-tree benchmark harness for all revisions and checks that zrad.__file__ points to the intended installation. Temporary source trees and environments are removed afterward.

Uncommitted production changes are excluded. Use run.py to measure working-tree changes. Pass --suite exhaustive for full coverage; --full is a deprecated alias for that option.

To include a release comparison, add --release RELEASE_TAG, replacing RELEASE_TAG with a published release compatible with the current harness. For example, v26.8.0 is incompatible because it lacks RieszLoG and Simoncelli. The runner imports the workload module before selecting a suite, so choosing a smaller suite does not avoid that incompatibility. Failed or incompatible references are reported with logs; partial results are labelled INVALID and excluded from comparisons. Candidate failures fail the run.

Dependencies are resolved from each revision’s requirements and may differ. Check *-dependencies.txt and the run metadata. To hold dependencies fixed, pass --constraints /absolute/path/constraints.txt with versions compatible with all revisions being compared.

Save individual runs#

Timing results use pytest-benchmark’s JSON format. To save a working-tree run:

mkdir -p reports/benchmarks
python tests/benchmarks/run.py --suite standard --benchmark-json=reports/benchmarks/current.json

If you already have an accepted master.json reference, compare it explicitly:

python tests/benchmarks/compare_results.py --current reports/benchmarks/current.json --master reports/benchmarks/master.json

Add --release reports/benchmarks/release.json when a release reference is available. For a side-by-side view of the underlying statistics, use:

pytest-benchmark compare reports/benchmarks/master.json reports/benchmarks/current.json --columns=median,iqr,mean,stddev,min,max,rounds,iterations

Maintain references#

Master reference: an accepted master SHA, updated deliberately after accepted changes reach master. A passing PR does not automatically replace it.
Release reference: a published tag/SHA, retained until you deliberately select a new release reference.
Current/PR: a candidate, saved separately from both references.

Save names are labels; they do not check out revisions. Ensure that each run executes the intended code, especially when using editable installations across checkouts. Retain the reference’s workload and environment metadata in a durable artifact archive. Generated JSON, .benchmarks, reports, and profiling captures are ignored by git.

Note

Create references with the version-1 suite. Development runs used changing workload definitions, and early complete-extraction timings reused cached local means. Those timings are invalid as fresh-extraction references.

Interpret results#

Use the median runtime as the main comparison metric. compare_results.py calculates each change relative to the explicitly selected reference:

100 * (current_median - reference_median) / reference_median

A change from 1 s to 2 s is a +100% regression; 2 s to 1 s is a -50% improvement. These labels describe direction, not statistical significance. The interquartile range (IQR) describes spread within a run; it is not a confidence interval for the difference between runs.

Retain mean, minimum, maximum, standard deviation, IQR, rounds, and iterations. Repeat noisy measurements on an idle machine, running revisions in both orders. Differences comparable to normal run-to-run variability are inconclusive. The suite has no timing or memory regression thresholds.

Results are matched by extra_info.workload_id, falling back to the full test name when that field is absent. Check that workload definitions and environments are compatible before comparing. Missing workloads are reported as unavailable; zero or invalid reference timings produce no percentage verdict. Missing results do not indicate unchanged performance.

Warning

Use compare_results.py for candidate percentage changes. In pytest-benchmark 5.3.0, compare --between sorts filenames and may choose current.json as the baseline regardless of argument order. Regenerate summaries made with that command from the raw JSON.

Results from CI#

The performance workflow publishes comparison summaries and raw artifacts; see Continuous integration for triggers and artifact names. On pull requests, the candidate is GitHub’s tested merge commit. The master reference is resolved at checkout; on pushes to master, candidate and reference may be identical, providing an observation of measurement noise. The latest published stable release is resolved to a fixed SHA for that job. These runs do not replace your saved accepted references.

GitHub-hosted hardware and background load vary. Treat the measurements as performance indicators and investigate changes locally. Candidate measurement or validation failures fail the job; timing and memory changes have no configured regression threshold.

Advanced: named saves#

You can also use pytest-benchmark’s named saves. Run the first command at the accepted master revision, then switch to the candidate before saving it:

python tests/benchmarks/run.py --suite standard --benchmark-save=master-accepted --benchmark-save-data
pytest-benchmark list
# Replace 0001 with the accepted reference ID from the listing
python tests/benchmarks/run.py --suite standard --benchmark-compare=0001 --benchmark-save=pr-candidate --benchmark-save-data

--benchmark-json includes individual round samples. For named saves, --benchmark-save-data retains them. Always specify the reference ID: --benchmark-compare without an ID uses the latest saved run, which may be another candidate.

For direct pytest invocation and its thread-control requirements, see Benchmark methodology and maintenance.