IBSI validation#

Use this guide when investigating numerical agreement with IBSI references or maintaining the validation tests. For everyday test commands, see Testing; for implementation coverage and unavailable references, see IBSI validation.

These checks compare numerical results. Execution time and memory measurements are covered separately in Performance benchmarking.

IBSI validation reports#

CI saves per-case integration results and an IBSI summary for each Python version as ibsi-results-python-* artifacts, including failures and skips. To produce the same report locally:

python -m pytest -m integration --junitxml=reports/integration.xml
python tests/ibsi_report.py reports/integration.xml reports/ibsi.md

This uses the same integration-test selection as CI, including the 53 supplemental IBSI asset and filter checks. The report selects IBSI benchmark and supplemental results from the JUnit file and lists their totals separately; unrelated integration tests are not included in the IBSI summary.

The report records each executed benchmark case from JUnit. Revision, working-tree state, environment, dependency versions, and reference hashes describe report generation, so generate the report immediately after testing in the same checkout and environment. This metadata does not authenticate an older or imported JUnit file. It is execution evidence, not a certification. The reference-coverage matrix and limitations are in IBSI validation.

Test data integrity#

IBSI fixtures unpack archives from tests/data/. Extracted files are checked against the archive’s SHA-256 fingerprint and each member’s CRC. Missing, modified, or outdated files trigger extraction under a process lock; a completion marker is written only after successful extraction. Preserve the licensing and attribution information in tests/data/README.md when changing test data.

Reference validation#

The comparison rules define benchmark agreement. The following checks keep reference selection and precision handling explicit.

Reference loaders reject empty selections, duplicate tags, unexpected blanks, non-finite references, and negative tolerances. Feature comparisons require all expected keys for the selected aggregation mode, independently of the keys returned by extraction. IBSI I preprocessing diagnostics are compared separately from radiomic features.

Invalid aggregation modes and modes without texture references fail selection. If a previously unavailable reference gains a value or tolerance, loading fails until its documented exception has been reviewed.

Trailing zeros in plain integer references are treated as placeholders (thus 1494.6 matches 1490 at three significant figures). Decimal or scientific notation retains explicitly written precision; 1.490e3 records four significant figures. References containing more significant digits, such as a voxel count of 125256, retain those digits. A zero reference with zero tolerance requires an exact zero.

Supplemental checks#

tests/test_ibsi_supplemental.py contains 53 cases using IBSI II digital assets. They exercise loading, geometry, and filter behavior independently of published consensus comparisons:

  • Nine DICOM/NIfTI geometry checks cover all supplied phantoms, including orientation, noise, empty, and patterns 2 and 3. NIfTI images and the eight supplied masks are loaded through Z-Rad; DICOM geometry is inspected through SimpleITK. The formats have different origins and their voxel arrays agree after reversing the slice axis; they are not identical physical grids.

  • Nine Z-Rad DICOM loading checks require successful decoding of the synthetic CT series, checking voxel values, dimensions, spacing, direction, and origin. Entirely nonnegative decoded CT intensities are accepted: intensity sign alone does not establish whether HU conversion succeeded. SimpleITK applies the declared DICOM rescaling; separate DICOM regression tests verify known slopes and intercepts, including fractional and nonnegative outputs.

  • Twenty-four zero-input cases check mean, Laplacian-of-Gaussian, and signed Laws filtering in 2D and 3D with constant, nearest, wrap, and reflect padding.

  • Two mean-filter cases compare orientation and noise outputs to explicit periodic neighbourhood averages and check geometry and input preservation.

  • Nine directional Laws cases compare the orientation and pattern 2/3 outputs with explicit three-tap convolution stencils along each axis. These check axis assignment and response sign without using generated golden outputs.