Validation strategy

PyMetaAnalysis uses independent implementations plus layered tests. Validation reduces implementation risk; it does not prove that a method is suitable for a particular review or replace external statistical review.

Validation layers

Closed-form and hand calculations

Unit tests compare common-effect estimates, weights, confidence intervals, heterogeneity statistics, effect-size equations, and subgroup-difference tests with direct calculations. These tests make formula and scale errors visible without depending on another package.

Statistical invariants

Property-based and targeted tests check invariants such as:

  • normalized included-study weights sum to one;
  • swapping treatment and control inverts OR/RR and changes the sign of difference measures;
  • ratio confidence limits transform consistently between log and display scales;
  • the ad hoc Hartung-Knapp variance is never below the classic variance;
  • excluded rows do not contribute weights or heterogeneity;
  • result DataFrames and reports are defensive copies;
  • subgroup and repeated-fit row IDs map back to original input positions.
  • Meta-regression is invariant to row order and moderator centering; changing a categorical reference preserves fitted values, residual heterogeneity, and joint tests.
  • Bubble-plot coordinates, precision-weight marker areas, fitted predictions, interval bands, exclusions, and optional Matplotlib behavior are tested with a non-interactive backend.

Numerical edge cases

The test suite covers extreme sampling variances, identical effects, tau- squared boundary solutions, iterative convergence controls, zero cells, double-zero/double-all tables, missing rows, and minimum study counts.

Cross-software references

Committed JSON fixtures are generated by adjacent R scripts using documented metafor interfaces. Ordinary Python tests read the committed artifacts and do not require R.

The current fixture families cover:

Fixture Coverage
generic_metafor.json Common effects, DL/PM/REML random effects, HK variants, HTS prediction interval
binary_metafor.json OR/RR/RD study effects, IV pooling, MH pooling/heterogeneity, sparse tables
continuous_metafor.json MD and exact-corrected SMD effects and pooled fits
workflow_metafor.json Subgroups, leave-one-out, and cumulative common/random fits
meta_regression_metafor.json Numeric, categorical, and multivariable common/mixed meta-regression; DL/PM/REML; HK variants; joint tests, predictions, zero tau-squared, missing rows, and small samples

Meta-regression is additionally covered by hand-calculated weighted least squares, generalized tau-squared score equations, intercept-only equivalence with the independently validated pooling path, inference covariance checks, and encoding invariants.

The no-intercept metafor fixture validates coefficients, covariance, fitted values, residuals, weights, leverage, and the moderator test. Its QE is not used as a reference because metafor retains a different no-intercept QE reporting convention from the weighted residual sum of squares reported by PyMetaAnalysis.

Every artifact records the R, metafor, and jsonlite versions used to produce it. Method-specific numerical tolerances distinguish closed-form from iterative comparisons.

Sparse RD reference tests make one intentional convention difference explicit: metafor::escalc corrects the displayed single-zero RD when correction is requested, while PyMetaAnalysis retains the raw RD and uses corrected counts only for variance. Boundary-table variances and unaffected effects are still cross-checked.

Regenerate R fixtures

Install R plus metafor and jsonlite, then run from the repository root:

Rscript tests/reference/generate_generic_metafor.R
Rscript tests/reference/generate_binary_metafor.R
Rscript tests/reference/generate_continuous_metafor.R
Rscript tests/reference/generate_workflow_metafor.R
Rscript tests/reference/generate_meta_regression_metafor.R

Pass an optional output file to review regenerated values before replacement:

Rscript tests/reference/generate_generic_metafor.R regenerated.json

Fixture regeneration is a statistical-review action. Do not replace values only to make a failing test pass; identify and document the reason for every difference.

Continuous integration

GitHub Actions runs:

  • the test suite with branch coverage on Python 3.10–3.13;
  • a Python 3.10 job at declared direct-dependency lower bounds;
  • Ruff formatting and lint checks;
  • strict Mypy type checking;
  • MkDocs strict link/navigation validation;
  • execution of every committed example notebook;
  • release-metadata and distribution-content checks;
  • wheel and source-distribution builds plus an installed-wheel smoke test.

The configured branch-coverage floor is 90%.

Reproduce local validation

From an editable development installation:

python -m ruff format --check .
python -m ruff check .
python -m mypy
python -m pytest --cov=meta_analyze --cov-branch --cov-report=term-missing
python -m mkdocs build --strict
python tools/execute_notebooks.py
python tools/check_release.py
python -m build
python tools/inspect_distribution.py dist
python benchmarks/benchmark_core.py

Validation boundary

The project has cross-software regression coverage but has not yet undergone a formal independent statistical audit. Consequential analyses should pin the package version, retain the generated report/provenance, and independently check important results using a trusted implementation and the review's prespecified methods.