Validation strategy

PyMetaAnalysis uses independent implementations plus layered tests. Validation reduces implementation risk; it does not prove that a method is suitable for a particular review or replace external statistical review.

Validation layers

The core inference review packet maps pooling, heterogeneity, HK, prediction and Q-profile formulas to tests and pinned R references. Its 24 deterministic datasets include explicit QP search limits and native truncation signs; independent formula tests cover numerical boundaries. This evidence remains pending external statistical sign-off.

The sparse binary review packet maps study-level OR/RR/RD, MH OR/RR/RD and Peto formulas to exact rational oracles and 28 studies in six pinned-R case families. It records correction/exclusion policies and the intentional raw-RD difference. This evidence also remains pending external statistical sign-off.

Closed-form and hand calculations

Unit tests compare common-effect estimates, weights, confidence intervals, heterogeneity statistics, effect-size equations, and subgroup-difference tests with direct calculations. These tests make formula and scale errors visible without depending on another package.

Statistical invariants

Property-based and targeted tests check invariants such as:

  • normalized included-study weights sum to one;
  • swapping treatment and control inverts OR/RR and changes the sign of difference measures;
  • Peto is invariant to study order; swapping treatment and control negates its log OR and reciprocates its display estimate and confidence limits;
  • Mantel-Haenszel RD is invariant to study order and common multiplication of every cell count; swapping treatment and control changes its sign, reverses its confidence limits, and preserves its standard error;
  • uncorrected RR accepts a zero non-event cell when the effect and variance remain defined, while zero event cells still require correction;
  • ratio confidence limits transform consistently between log and display scales;
  • Fisher's z correlation pooling is sign-symmetric and invariant to study order, and its confidence limits back-transform consistently with tanh;
  • the ad hoc Hartung-Knapp variance is never below the classic variance;
  • excluded rows do not contribute weights or heterogeneity;
  • result DataFrames and reports are defensive copies;
  • duplicate study labels produce an auditable row-position warning while row_id remains unique;
  • explicitly inapplicable tau-squared and SMD variance options raise domain errors instead of being ignored;
  • missing pandas timestamps serialize as JSON null;
  • subgroup and repeated-fit row IDs map back to original input positions;
  • subgroup-differences tests are invariant to the confidence-interval method, including Hartung-Knapp adjustments;
  • unestimable repeated Mantel-Haenszel refits remain visible without aborting other leave-one-out or cumulative fits;
  • direct tau-squared estimators reject non-positive residual degrees of freedom, and positive roots below atol remain distinct from exact boundary solutions;
  • Q-profile bounds satisfy their generalized-Q inversion equations, are independent of the selected tau-squared point estimator, transform consistently under effect rescaling, and distinguish zero lower bounds from formal empty confidence sets;
  • empty or zero-total Mantel-Haenszel inputs raise domain errors before any non-finite intermediate value is formed;
  • MH and Peto remain finite across strata separated by roughly 307 orders of magnitude when the model-scale estimate and variance are representable; exact rational formulas check estimates, SEs, weights and Peto Q without relying on another float implementation;
  • Meta-regression is invariant to row order, moderator centering, representable positive unit changes through 1e100, and large exactly representable effect offsets; changing a categorical reference preserves fitted values, residual heterogeneity, and joint tests. Generalized DL has an exact rational oracle through a 1e300 precision ratio;
  • Meta-regression VIF/GVIF is invariant to positive moderator rescaling and row order; weighted condition indices match an independent singular-value decomposition and coefficient variance proportions sum to one;
  • Explicit linear contrasts match direct matrix calculations, preserve scientifically equivalent hypotheses across category-reference, unit, and contrast-row changes through factors of 1e200, and reject ambiguous or rank-deficient contrast specifications;
  • Riley Meta-regression prediction intervals are symmetric around the fitted effect, preserve mean-effect inference, use one fewer residual degree of freedom than the default rule, and remain wider at the zero-tau-squared boundary;
  • Bubble-plot coordinates, precision-weight marker areas, fitted predictions, interval bands, exclusions, and optional Matplotlib behavior are tested with a non-interactive backend.
  • Contour-enhanced funnel regions match direct normal-quantile calculations on identity, exponentiated-ratio, and back-transformed-correlation scales; colors, legends, null references, pseudo-limit interaction, and invalid inputs are tested.
  • Begg-Mazumdar rank correlation is invariant to study order, effect-location shifts, positive variance rescaling, and the source pooling model; its common-effect-centered standardized response, exact permutation inference, tie-adjusted asymptotic variance, continuity correction, tie reporting, and invalid inference combinations are tested.
  • Classical Egger regression is invariant to study order and effect-location shifts; its weighted coefficients, multiplicative-dispersion covariance, t inference, and limit estimate match direct calculations.
  • Harbord regression is invariant to study order, source pooling, and continuity-correction settings; treatment/control reversal negates its score coefficients, while its efficient scores, variance, multiplicative dispersion, and t inference match direct calculations.
  • Peters regression is invariant to study order and source pooling method; treatment/control reversal changes the log-scale coefficient signs, while its S*F/N weighting, correction reuse, covariance, and t inference match direct calculations.

Numerical edge cases

The test suite covers extreme sampling variances, identical effects, tau- squared boundary solutions, iterative convergence controls, overflow-safe binary counts, nonrepresentable derived effects, explicit subnormal-variance rejection, zero cells, double-zero/double-all tables, missing rows, and minimum study counts.

Cross-software references

Committed JSON fixtures are generated by adjacent R scripts using documented metafor or meta interfaces. Ordinary Python tests read the committed artifacts and do not require R.

The current fixture families cover:

Fixture Coverage
generic_metafor.json Common effects, DL/PM/REML random effects, Q-profile heterogeneity intervals, HK variants including a two-study t_1 case, and HTS/HK-PR prediction intervals
binary_metafor.json OR/RR/RD study effects, IV pooling, uncorrected and explicitly corrected MH pooling/heterogeneity, Sato-Greenland-Robins MH RD variance, Peto study effects/pooling/O-minus-E heterogeneity, sparse tables
sparse_binary_review_metafor.json Six deterministic no-zero, single-zero, double-boundary, unequal-arm, opposing-effect and large-count datasets; OR/RR/RD study effects, raw/corrected MH fits, and Peto study effects/pooling/O-minus-E heterogeneity
continuous_metafor.json MD and exact-corrected SMD effects and pooled fits
correlation_metafor.json Fisher's z effects, 1 / (n - 3) variances, common and REML fits, weights, and back-transformation
workflow_metafor.json Common subgroups, random subgroup Q-profile intervals, random singleton fallback, leave-one-out, and cumulative common/random fits
meta_regression_metafor.json Numeric, categorical, and multivariable common/mixed meta-regression; DL/PM/REML; HK variants; joint tests; default and Riley predictions, including a no-intercept Riley case; zero tau-squared, missing rows, and small samples
meta_regression_influence_metafor.json Exact deleted residuals and their standard errors, externally standardized residuals, Cook's distances, and DFBETAS across numeric, categorical, and multivariable common/mixed designs
meta_regression_collinearity_metafor.json Term VIF, categorical moderator GVIF, and dimension-adjusted inflation factors for common and REML Meta-regression
meta_regression_contrasts_metafor.json Individual linear-combination estimates, standard errors, z/t tests, and joint chi-squared/F tests across common, REML, and Hartung-Knapp fits
small_study_effects_metafor.json Classical standard-error Egger regression intercept, multiplicative-dispersion standard errors, t test, and limit estimate
begg_ranktest_metafor.json Begg-Mazumdar standardized response, Kendall tau-b, concordance statistic, exact two-sided p-value, and tied asymptotic inference
harbord_small_study_effects_meta.json Harbord efficient-score asymmetry intercept, limit coefficient, multiplicative dispersion, and t test for binary OR data with single-zero studies
peters_small_study_effects_meta.json Peters binary-OR slope, S*F/N weights, multiplicative dispersion, t test, limit estimate, and single-zero continuity corrections
trimfill_metafor.json 24 common/REML L0/R0 cases with explicit/automatic sides and tied effects: missing counts, uncertainty, augmented effects/variances, imputed flags, adjusted estimate/SE/CI and Q/I-squared/H-squared

Meta-regression is additionally covered by hand-calculated weighted least squares, generalized tau-squared score equations, intercept-only equivalence with the independently validated pooling path, inference covariance checks, and encoding invariants. Exact rational generalized-DL tests exercise projection traces too small for subtractive float arithmetic; power-of-two moderator and contrast rescaling plus large effect offsets check equivalent fitted geometry. See the Meta-regression review packet for the formula map, finding disposition, tolerances, and pending review scope.

The no-intercept metafor fixtures validate coefficients, covariance, fitted values, residuals, weights, leverage, the moderator test, and default and Riley true-effect predictions. Their QE is not used as a reference because metafor retains a different no-intercept QE reporting convention from the weighted residual sum of squares reported by PyMetaAnalysis. For the mixed no-intercept case, the generator independently solves the standard raw-outcome REML score in R and cross-checks it with metafor::rma.mv, whose general random-effects fit supplies the prediction reference. This avoids rma.uni's internal outcome-centering convention, which is immaterial with an intercept but changes a through-origin model; the fixture also records rma.uni's default no-intercept estimate for auditability.

Every artifact records the R and jsonlite versions plus the applicable metafor or meta version used to produce it. The four Meta-regression generators now enforce R 4.6.1, metafor 5.0.1, and jsonlite 2.0.0 before generation. Method-specific numerical tolerances distinguish closed-form from iterative comparisons.

Trim-and-fill comparisons use rtol=5e-13, atol=5e-15 for common-effect statistics and missing-count uncertainty, and rtol=2e-10, atol=2e-11 for REML and derived augmented rows. The generator passes the recorded REML threshold and iteration limit explicitly. Since trimfill()'s final fit can use its own defaults, the artifact retains that native output and independently refits the augmented data with explicit controls for the detailed reference. Iteration traces still lack an independent R reference, and selected tied cases do not establish universal tie equivalence. Formal method sign-off remains open; see the review findings.

Sparse RD reference tests make one intentional convention difference explicit: metafor::escalc corrects the displayed single-zero RD when correction is requested, while PyMetaAnalysis retains the raw RD and uses corrected counts only for variance. Boundary-table variances and unaffected effects are still cross-checked.

Regenerate R fixtures

Install R plus metafor, meta, and jsonlite, then run from the repository root:

Rscript tests/reference/generate_generic_metafor.R
Rscript tests/reference/generate_core_inference_metafor.R
Rscript tests/reference/generate_binary_metafor.R
Rscript tests/reference/generate_continuous_metafor.R
Rscript tests/reference/generate_correlation_metafor.R
Rscript tests/reference/generate_workflow_metafor.R
Rscript tests/reference/generate_meta_regression_metafor.R
Rscript tests/reference/generate_meta_regression_influence_metafor.R
Rscript tests/reference/generate_meta_regression_collinearity_metafor.R
Rscript tests/reference/generate_meta_regression_contrasts_metafor.R
Rscript tests/reference/generate_small_study_effects_metafor.R
Rscript tests/reference/generate_begg_ranktest_metafor.R
Rscript tests/reference/generate_harbord_small_study_effects_meta.R
Rscript tests/reference/generate_peters_small_study_effects_meta.R
Rscript tests/reference/generate_trimfill_metafor.R

Pass an optional output file to review regenerated values before replacement:

Rscript tests/reference/generate_generic_metafor.R regenerated.json

Fixture regeneration is a statistical-review action. Do not replace values only to make a failing test pass; identify and document the reason for every difference.

Continuous integration

GitHub Actions runs:

  • the test suite with branch coverage on Python 3.10–3.14;
  • a Python 3.10 job at declared direct-dependency lower bounds;
  • Ruff formatting and lint checks;
  • strict Mypy type checking;
  • MkDocs strict link/navigation validation;
  • execution of every committed example notebook;
  • release-metadata and distribution-content checks;
  • wheel and source-distribution builds plus an installed-wheel smoke test.
  • a 1,000-study trim-and-fill performance smoke test and PNG rendering checks for observed/imputed markers composed with significance contours.

These jobs currently run on Linux. Windows and macOS CI remain open gates in the 1.0 acceptance matrix; a local Windows run does not close the full platform matrix. The API inventory is checked by the test suite.

The configured branch-coverage floor is 90%.

Python examples in the README and documentation execute sequentially per page with visible input data, an empty namespace and a temporary working directory. Created figures render with Agg, report objects pass JSON export/parse checks, and the plotting guide writes a verified PNG. Only the explicitly labeled public API signatures with omitted ... arguments are syntax-checked without execution. This does not execute shell commands, R snippets or schema-shaped text blocks. Notebook execution and R fixture validation remain separate.

Binary workflow tests independently calculate corrected four-cell log ORs, sampling variances, IV/MH pooled estimates, Q and two-group Wald comparisons. They cover if_any_zero, only_zero_studies and all_studies, IV common/random fits, raw/explicitly corrected MH, and the distinction between study and pooling corrections. Excluded zero/missing rows, duplicated labels, source IDs, retained settings and correction provenance are checked across subgroups, leave-one-out and cumulative fits (including tied publication years). In particular, if_any_zero is reevaluated when the only informative zero-cell study is removed or added; source results remain unchanged.

Reproduce local validation

From an editable development installation:

python -m ruff format --check .
python -m ruff check .
python -m mypy
python -m pytest --cov=meta_analyze --cov-branch --cov-report=term-missing
python -m mkdocs build --strict
python tools/execute_notebooks.py
python tools/check_release.py
python -m build
python tools/inspect_distribution.py dist
python benchmarks/benchmark_core.py

Validation boundary

Formal review preparation and sign-off requirements are in the independent statistical review checklist. None of its packets has been accepted merely by introducing that checklist.

The project has cross-software regression coverage but has not yet undergone a formal independent statistical audit. Consequential analyses should pin the package version, retain the generated report/provenance, and independently check important results using a trusted implementation and the review's prespecified methods.