Interpret Diagnostics

Objective

Read and act on the diagnostics report produced by a DSAMbayes runner execution, understanding which checks matter most and what remediation steps to take.

This is the operational triage guide. For the methodological meaning of the gates themselves, see Stage 4: Computation and Sampler and Stage 5: Model Adequacy.

Prerequisites

  • A completed runner run execution with artefacts under 40_diagnostics/.
  • Familiarity with Diagnostics Gates definitions.

Steps

1. Open the diagnostics report

cat results/<run_dir>/40_diagnostics/diagnostics_report.csv

Each row is one diagnostic check. The key columns are:

Column What to look at
check_id Identifies the specific diagnostic
status pass, warn, fail, or skipped
value The observed metric value
threshold The threshold that was applied
message Human-readable explanation

2. Check the overall status

The overall status follows a simple rule:

  • Any fail → overall fail.
  • Any warn (no fails) → overall warn.
  • All pass → overall pass.

If the overall status is pass, no enabled check breached its configured threshold. Continue with substantive fit, predictive, sensitivity, identifiability and intended-use review. A pass is not model approval.

3. Triage failing checks

Focus on fail rows first, then warn rows. Use the check phase to prioritise:

Phase Priority Meaning
P0 Highest Data integrity or critical MCMC reliability, fix before interpretation
P1 High Conditioning, residual or identifiability concerns, review before use
P2 Supplementary Predictive scoring and influence evidence, never causal validation

4. Common diagnostics and actions

The phase assignments below match the emitted phase field:

Check ID Phase
pre_response_finite P0
pre_design_constants_duplicates P0
pre_design_rank_deficit P0
post_rhat_max P0
post_ess_bulk_min P0
post_ess_tail_min P0
post_divergences P0
post_treedepth_saturation P0
post_ebfmi P0
pre_design_condition_number P1
post_residual_ljung_box_p_min P1
post_residual_max_abs_acf P1
post_boundary_hit_rate_max P1
pre_within_variation_ratio_min P1
pre_identifiability_baseline_media_corr P1

Design integrity (P0)

Check Symptom Action
pre_response_finite fails Non-finite values in response Clean data; remove or impute NA/Inf rows
pre_design_constants_duplicates fails Constant or duplicate columns Remove redundant terms from formula
pre_design_rank_deficit warns/fails Rank-deficient design Remove redundant terms; review whether the specification can identify separate effects

Design conditioning (P1)

Check Symptom Action
pre_design_condition_number warns/fails High collinearity Reduce correlated predictors; simplify formula

Sampler quality (P0, MCMC only)

Check Symptom Action
post_rhat_max warns/fails Poor convergence Increase fit.mcmc.iter and fit.mcmc.warmup; simplify model
post_ess_bulk_min or post_ess_tail_min warns/fails Insufficient effective samples Increase iterations; check for multimodality
post_divergences fails Divergent transitions Increase fit.mcmc.adapt_delta (e.g. 0.95 → 0.99); consider reparameterisation
post_treedepth_saturation warns/fails Max treedepth saturation Increase fit.mcmc.max_treedepth
post_ebfmi warns/fails Low energy diagnostic Indicates difficult posterior geometry; simplify model or increase warmup

Residual behaviour (P1)

Check Symptom Action
post_residual_ljung_box_p_min warns/fails Significant residual autocorrelation Add time controls (trend, seasonality, holidays)
post_residual_max_abs_acf warns/fails High residual ACF at early lags Same as above; check for missing structural components

Boundary and variation checks (P1)

Check Symptom Action
post_boundary_hit_rate_max warns/fails Posterior draws hitting parameter bounds Review boundary specification; widen constraints or remove unnecessary bounds
pre_within_variation_ratio_min warns/fails Low within-group variation (hierarchical) Check group structure; some groups may have insufficient temporal variation

Identifiability gate (P1)

Check Symptom Action
pre_identifiability_baseline_media_corr warns/fails High baseline-media correlation Add controls to separate baseline from media effects; review formula specification

5. Review diagnostic plots

Cross-reference the numeric report with visual diagnostics in 40_diagnostics/:

  • Residual diagnostics plot: check for patterns in residuals over time.
  • Boundary hits plot: identify which parameters are constrained.
  • Latent residual ACF plot: confirm autocorrelation structure.

See Diagnostics Plots for interpretation guidance.

6. Decide on next steps

Overall status Action
pass Continue the remaining adequacy and intended-use review; do not treat the pass as approval
warn Review warnings; proceed if acceptable for the use case
fail Remediate failing checks before using model results for decisions

7. Change policy mode if appropriate

If you are in early model development, consider switching to explore mode to relax thresholds:

diagnostics:
  policy_mode: explore

For production or audit runs, use publish (default) or strict.