Compare Runs

Objective

Compare multiple DSAMbayes runner executions and select a candidate model for reporting or decision-making, using predictive scoring and diagnostic summaries.

This page is a late-stage selection aid, not a full workflow. Use it only after the candidate runs are computationally trustworthy enough to compare. In the principled workflow, that means Stage 4 and Stage 5 work has already been done: the sampler is behaving acceptably, and the model is at least adequate enough to remain a candidate. For the surrounding methodology, see Stage 4: Computation and Sampler and Stage 5: Model Adequacy.

Prerequisites

  • Two or more completed runner run executions (MCMC fit method).
  • Artefacts under 50_model_selection/ for each run (LOO summary, ELPD outputs).
  • Familiarity with Diagnostics Gates and Model Selection Plots.
  • Candidate runs that are still eligible after basic diagnostic review. Do not compare obviously broken runs just because they produced LOO outputs.

Steps

1. Collect run directories

Identify the run directories to compare:

results/20260228_083808_blm_synth_kpi_os_hfb01/
results/20260228_084410_blm_synth_kpi_os_hfb01/
results/20260228_084602_blm_synth_kpi_os_hfb01/

2. Compare ELPD scores

The compare_runs() helper ranks runs by expected log predictive density (ELPD):

library(DSAMbayes)
comparison <- compare_runs(
  run_dirs = c(
    "results/20260228_083808_blm_synth_kpi_os_hfb01",
    "results/20260228_084410_blm_synth_kpi_os_hfb01"
  )
)
print(comparison)

The output ranks runs by ELPD (higher is better) and reports Pareto-k diagnostics. When TSCV summaries are present, the table also carries tscv_method, tscv_horizon_weeks, tscv_stride_weeks, tscv_min_train_weeks, and tscv_gap_weeks so you can see whether holdout policies actually match. When 00_run_metadata/artifact_schema.yaml is present, the table also carries artifact_schema_version. Treat this as ranking among plausible candidates, not as an automatic winner-selection rule.

3. Check Pareto-k reliability

Examine the loo_summary.csv in each run’s 50_model_selection/ folder:

cat results/<run_dir>/50_model_selection/loo_summary.csv

Key metrics:

Metric Interpretation
elpd_loo Expected log predictive density; higher is better
p_loo Effective number of parameters
looic LOO information criterion; lower is better
Pareto-k counts Observations with k > 0.7 indicate unreliable LOO estimates

If many observations have high Pareto-k values, the LOO approximation is unreliable for that run. Consider time-series cross-validation as an alternative.

4. Review time-series CV (if available)

If diagnostics.time_series_selection.enabled: true was configured, check:

cat results/<run_dir>/50_model_selection/tscv_summary.csv

This provides blocked-CV or leave-future-out scores (holdout ELPD, RMSE, SMAPE), optionally with an embargo gap when gap_weeks is configured, and is usually more appropriate for time-series data than standard LOO.

compare_runs() warns if candidate runs used different TSCV policies. When that happens, rank_tscv and delta_tscv_elpd are left NA. Treat those fields as comparable only when method, horizon_weeks, stride_weeks, min_train_weeks, and gap_weeks match.

compare_runs() also warns when explicit artifact_schema_version values differ across runs. That warning does not block ranking, but it means the helper is reading known artifacts on a best-effort basis across evolving run contracts.

Time-series selection is advisory in the current runner contract. It is useful for model comparison, but it does not change publish-gate status.

5. Cross-reference diagnostics

For each candidate run, check the diagnostics overall status:

head -1 results/<run_dir>/40_diagnostics/diagnostics_report.csv

A model with better ELPD but failing diagnostics should not be preferred over a model with slightly lower ELPD and passing diagnostics.

If a run is computationally untrustworthy, remove it from contention before you start arguing about small predictive-score differences.

6. Compare fit quality visually

Review the fit time series and scatter plots in 20_model_fit/ for each run:

  • Fit time series — does the model track the observed KPI?
  • Fit scatter — is the predicted-vs-observed relationship close to the diagonal?
  • Posterior forest — are coefficient estimates reasonable and well-identified?

7. Selection decision matrix

Criterion Weight Run A Run B
Eligible after diagnostics review? Gate yes/no yes/no
ELPD (higher is better) High value value
Pareto-k reliability (fewer high-k) High value value
Diagnostics overall status High pass/warn/fail pass/warn/fail
TSCV holdout RMSE (if available) Medium value value
Coefficient plausibility Medium judgement judgement
Fit visual quality Low judgement judgement

Use the matrix in order:

  1. Remove runs that are not computationally trustworthy enough to compare.
  2. Rank the remaining candidates by predictive evidence.
  3. Prefer the run whose coefficients, decomposition, and fit behaviour remain most defensible for the business question.

8. Record the selection

Document the selected run directory and rationale. If using the runner for release evidence, the selected run’s artefacts form part of the evidence pack.

Caveats

  • ELPD is not causal validation. Predictive scoring measures in-sample predictive quality, not whether the model identifies causal media effects correctly.
  • ELPD is not a substitute for adequacy. Stronger predictive ranking does not rescue a run that is diagnostically broken or substantively implausible.
  • Pooled models do not support time-series CV (rejected by config validation).
  • Adstock/Hill media transforms are not supported by time-series CV; lower-level scoring aborts if transformed-media paths are used.
  • MAP-fitted models do not produce LOO diagnostics. Use MCMC for model comparison.