Testing and Validation

Purpose

Define the canonical testing and validation workflow for DSAMbayes v1.3.3, from local pre-merge checks through release-quality gates.

Audience

  • Engineers running local checks before merge
  • Maintainers preparing release candidates
  • Reviewers validating release evidence

Validation layers

Layer Objective Primary command(s) Output proof
Lint Catch style and static issues early Rscript scripts/check.R --lint Exit code 0, no lint failures
Style Enforce formatting compliance on changed files Rscript scripts/check.R --style Exit code 0, no reformat-required files
Unit tests Catch behavioural regressions in package logic Rscript scripts/check.R --test Exit code 0, no test failures
Minimal smoke Keep a cheap Stan-backed safety check in the routine local loop Rscript scripts/check.R --smoke Exit code 0, fast unit tests plus minimal Stan smoke pass
Stan smoke Exercise the broader compiled Stan suite beyond the minimal smoke tier Rscript scripts/check.R --stan-smoke Exit code 0, Stan smoke tests enabled
Stan recovery evidence Re-run the full coefficient-recovery file under the nightly Stan gate Rscript scripts/check.R --stan-recovery Exit code 0, full-Stan DGP recovery tests pass
Stan release evidence Re-run high-budget pooled and hierarchical evidence checks for release candidates Rscript scripts/check.R --stan-release-evidence Exit code 0, targeted high-budget Stan evidence passes diagnostic thresholds
Docs sanity Catch local docs-link breakage and high-value contract drift Rscript scripts/check.R --docs Exit code 0, docs sanity checks pass
Package check Validate package-level install and check behaviour R -q -e 'rcmdcheck::rcmdcheck(...)' No ERROR; no unresolved WARNING
Runner validate Validate config and data contracts without fitting Rscript scripts/dsambayes.R validate ... Exit code 0, metadata artefacts
Runner run Validate end-to-end runner execution and artefacts Rscript scripts/dsambayes.R run ... Exit code 0, core run artefacts
Docs build Validate docs-site/Hugo buildability python3 docs-site/build_content.py && (cd docs-site && hugo --cleanDestinationDir) Exit code 0, successful site build

Environment setup

Run all commands from repository root:

# Navigate to your local DSAMbayes checkout
cd /path/to/DSAMbayes
source scripts/r-library-path.sh
dsambayes_set_r_library host
mkdir -p "$R_LIBS_USER" .cache
export XDG_CACHE_HOME="$PWD/.cache"

Expected outcome: checks run in a repo-scoped environment with reproducible library and cache paths.

Dependency source portability

Before a release candidate is signed off, verify all non-CRAN dependency sources in renv.lock and DESCRIPTION.

The preferred release path is a pinned private GitHub remote for optional decomposition dependencies. A local fallback may restore those dependencies from sibling checkouts:

/home/user/Documents/GITHUB/tandpds/DSAMdecomp
/home/user/Documents/GITHUB/tandpds/teller

If any entries remain as file:// sources in renv.lock, release evidence must record the required local paths and pinned commit hashes. For an external release or clone-portable handoff, prefer reachable pinned remotes before sign-off.

When validating either dependency source path, use telemetry-disabled restores:

export GITHUB_PAT="<token with access to private tandpds repos>"
export DSAMDECOMP_DISABLE_TELEMETRY=1
export DSAMBAYES_DISABLE_TELEMETRY=1
export USE_BUNDLED_LIBUV=1
XDG_CACHE_HOME="$PWD/.cache" \
  Rscript -e 'renv::restore(packages = c("teller", "DSAMdecomp"), prompt = FALSE)'

Expected outcome: teller and DSAMdecomp restore and load from the recorded sources. If private GitHub authentication is unavailable, or a local fallback path is missing, hold the release until dependency sources are reproducible in the release environment.

Local validation workflows

Developer fast path (pre-merge)

Use the explicit local ladder:

Rscript scripts/check.R --test
Rscript scripts/check.R --smoke
Rscript scripts/check.R --docs
Rscript scripts/check.R --release

Expected outcomes:

  • --test stays fast and runs the unit path only.
  • --smoke runs the unit path plus a minimal Stan-backed subset: one cheap BLM MCMC smoke, one pooled deployment-artifact smoke on a real pooled fit, and tiny run_from_yaml() runner smokes including pooled deployment_model.rds roundtrip coverage.
  • --docs runs the local docs/link sanity helper against README and the tracked docs surfaces.
  • --release runs lint, style, unit tests, minimal Stan smoke, docs sanity, and coverage as a broader local code gate.

Implementation note:

  • scripts/check.R --all remains a legacy convenience gate for lint, style, tests, and coverage only.
  • scripts/check.R --docs is the explicit docs/link drift check.
  • scripts/check.R --release is the clearer code-focused local release profile.
  • Neither profile replaces rcmdcheck, runner smoke checks, or docs build.

Stan-specific note:

  • use Rscript scripts/check.R --stan-smoke for the broader opt-in compiled Stan suite
  • use --stan-smoke-full for the fuller nightly variant
  • use --stan-recovery when you need a targeted rerun of the DGP recovery evidence without invoking the rest of the full Stan suite
  • use --stan-release-evidence for the high-budget release-candidate lane covering the warning-prone pooled and hierarchical paths

Release-candidate full path

Run mandatory gates in this exact order:

Rscript scripts/check.R --lint
Rscript scripts/check.R --style
Rscript scripts/check.R --smoke
Rscript scripts/check.R --stan-release-evidence
_R_CHECK_FORCE_SUGGESTS_=false \
  R -q -e 'rcmdcheck::rcmdcheck(args = c("--no-manual"), error_on = "warning")'
Rscript scripts/dsambayes.R validate --config config/blm_timeseries.yaml --run-dir results/quality_gate_validate
Rscript scripts/dsambayes.R run --config config/blm_timeseries.yaml --run-dir results/quality_gate_run
python3 docs-site/build_content.py
(cd docs-site && hugo --cleanDestinationDir)

Expected outcome: all gates complete with exit code 0, with no unresolved release blockers.

Local package-check note:

  • DSAMdecomp remains an optional Suggests dependency. Canonical local rcmdcheck runs set _R_CHECK_FORCE_SUGGESTS_=false so release gates do not depend on installing a telemetry-enabled decomposition package.

Runner smoke-test expectations

Minimum release smoke expectations:

  1. scripts/check.R --smoke succeeds, proving at least one cheap Stan MCMC path, pooled deployment-artifact coverage on a real pooled fit, and tiny runner fit paths.
  2. validate command succeeds and writes metadata artefacts.
  3. run command succeeds and writes model, fitted/observed output, and diagnostics artefacts.
  4. Required runner artefact paths exist under results/quality_gate_validate/ and results/quality_gate_run/.

For matrix and exact artefact paths, use Runner Smoke Tests.

Evidence capture requirements

Before sign-off, capture:

  1. Full command logs and exit codes for all mandatory gates.
  2. Runner smoke artefacts from validate and run directories.
  3. Candidate commit hash and top changelog section.

Use Release Evidence Pack as the authoritative bundle contract.

Failure handling

  1. Any gate failure is a release blocker until resolved.
  2. Re-run the full failed gate after remediation.
  3. If rcmdcheck emits NOTE, record reviewer rationale explicitly.
  4. If runner artefacts are missing, inspect resolved config and outputs.* flags.