Reproducible Digital Research.
Reproducibility is not getting the same number forever. It is preserving enough of the research system to rebuild a result, determine why a rerun agrees or differs, and separate methodological failure from genuine change in the digital environment.
A result is reproducible when its production can be reconstructed.
Dynamic digital environments may change between runs. A valid reproduction therefore preserves the method while explicitly modeling source drift, access loss and time-dependent state.
Reproducible digital research is the controlled preservation and re-execution of a versioned research process—including questions, boundaries, inputs, snapshots, transformations, decision rules and outputs—so agreement and difference between runs can be independently traced and explained.
A different output may be correct when a new snapshot contains a real state change.
Inputs, versions, filters, transformations and decisions must also be recoverable.
Human inclusion decisions, unresolved states and interpretation rules also require records.
Nine invariants define what must be preserved.
The contract separates stable research logic from time-sensitive environmental state.
Question version
Preserve the exact operational question, claim form and intended decision.
Boundary version
Record inclusion, exclusion, language, geography, time and unit definitions.
Source identity
Resolve the source object, access route, version and observed location.
Snapshot state
Freeze or fingerprint eligible content at the declared observation time.
Collection rule
Preserve request parameters, pagination, ordering, retry and missing-state rules.
Transformation log
Record normalization, filtering, joining, calculation and manual revision steps.
Decision rules
Version thresholds, classifications, conflict treatment and reviewer overrides.
Output schema
Preserve field meanings, units, null behavior, sort order and expected artifacts.
Execution context
Record material environmental conditions that can change process behavior.
Freeze logic. Rebuild outputs. Explain every delta.
A reproduction run is not successful merely because it finishes. It must establish whether each divergence originates in the method, input state, execution, decision layer or digital environment.
Select baseline
Resolve the exact research run and result to reproduce.
Inspect manifest
Confirm required components and known limitations.
Load method
Recover question, boundary and decision versions.
Load inputs
Use baseline snapshots or declared new-time sources.
Repeat capture
Execute the same request, traversal and state rules.
Rebuild dataset
Apply the preserved sequence and calculations.
Reapply rules
Classify observations using the same decision model.
Calculate parity
Compare inputs, intermediate states and outputs.
Explain drift
Assign each delta to a traceable change event.
Issue outcome
Record reproducibility state, differences and limits.
Four operations. Every protocol remains visible.
Without scripting all protocols display in sequence. With scripting they operate as a focused control console.
Freeze everything needed to reconstruct the run
Connect every output to its contributing inputs
Rebuild the result under a declared rerun mode
Explain where and why the two runs diverge
The complete run, readable as one control object.
This synthetic manifest shows the minimum relationships needed to recover a topical coverage audit without pretending that a filename alone preserves the method.
One run identity. Every dependency declared.
The manifest does not need to store every source inside one object. It must resolve each required component unambiguously and preserve its version, location and role.
Secrets and access credentials are never part of the published record. Their required capability and access condition can be documented without exposing them.
RQ-COVERAGE / v1.3CORPUS-BOUNDARY / v2.1TOPICAL-REQ / v4.0 / 100 eligibleSNAP-20260801 / 124 documentsCOLLECT-WEB / v1.7extract → normalize → resolve → classify / v3.2COVERAGE-STATES / v1.2COV-MATRIX / v2.0 / 9 fieldssupported_coverage / 72.0%4 unknown freshness states / disclosedEvery finding has a path back to evidence.
The graph makes intermediate datasets and decision states visible instead of drawing a false direct line from source pages to a final percentage.
A changed output becomes useful when its cause is located.
The synthetic rerun uses the same topical requirement model one month later. Three source changes explain the result delta.
| COMPONENT | RUN A | RUN B | PARITY | DELTA ORIGIN | RESULT EFFECT | DECISION |
|---|---|---|---|---|---|---|
| Question version | v1.3 | v1.3 | IDENTICAL | None | None | Method stable |
| Requirement universe | 100 eligible | 100 eligible | IDENTICAL | None | Denominator stable | Method stable |
| Source corpus | 124 documents | 121 documents | CHANGED | 2 removed; 1 inaccessible | 5 evidence paths affected | Environmental drift |
| Transformation chain | v3.2 | v3.2 | IDENTICAL | None | None | Execution stable |
| Coverage decisions | 72 supported | 68 supported | −4 ITEMS | Lost valid evidence | −4.0 pp | Explained result drift |
| Freshness decisions | 4 unknown | 6 unknown | +2 UNKNOWN | Source dates removed | Uncertainty rises | Disclose limitation |
Five rerun outcomes. Only one is a hard failure.
Reproducibility language must identify whether equality was expected and which components changed.
Identical
Declared inputs, intermediate states and final outputs match under the comparison rule.
Equivalent
Non-material formatting or ordering differs while analytical values and decisions match.
Explainable drift
Output differs and every material delta maps to a documented source, method or context change.
Unexplained drift
Execution completes but one or more material differences cannot be located in available lineage.
Non-reproducible
Required inputs, rules, versions or decisions are missing, preventing controlled reconstruction.
Six measures. No single reproducibility score.
Each measure diagnoses a different part of the research package or rerun result.
Manifest completion
available required components ÷ required componentsShows whether the package contains everything its reproduction contract demands.
Input identity rate
resolved input objects ÷ declared input objectsTests whether source and snapshot references identify usable objects unambiguously.
Lineage completion
traceable output fields ÷ tested output fieldsMeasures whether each tested result can be reconstructed through its dependencies.
Decision agreement
matching decisions ÷ comparable decisionsExcludes cases whose inputs legitimately changed between observation windows.
Drift attribution
explained material deltas ÷ material deltasShows how much rerun difference can be connected to documented change events.
Unresolved divergence
unexplained material deltas ÷ compared outputsPreserves differences that remain open after available method and lineage checks.
Four reruns. Four different valid interpretations.
All values are synthetic and illustrate methodology rather than live external findings.
Eight omissions that break the reconstruction path.
These failures may leave a polished final report while making its production impossible to inspect or repeat.
Unversioned question
The analytical target changes without a new method identity.
Live-source substitution
A current page silently replaces the historical state used by the baseline.
Parameter loss
Location, language, ordering, pagination or filtering inputs are unavailable.
Hidden manual edit
A reviewer changes records outside the documented transformation chain.
Schema drift
A field changes meaning or null behavior without an output version change.
Outcome forcing
Rules are adjusted during rerun to reproduce the expected number.
Secret dependency
An undocumented private input materially influences the result.
Drift denial
A genuine digital state change is mislabeled as methodological inconsistency.
Every rerun should be auditable as its own research object.
The record links baseline and reproduction without overwriting either state.
Every method node. One controlled research route.
MTH/10 reconstructs the complete run. MTH/11 then defines how unresolved limitations, missing access and uncertainty constrain the final claims.