Reproducibility is an infrastructure property, not a documentation task

Documentation can explain how a system was built. Reproducibility proves that the organisation can build it again from controlled inputs, without borrowing invisible state from the environment or the person who created it.

A platform can have excellent documentation and still be impossible to reproduce.

The diagrams are current. The repositories are organised. The installation guide has been reviewed. Every component has an owner.

Then a team attempts to create the system in a clean environment.

A package has disappeared. A network rule was added manually. The deployment expects a secret that nobody can regenerate. The base image has changed. A bootstrap script reaches an internal service that no longer exists. The documentation describes the intended architecture, but the running platform contains years of unrecorded decisions.

Nothing is missing from the document.

What is missing is a controlled path from evidence to working infrastructure.

The central test

A system is reproducible when a qualified team can reconstruct a known operating state from controlled source, declared dependencies, protected recovery material and an executable procedure.

A document can describe a system that no longer exists

Documentation records knowledge. Reproducibility constrains reality.

That distinction matters because documents can remain plausible long after the platform has drifted away from them. A diagram can show the correct components while omitting the order in which they must be created. A runbook can list commands without recording which versions were used. A configuration guide can explain the standard path while production depends on an exception added during an incident.

The document may not be wrong. It may simply be incomplete in ways that become visible only during reconstruction.

Common mistake

Treating detailed documentation as proof that another team can recreate the system.

Better framing

Use documentation to explain the architecture, then use a clean reconstruction to prove that the explanation, inputs and procedures are sufficient.

Reproducibility does not make documentation less important. It gives documentation a harder job.

Instead of merely describing components, the documentation must explain boundaries, assumptions, inputs, ordering, failure conditions and acceptable variation. It must help someone understand why the procedure works, not serve as a substitute for testing whether it works.

Reproducibility begins with controlled inputs

A reproducible platform starts with a clear answer to a simple question:

What is allowed to influence the result?

The answer usually includes more than source code.

Infrastructure definitions, configuration, dependency versions, base images, package repositories, policies, secrets, certificates, identity mappings and external services may all affect the final state.

If those inputs are not declared, they have not disappeared. They have become environmental assumptions.

Controlled source
      |
      v
Declared dependencies
      |
      v
Build and provisioning
      |
      v
Configuration and policy
      |
      v
Known operating state
      |
      v
Verification evidence

The chain is only as reproducible as its least controlled link.

Version-controlled infrastructure code cannot compensate for an untracked base image. A pinned application release cannot compensate for manually configured identity. A complete deployment pipeline cannot compensate for a secret that can be copied but not regenerated or recovered through an organisational process.

The goal is not to freeze every dependency forever. Infrastructure must change.

The goal is to make change deliberate enough that the organisation can identify which inputs created a particular result and which substitutions remain acceptable.

Rebuild, redeploy and restore are different operations

Teams often use the word “rebuild” for several different activities.

A redeployment places an existing artifact into a new or existing environment.

A rebuild creates a new artifact or environment from declared inputs.

A restore returns protected state—such as databases, object data or configuration records—to a usable point.

A functioning service may require all three.

The infrastructure can be rebuilt successfully while the service remains unusable because its data was not restored. Data can be restored successfully while the application version required to read it cannot be reproduced. An application can be redeployed while identity, network policy and certificates still depend on the failed environment.

This is why a recovery plan that says “redeploy from the repository” is usually incomplete.

The organisation needs to know which parts are regenerated, which are restored, which are imported from independent systems and which require a deliberate replacement.

Operational distinction

Reproducibility answers whether the environment and software can be recreated. Recoverability answers whether the service, including its necessary state and authority, can be returned to use.

The two capabilities support each other, but they are not interchangeable.

Drift is a decision even when nobody made it

A reproducible system has an intended state.

A live system has an observed state.

The difference between them is drift.

Some drift is harmless. A platform may create timestamps, runtime identifiers or temporary capacity that should not be reproduced exactly. Other drift represents an operating decision: a firewall exception, a manually increased limit, an emergency package change or an identity grant added outside the normal process.

When those changes remain only in the live environment, the current platform becomes the sole record of its own design.

That is fragile for two reasons.

First, reconstruction from source produces something different from production.

Second, nobody can say with confidence whether the difference is intentional.

A strong drift process does not blindly force every system back to a declared configuration. It distinguishes expected runtime variation from undocumented changes that affect security, continuity or behaviour.

The important outcome is evidence:

what differs
when it changed
who or what changed it
whether the difference is accepted
how the declared state will be updated or the drift removed

Without that evidence, drift is not merely a technical discrepancy. It is an architecture decision without an owner.

Provenance connects source to the running system

A repository can contain the right code while the running service came from somewhere else.

An operator may have built an artifact locally. A pipeline may have used a different branch. A container may have been replaced without updating the release record. A package mirror may have supplied a changed dependency under the same name.

Reproducibility therefore needs provenance: a traceable connection between declared inputs, produced artifacts and deployed state.

The organisation should be able to answer:

Which source revision produced this release? Which dependency set was resolved? Which process built it? Which checks were performed? Which artifact was deployed? Which configuration and policy were applied afterward?

This does not require an elaborate certification system for every internal tool.

It requires enough evidence to distinguish a controlled release from an artifact whose origin is assumed.

The same principle applies to infrastructure.

If a virtual machine image, cluster template or network policy is treated as a reusable artifact, the organisation should know how it was created and whether it can be created again.

A binary object without provenance may be convenient to restore. It is weak evidence of reproducibility.

Secrets and external services define the real boundary

Reconstruction procedures often become vague at the point where protected material enters the system.

A document says to “add the production credentials” or “restore the certificates”. That instruction hides several ownership questions.

Who can obtain them? Can they be regenerated? Are they protected outside the environment being rebuilt? Do they depend on the same identity system that has failed? Which services must trust the new instance before it can operate?

Secrets should not be placed in source merely to make a process appear self-contained. Reproducibility does not mean that every input is public or stored in one repository.

It means protected inputs have known custody, recovery procedures and interfaces.

External services need the same treatment.

A build may depend on a package registry. A deployment may depend on a certificate authority. An application may require a directory, payment service, research data source or institutional network.

Those dependencies do not need to be reproduced locally. They do need to be declared and tested at the boundary.

Otherwise, the reconstruction succeeds only in an environment that happens to contain the same invisible surroundings.

The clean environment is the honest reviewer

The most useful reproducibility test is intentionally ordinary.

Create a disposable environment. Give the work to a qualified person who does not maintain the original platform. Provide only organisational repositories, approved build services, protected recovery material and the documentation expected to survive staff turnover.

Then observe what they need to ask.

Do they need a file from someone’s laptop? Do they need to copy a value from production? Do they rely on a package that can no longer be obtained? Do they know which deviations are expected? Can they verify that the resulting service corresponds to the intended release?

Operational test

Reconstruct a representative environment from controlled inputs without cloning the live system. Record every undeclared dependency, manual intervention, ambiguous instruction and unverifiable artifact.

The purpose is not to embarrass the original authors.

The missing assumptions are the result.

A failed rehearsal performed while the current system and its maintainers are available is useful evidence. The same failure during an outage or staff transition is an operational crisis.

A practical reproducibility review

Area	Weak signal	Stronger evidence
Source	Repositories exist	Approved revisions and clear ownership of every required source
Dependencies	Installation works today	Declared versions, trusted retrieval paths and known replacement rules
Build	A pipeline produces artifacts	A clean build with traceable inputs and recorded verification
Infrastructure	Configuration is stored as code	A new environment created without copying undocumented live state
Secrets	Credentials are backed up	Organisational custody, independent recovery and regeneration procedures
Drift	Production appears stable	Recorded differences between declared and observed state
Data	Backups are present	Compatible state restored into the reconstructed service
Verification	The service starts	Defined functional, security and recovery checks pass
Transfer	The maintainer can rebuild it	A different qualified operator completes the reconstruction

The review should focus on the systems whose loss would threaten continuity.

Not every temporary development environment needs the same reconstruction standard as identity, storage, publication or research infrastructure. Reproducibility should reflect the consequence of failure.

The important point is that the standard is chosen deliberately.

Reproducibility is maintained through use

A reconstruction procedure decays when it is not exercised.

Dependencies move. Certificates expire. package repositories change. Staff leave. External interfaces are replaced. Documentation becomes inaccurate without anyone deciding to make it inaccurate.

Reproducibility is therefore not a document completed at the end of a project.

It is an infrastructure property that must be kept alive through repeated builds, environment creation, restore exercises and transfer rehearsals.

A system that was reproducible two years ago may not be reproducible today.

A system that is continuously rebuilt from controlled inputs provides stronger evidence because the reconstruction path participates in normal operation. Disposable environments, regular release builds and recovery exercises can all expose breakage before the path is urgently needed.

The goal is not perfect duplication of every historical machine.

It is confidence that the organisation can recreate a known, supportable state with evidence it controls.

One question for the next architecture review

Do not ask whether the platform is documented.

Ask:

If the current environment disappeared tonight, could a different team recreate a verified service from controlled inputs without copying any unexplained state from the system that was lost?