LibreInfra Field Notes

Kubernetes is an orchestration engine, not an operating model

Clusters schedule workloads; the operating model decides ownership, recovery, upgrades and transfer.

Abstract line-and-node platform artwork representing Kubernetes control planes and workload boundaries.
Pexels ↗

Kubernetes is an orchestration engine, not an operating model

Clusters can schedule containers and replace failed workloads. They do not decide who owns the platform, how state is recovered, which controllers may act, or whether another team can operate the system after its original builders leave.

A Kubernetes cluster can be healthy while the platform around it is already failing.

Nodes are ready. Workloads are running. The control plane responds. Dashboards show no urgent alerts.

But the ingress controller is several releases behind. Nobody knows whether the storage driver can survive the next upgrade. Production secrets are loaded through a process owned by one engineer. Backups protect cluster objects but not the application data. A collection of operators has accumulated privileges across every namespace.

The cluster is working.

The operating model is not.

The central test

Kubernetes becomes ownable infrastructure when the organisation can define the platform contract, control its extensions, recover its state, upgrade it deliberately and transfer operations without depending on the team that assembled it.


A cluster solves a narrower problem than a platform

Kubernetes coordinates containerised workloads.

It schedules them, monitors declared state, replaces failed instances, exposes service abstractions and provides APIs through which other systems can manage the environment.

That is substantial capability.

It is still only part of a usable platform.

A production environment also needs decisions about:

  • identity and administrative authority
  • network policy and ingress
  • persistent storage
  • secrets and certificates
  • container images and registries
  • policy enforcement
  • monitoring and evidence
  • backups and recovery
  • node lifecycle
  • upgrades
  • workload ownership
  • incident response

Kubernetes provides interfaces into many of these concerns. It does not choose the operating model for the organisation.

Common mistake

Treating the successful installation of a cluster and several add-ons as the completion of a platform.

Better framing

Treat the cluster as one control plane inside a larger service whose responsibilities, boundaries and recovery path must be designed explicitly.

The distinction matters because add-ons are not decorations. They often carry the responsibilities that make the platform usable.

A failure in identity, storage, networking or certificate management can stop the service even when Kubernetes itself remains available.

The real platform surface is larger than the API server

The Kubernetes API is the visible centre of the system.

Around it sits a wider platform surface:

Organisational identity
          |
          v
Kubernetes control plane
          |
   ┌──────┼────────┐
   v      v        v
Network  Storage  Policy
   |      |        |
   v      v        v
Ingress  Data     Admission
   |
   v
Workloads and services
   |
   v
Evidence, recovery and transfer

Each connection introduces another operating responsibility.

The network plugin determines how workloads communicate. The storage layer determines where persistent state lives and how volumes are attached. Admission controllers may approve or reject deployments. Operators can create resources, change configuration and act with broad privileges.

A cluster may therefore contain several control planes, each maintained through a different release process.

The organisation needs to know which component is authoritative for each responsibility.

Without that map, incidents become arguments between layers. The application team sees a storage failure. The platform team sees a healthy cluster. The storage system sees a valid request. Nobody owns the complete path.

Controllers are privileged software, not convenient extensions

Kubernetes encourages automation through controllers.

A controller observes resources, compares them with desired state and acts until the two match. This pattern makes the platform extensible and powerful.

It also means that installing an operator or controller gives software continuing authority inside the environment.

A certificate controller may create and renew credentials. A database operator may modify persistent workloads. A GitOps controller may apply changes across many namespaces. An autoscaler may add infrastructure. A policy controller may prevent or permit deployment.

These components should be reviewed as privileged automation.

The important questions are not only:

  • Does the controller provide a useful feature?
  • Is the project popular?
  • Does installation succeed?

The review should also ask:

  • Which resources can it read and change?
  • Which credentials does it hold?
  • What happens if its logic is wrong?
  • Can it act across environments?
  • How is its own state recovered?
  • Which version combinations are supported?
  • Can it be removed without leaving unmanaged resources?

An extension that cannot be upgraded, replaced or safely disabled has become part of the platform’s permanent operating burden.

Workloads need a contract with the platform

Application teams need to know what the platform guarantees.

Without a clear contract, every workload arrives with different assumptions.

One application expects persistent volumes to survive node loss. Another assumes local storage is temporary. One expects outbound internet access. Another requires internal-only networking. Some applications expect the platform to manage database recovery. Others bring their own stateful services and assume snapshots are sufficient.

The result is a cluster full of incompatible expectations.

A useful platform contract defines:

  • supported workload types
  • identity and access model
  • network boundaries
  • storage classes and durability expectations
  • secret delivery
  • deployment and rollback paths
  • observability requirements
  • backup responsibilities
  • recovery objectives
  • resource and scaling limits
  • support and escalation boundaries

The contract does not need to remove every exception.

It needs to make exceptions visible.

A workload that requires direct node access, unusual privileges or an unsupported storage pattern may still be accepted. The decision should identify who owns the additional risk and how the service will be recovered.

Stateful workloads expose the ownership boundary

Containers are replaceable.

State is not.

A deployment can recreate an application process, but it cannot recreate a database, a message log, an index or an uploaded document unless those states are protected independently.

Kubernetes objects describe part of the system. Persistent data usually lives elsewhere.

Backing up the cluster API may preserve namespaces, deployments, service definitions and policies. It may not preserve the contents of attached volumes. A storage snapshot may preserve blocks while failing to create an application-consistent recovery point. Replication may protect current availability while copying corruption or administrative error.

Recovery requires coordination between layers.

Operational test

Rebuild a clean cluster and restore one representative stateful service using only organisational repositories, protected data, documented credentials and the normal recovery procedure.

The exercise should answer:

  • Can the cluster be recreated?
  • Can required controllers and policies be restored?
  • Can the correct application version be deployed?
  • Can persistent state be recovered?
  • Can identity, DNS and certificates be re-established?
  • Can the service be verified before users return?

A cluster backup is useful.

It is not automatically a service recovery plan.


Upgrades are part of the architecture

Kubernetes platforms contain several release cycles.

The control plane changes. Worker-node images change. Networking and storage plugins change. Operators, admission policies, observability components and workload APIs change.

These cycles are connected.

An upgrade may be supported by Kubernetes while remaining incompatible with a storage driver or operator. A deprecated API may still be used by an internal workload. A node-image change may affect kernel behaviour or container networking.

This means platform lifecycle cannot be delegated to a calendar reminder.

The organisation needs:

  • an inventory of platform components
  • supported version relationships
  • pre-upgrade tests
  • representative workloads
  • migration paths for deprecated interfaces
  • rollback or forward-recovery decisions
  • evidence from the completed change

A platform that cannot be upgraded safely becomes a frozen dependency.

Eventually, the choice is no longer between upgrading and remaining stable. It is between a controlled change and an emergency migration from an unsupported estate.

GitOps does not remove the need to understand live state

GitOps can provide a disciplined path from reviewed source to cluster configuration.

The repository records declared state. Controllers reconcile that state into the environment. Changes can pass through version control and review.

That improves inspectability.

It does not make the repository the only state that matters.

Controllers have credentials. Secrets may come from another system. Some resources are created dynamically. Operators maintain internal state. Emergency changes may occur outside the normal path. External systems may alter infrastructure the repository describes but does not control.

Git is a strong source of intended state.

The cluster remains the source of observed state.

The platform needs a controlled way to compare the two.

If operators assume that every difference is malicious drift, they may destroy valid runtime state. If they assume the repository tells the whole story, they may miss changes made by privileged controllers or external systems.

Reconciliation is an operating process, not proof that the architecture is understood.

More clusters do not automatically create more resilience

When one cluster becomes difficult to manage, adding clusters can appear to reduce risk.

Separate clusters can improve isolation. They can limit blast radius, support different trust levels and allow recovery across failure domains.

They also multiply control planes, upgrades, credentials, policies, registries, observability paths and recovery procedures.

The useful question is not whether the organisation should have one cluster or many.

It is what boundary each cluster creates.

A separate cluster may be justified by:

  • production isolation
  • legal or data boundaries
  • independent site operation
  • different lifecycle requirements
  • recovery architecture
  • workload risk
  • administrative separation

A cluster created merely because the existing one is confusing reproduces confusion at a larger scale.

Every cluster should have a purpose, an owner and a retirement condition.


A practical Kubernetes platform review

Area Weak signal Stronger evidence
Platform boundary A cluster diagram exists Every control plane, external dependency and owner is mapped
Identity Administrators can log in Organisational authority, scoped roles and tested emergency access
Extensions Add-ons are installed Privileges, lifecycle, state and removal paths are understood
Workload contract Teams can deploy containers Supported storage, networking, security and recovery responsibilities are explicit
State Persistent volumes are available Application-consistent protection and tested restoration exist
Supply chain Images come from a registry Approved source, provenance, scanning and rebuild paths are controlled
Upgrades Versions are monitored Compatibility tests, migration decisions and completion evidence exist
Drift Git contains configuration Declared and observed state are compared with controlled exceptions
Recovery Cluster objects are backed up A clean cluster and representative service have been reconstructed
Transfer The platform is documented Another team can upgrade, recover and operate it

The platform does not need to solve every infrastructure problem.

It needs to state clearly which problems it does solve, which it delegates and which remain the responsibility of workload teams.

The cluster should disappear from the ownership question

The strongest Kubernetes platform is not the one with the most automation or the largest catalogue of extensions.

It is the one whose operating responsibilities remain visible.

Applications know what the platform guarantees. Controllers have bounded authority. State is protected according to its real failure modes. Upgrades occur through evidence rather than hope. Another qualified team can reconstruct and operate the environment.

At that point, Kubernetes is doing what it should do.

It orchestrates workloads without becoming an unexplained dependency at the centre of the organisation.

One question for the next platform review

Do not ask whether the cluster is healthy.

Ask:

If the cluster, its current administrators and the GitOps controller disappeared together, could another team reconstruct the platform and recover one stateful production service from evidence the organisation controls?

Make the next decision with clarity

Use the note as a starting point, not a substitute for context.

Open consultation form