Vol. XV / Issue 04

The McKinnie Dispatch

Filed from the dev cluster

DevOps archaeology Persistent environments 2025 to 2026

A working cluster and a team that still needed me to operate it

Making the cluster less dependent on me.

Archibus was running in Kubernetes. Now someone other than me had to be able to operate it.

The research cluster proved that Archibus could run in Kubernetes with a self-hosted identity layer and real SAML SSO. It also exposed the next problem: I was the only person who could operate it.

A team platform can't depend on one person's shell history. When a shared dev environment drifted or broke, someone had to find me. I handled each deployment. Anyone trying to work out what was running, or why, needed the context in my head. kubectl wasn't a team interface under those conditions.

Runbooks helped, but they couldn't carry the whole job. Cluster state had to be visible enough for another person to read. Changes needed a path that worked without me sitting at the terminal.

A newspaper-style technical illustration of one operator using a laptop terminal to drive multiple Kubernetes environments through tangled command paths.
Plate I: The private control plane Before the GitOps pass, the working knowledge stayed too close to the person running the commands. The cluster existed, but memory, shell history, and direct intervention still ran it.

Making Kubernetes less private

ArgoCD gave the team visibility. From its UI, someone could see what was deployed, what changed, whether it was healthy, whether it matched the source repo, and which branch it tracked. A shell session from last week couldn't answer those questions for a person who wasn't there.

First, branch names got operational meaning. dev and staging deployed automatically. main needed a manual gate before touching production. Codifying that rule turned it into a workflow instead of a convention people had to remember.

Exhibit A: Branch names as environment contracts Each branch name says which environment a change should affect and whether the controller can apply it automatically or must wait for a person. That makes the config reviewable as well as executable.
environments:
  - name: dev
    branch: dev
    autoSync: true
  - name: staging
    branch: staging
    autoSync: true
  - name: prod
    branch: main
    autoSync: false

The branch mapping still needed a repeatable environment shape. ApplicationSets let one template generate several environment-specific Applications from a list. A new persistent Archibus dev or test environment now followed the same pattern as the older ones. Nobody had to rebuild a one-off shell session from memory.

Exhibit B: One pattern, multiple environments Persistent Archibus environments now followed a known, inspectable pattern. Adding one meant adding an item to a list instead of running another command sequence and hoping the result matched.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  generators:
    - list:
        elements:
          - env: dev
            branch: dev
            namespace: archibus-dev
          - env: test
            branch: test
            namespace: archibus-test
  template:
    spec:
      source:
        targetRevision: "{{branch}}"
        path: overlays/archibus/{{env}}
      destination:
        namespace: "{{namespace}}"

The CI pipeline made the handoff concrete. Instead of running kubectl apply directly, it pushed to the source branch and asked the GitOps controller to reconcile. Direct apply had made the CI runner the deployment interface. The sync request moved that job to the controller, where status, history, and drift were visible.

Exhibit C: Sync as an explicit handoff The controller now handled deployment and kept an audit trail with visible current state. Direct kubectl apply had left the useful history inside one person's shell session.
git push origin dev

# CI validates the change, then asks the GitOps controller to reconcile.
argocd app sync archibus-dev
argocd app get archibus-dev

The early ArgoCD work also added explicit RBAC, team branch-workflow docs, and a clear split between environments that auto-deployed and those that needed approval. Source and controller status could now answer questions that had gone to whoever touched the cluster last.

A newspaper-style technical architecture diagram showing developers pushing to Git, CI validation, a GitOps controller, Kubernetes environments, a health board, and staged dependency layers.
Plate II: Source, status, and shared ownership The team could read the source, see controller status and health, and follow a repeatable environment pattern. I no longer had to be the deployment interface.

What it became

The model held through the dev cluster work and changed again in later environments. Those later environments eventually moved from the original ArgoCD pattern to Flux.

The order became explicit: base platform resources, supporting services, application releases, then follow-up jobs. That dependency graph lives in source. Nobody has to find the person who remembers the original decisions.

Exhibit D: Explicit dependency order The dependsOn fields say which resources must be ready before the next ones run. With that order in a manifest, recovery doesn't require the original architect to recite it.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: environment-helmreleases
spec:
  dependsOn:
    - name: environment-platform
    - name: environment-secrets
  wait: true
  path: ./flux/environment/helmreleases

That dependency order took about a year of environment work to earn. Bash scripts from the research cluster made the manual steps readable. The ArgoCD experiments exposed the gap between "what is deployed" and "what the repo says." Flux staging put the full bootstrap order somewhere anyone could inspect without a conversation.

I could stay in the system. I just couldn't keep being its only API.

The question hasn't changed since the first ArgoCD experiment. Can someone understand and repair an environment from source and controller status, or do they have to ask me? The parts that held up put the answer in source instead of one person's memory.