Vol. XV / Issue 03

The McKinnie Dispatch

Filed from the research cluster

DevOps archaeology Self-hosted infrastructure 2024 to 2025

A hosting question and a lot of Bash

The first cluster was mostly Bash.

I wasn't trying to become a Kubernetes person. Potential customers were asking about hosting, and I needed to know what self-hosting identity would cost before I committed to a SaaS identity bill.

Potential customers were asking whether we could host Archibus environments for them. That meant running identity too. Okta and Auth0 were the obvious options, but the numbers got uncomfortable fast when I modeled them against the deployments I had in mind.

Self-hosting identity looked like a way around that bill. The self-hosted subreddit led me to Authentik: open source, solid protocol support, and reasonable documentation. I used the research cluster to find out whether self-hosting worked or only looked good on paper.

It worked. Then I found out what I had agreed to own.

Mostly manual

Terminal coding agents weren't part of my workflow yet. I did the work by hand with Bash, kubectl, Helm, openssl, ingress configuration, DNS, certificate management, proxy headers, and a lot of log reading. Every deployment cycle meant running each step myself.

It was slow, but the slowness taught me the protocol. Generating and applying the SAML certificates by hand made the dependencies hard to miss. The SP certificate and IDP metadata had to agree. The assertion consumer URL had to match exactly. Cookies had to survive the redirect through the identity provider. Each mismatch failed differently, and most error messages didn't say which assumption had broken.

Exhibit A: Certificate generation as a shell step Identity work leaves files behind. Before the cluster agreed on anything, the SP certificate had to exist, load as a Kubernetes Secret, and match the metadata Authentik expected. Doing it by hand showed me exactly how those pieces depended on each other.
mkdir -p certs

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout certs/sp-key.pem \
  -out certs/sp-cert.pem \
  -days 3650 \
  -subj "/CN=archibus-saml-sp"

kubectl create secret tls archibus-saml-sp \
  --cert=certs/sp-cert.pem \
  --key=certs/sp-key.pem \
  --namespace archibus-dev \
  --dry-run=client -o yaml | kubectl apply -f -

Deploying an image had its own loop: build it, tag it with a timestamp, push it to the registry, bump the chart value, then run the Helm upgrade. That got the first deployment out and immediately raised the next problem.

Exhibit B: Build, push, bump, deploy A timestamp tag and a one-liner updated the chart value in place. It was easy to start and easy to outgrow. Soon I couldn't answer which version was running or when it had changed without checking by hand.
tag="$(date +%Y%m%d%H%M%S)"
image="registry.example.com/platform/component:${tag}"

docker build -t "${image}" .
docker push "${image}"

yq -i ".image.tag = \"${tag}\"" ./charts/component/values.yaml

helm upgrade --install archibus-sso ./charts/component \
  --namespace archibus-dev \
  --values ./charts/component/values.yaml

The script handled the first deployment. Then I had to ask what was actually running and whether it matched the repo. After enough broken deployments, GitOps started to feel like relief instead of overhead.

The first SAML win

The concrete goal was getting Archibus running in Kubernetes with SAML SSO through a self-hosted Authentik installation. Getting there was not clean. SAML behind a Kubernetes ingress has real complexity: HTTPS URL reconstruction, protocol handling, certificate trust, cookie forwarding through multiple layers. The SP configuration has to know about its own public URL and handle the redirect loop correctly, or the login flow breaks in ways that are hard to diagnose from logs alone.

A direct OIDC path was available in theory. The working path was simpler. One component handled the authentication exchange, and the application server received identity through request headers. It was less interesting on an architecture diagram and more reliable for an upstream app that didn't speak OIDC natively.

Exhibit C: The boring proxy was the right answer Accept identity headers only from the trusted auth proxy, and strip any copies sent by the client. Let the proxy handle OIDC or SAML. That plain boundary held up in later environments too.
trust_boundary:
  public_entry: ingress
  client_identity_headers: stripped
  accepted_identity_source: auth-proxy
  direct_application_access: blocked

When SAML SSO finally worked end to end for an Archibus environment, I was relieved. By then, "hosting this for a customer" had a much bigger scope. It meant running the application server, identity layer, certificates, ingress, DNS, database, backups, and the upgrade order that kept them working together.

What you are actually buying

Self-hosting does not remove the cost of identity. It converts the cost.

A SaaS identity bill is easy to see. It shows up as a number with a unit at the end of the month. The self-hosted bill is scattered across operator time, certificate rotation before it expires, ingress debugging when something changes upstream, and the runbook needed to recreate the setup six months later. The cost is still real. It's just harder to track because it doesn't arrive as an invoice.

The invoice I wanted to avoid was Okta. The invoice I actually picked up was operational responsibility.

For this use case, the trade-off was worth it. The benefits were hosted customer environments, control over the identity stack, and room to shape the product around the hosting model. Later architecture choices came back to the same question: which costs am I willing to own, and am I building the systems to manage them?

The manual work also showed me what GitOps was for. After enough kubectl apply sessions, I kept asking three questions: what is deployed now, why did it drift from the repo, and how do I recreate the environment from a clean state?

Exhibit D: The first control loop Apply the bootstrap manifest, poll until the cluster agrees. Manual operations starting to look like reconciliation. This is where ArgoCD started making sense as the next step, and where the Bash scripts started looking like something to replace rather than maintain.
kubectl apply -n argocd -f bootstrap-app.yaml

for attempt in $(seq 1 30); do
  status="$(argocd app get platform-bootstrap -o json | jq -r '.status.health.status')"
  sync="$(argocd app get platform-bootstrap -o json | jq -r '.status.sync.status')"

  if [ "$status" = "Healthy" ] && [ "$sync" = "Synced" ]; then
    break
  fi

  sleep 10
done

ArgoCD came first, then Flux in later environments. Both answered questions the manual phase had made concrete. The Bash scripts came first because I needed to feel what each command cost before replacing it. The research cluster taught me the system, and the bill made sure I paid attention.