ADR-0030 — External Secrets Operator + cloud KMS (no plaintext in Git)
- Status: Deferred
- Date: 2026-06-11
- Related: platform/09 secrets, ADR-0014, ADR-0028
V1 Freeze (2026-06-12): Deferred. V1 uses 12-factor env / keyfile secrets; no External Secrets Operator. Re-opens at P4 (K8s/SaaS).
Context
GitOps (ADR-0028) puts desired state in Git — but secrets must never be in Git in plaintext (base64 is not encryption). We need a secrets flow that fits the GitOps pull model, supports centralized rotation, works across clusters, and integrates with the KMS already chosen for envelope encryption (ADR-0014).
Decision
External Secrets Operator (ESO) backed by a cloud Secrets Manager / Vault as the production primary:
- Git holds only references (
SecretStore+ExternalSecret); secret material lives in the manager and is synced into a Kubernetes Secret on the destination cluster (ArgoCD’s recommended model). - ESO authenticates via workload identity (IRSA / GKE WI / Azure WI) — no static credentials anywhere.
- SOPS (age/KMS) covers the few bootstrap/config secrets that must live in Git before ESO exists. Sealed Secrets is the self-host alternative (no external vault required, ADR-0012).
- etcd Secret encryption via a KMS provider (defense in depth); per-env stores/keys.
Consequences
Positive
- No secret material in version control; Git diffs are safe to share.
- Centralized rotation: update in the manager → ESO re-syncs (no code redeploy).
- Works across clusters; least-privilege via scoped stores; every read is audited.
- Reuses the KMS/workload-identity already provisioned (ADR-0014, ADR-0031).
Negative / costs
- Requires a secret manager (cost) + a controller (overhead) + a dependency on manager availability (mitigated: synced Secrets cached in etcd, so running pods survive a transient manager outage).
Alternatives considered
- Sealed Secrets (primary): cluster-bound keys, poor multi-cluster rotation; kept as the self-host alternative, not the SaaS primary.
- SOPS-only: good for encrypted-in-Git config/bootstrap; CLI-centric, weak runtime injection. Used for bootstrap only.
- Plaintext/base64 in Git: forbidden.
- Vault Agent sidecar injection: powerful (dynamic/leased creds) but heavier; ESO suffices now; revisit for dynamic DB credentials.
Scaling
A single managed source + ESO scales linearly across secrets/envs/clusters
(ClusterSecretStore shared, SecretStore scoped); per-env keys bound rotation blast
radius.