Kubernetes Secrets Done Right: Vault, OpenBao and the External Secrets Operator

Kubernetes Secret objects are one of the most misunderstood primitives in the API. They look like a security feature, but by default they are just base64-encoded blobs stored in etcd, readable by anyone with get secrets in the namespace, and completely static. Add GitOps into the mix — where the whole cluster state is supposed to live in Git — and you get the classic chicken-and-egg problem: how do you declare a secret you must never commit?

The combination of HashiCorp Vault (or its open-source fork OpenBao) and the External Secrets Operator (ESO) has become the de facto answer for platform teams. This article covers how it actually works, how it compares to the alternatives, and the operational pitfalls nobody mentions in the quickstart.

What is actually wrong with native Secrets

To be fair, native Secrets are not useless — they are the delivery mechanism, and almost every tool in this space ultimately produces one. The problems are around them:

• Storage: unless you explicitly enable encryption at rest with a KMS provider in the API server configuration, secrets sit in etcd in plain base64. Anyone with an etcd backup has your database credentials.
• Access control: RBAC on secrets is namespace-wide and coarse. A workload that needs one secret often ends up in a namespace where a dozen others are readable.
• No lifecycle: there is no rotation, no TTL, no revocation. A leaked credential stays valid until a human notices.
• No audit trail: the API server audit log tells you a secret was read, not which value, nor who consumed it downstream.
• GitOps-hostile: you cannot commit them, so they become an out-of-band manual step that drifts.

The landscape of options

Approach	Source of truth	Rotation	Best for	Main drawback
Sealed Secrets	Git (encrypted)	Manual re-seal	Small clusters, no external secret store	Cluster-bound key, painful DR and multi-cluster
SOPS + age/KMS (Flux, kustomize)	Git (encrypted)	Manual	Teams already deep in GitOps, few secrets	Still a copy of the secret in Git, key distribution
Vault Agent Injector	Vault	Template re-render	Apps that can read files, dynamic secrets	Sidecar per pod, mutating webhook, app must reload
Secrets Store CSI Driver	Vault / cloud KMS	On remount	No Kubernetes Secret object at all	Volume-only, env var support requires secret sync anyway
External Secrets Operator	Vault / cloud provider	Polling + webhooks	Most platform teams, multi-backend	Still materialises a Kubernetes Secret

Our opinionated take: ESO is the default choice for a platform serving many teams, because it is declarative, backend-agnostic, and does not change how applications consume secrets. Vault Agent Injector is the right tool when you need short-lived dynamic credentials written to a file and your application already handles reload. The two can coexist in the same cluster.

Wiring Vault to Kubernetes

The key design decision is authentication. Do not hand ESO a long-lived Vault token. Use the kubernetes auth method, where Vault validates a ServiceAccount token via the cluster's TokenReview API.

vault auth enable -path=k8s-prod kubernetes

vault write auth/k8s-prod/config \
  kubernetes_host="https://kube-api.prod.internal:6443" \
  kubernetes_ca_cert=@ca.crt

# Least-privilege policy: one team, one path prefix
vault policy write payments-ro - <<EOF
path "kv/data/payments/*" {
  capabilities = ["read"]
}
path "database/creds/payments-app" {
  capabilities = ["read"]
}
EOF

vault write auth/k8s-prod/role/payments \
  bound_service_account_names="eso-payments" \
  bound_service_account_namespaces="payments" \
  policies="payments-ro" \
  audience="vault" \
  ttl=1h

Note the bound_service_account_namespaces: this is what makes multi-tenancy real. A ServiceAccount in fraud-detection cannot assume the payments role, even if a developer copies the YAML.

If you run Vault 1.21+ or OpenBao, prefer short-lived projected ServiceAccount tokens with a specific audience over the legacy long-lived token reviewer JWT. It removes a permanent credential from the cluster.

SecretStore vs ClusterSecretStore

ESO exposes two scopes. A SecretStore is namespaced and authenticates with a ServiceAccount in that namespace — perfect for tenant isolation. A ClusterSecretStore is global and convenient, but it authenticates once with a single identity, which means the operator becomes a confused deputy unless you lock it down with conditions.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-payments
  namespace: payments
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "kv"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "k8s-prod"
          role: "payments"
          serviceAccountRef:
            name: eso-payments

If you do use a ClusterSecretStore, constrain it:

spec:
  conditions:
    - namespaceSelector:
        matchLabels:
          tenant: payments

The ExternalSecret, and why templating matters

A naive ExternalSecret maps one Vault key to one Secret key. In practice, applications want a connection string, a .env file, or a JSON config. ESO's templating engine avoids a sidecar or an init container just to concatenate strings.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-payments
    kind: SecretStore
  target:
    name: payments-db
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .user }}:{{ .password }}@pg.internal:5432/payments?sslmode=require"
  data:
    - secretKey: user
      remoteRef:
        key: payments/db
        property: username
    - secretKey: password
      remoteRef:
        key: payments/db
        property: password

Use dataFrom with extract when you want every key of a Vault path, and find with a regex when you want to pull a family of secrets. Be careful: find at scale generates a lot of Vault LIST operations.

Dynamic secrets: where Vault actually pays off

Static KV secrets synced into Kubernetes are a convenience. The real security win is dynamic credentials — Vault creates a database user on demand with a TTL and revokes it automatically.

vault write database/roles/payments-app \
  db_name=pg-prod \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT app_rw TO \"{{name}}\";" \
  default_ttl="4h" max_ttl="24h"

ESO can consume this, but you must align refreshInterval with the TTL (typically refresh at around half the TTL) and make sure the application picks up the new value. Two patterns work:

• Reloader (Stakater) or ArgoCD's argocd.argoproj.io/hook: annotate the Deployment so a Secret change triggers a rolling restart. Simple, but a restart every few hours is not always acceptable.
• Application-side reload: watch the mounted file (projected Secret volumes are updated in place by the kubelet) and rebuild the connection pool. Best behaviour, requires app support.

If neither is possible, keep static credentials in KV and rotate them on a slower cadence with Vault's rotation features — a shorter blast radius is still better than nothing.

GitOps integration without the drift

With ArgoCD, the ExternalSecret is committed to Git; the generated Secret is not. Tell Argo to ignore it, otherwise you will see permanent OutOfSync noise:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  ignoreDifferences:
    - group: ""
      kind: Secret
      jsonPointers:
        - /data

A better pattern is to let ESO own the Secret entirely (creationPolicy: Owner) and never reference it as an Argo-managed resource. Also add a sync wave so the SecretStore is created before any ExternalSecret, and before the Deployments that mount the result — otherwise the first rollout fails on a missing Secret.

Operational pitfalls

The Secret still exists in etcd. ESO does not remove that risk; it manages the lifecycle. Enable etcd encryption at rest with a KMS provider, lock down RBAC on secrets, and disable the automounting of ServiceAccount tokens where not needed. If you truly cannot tolerate a Kubernetes Secret object, use the CSI driver or Vault Agent with a memory-backed volume.

Vault availability becomes a cluster dependency. ESO caches values in the Secret, so a Vault outage does not immediately break running pods — but new pods relying on freshly synced secrets will fail, and refreshes will stall. Run Vault in HA with Raft across availability zones, monitor seal status, and have a tested unseal runbook (auto-unseal via a cloud or on-prem KMS/HSM, with key shards stored offline).

Rate limiting and refresh storms. A thousand ExternalSecrets with refreshInterval: 1m will hammer Vault. Use realistic intervals (15m–1h for static values), and consider ESO's provider caching. Watch the externalsecret_sync_calls_error and externalsecret_status_condition metrics and alert on secrets that have not synced successfully for longer than their refresh interval.

Audit. Enable Vault's audit device and ship it to your SIEM. Combined with the ESO logs and Kubernetes API audit logs, you can answer "which workload read which credential, when" — which is exactly what an auditor will ask during an ISO 27001 or DORA review.

Sovereignty note: OpenBao

Since HashiCorp's licence change, many European teams have moved to OpenBao, the Linux Foundation fork of Vault. It is API-compatible for the core features discussed here — KV v2, Kubernetes auth, database secrets engine — and ESO supports it through the same Vault provider. If your compliance posture requires an OSI-licensed, self-hosted secret store running on European infrastructure, it is a credible drop-in. Validate the specific secrets engines you depend on before committing, as enterprise-only features (namespaces, HSM auto-unseal in some configurations) differ.

Where to start

Do not try to migrate everything at once. Pick one namespace, set up the Kubernetes auth mount with a scoped policy, convert its secrets to ExternalSecret objects, then delete the manual kubectl create secret runbook page. Once the pattern is proven, templatise it — a Helm library chart or a Crossplane/Terraform module that provisions the Vault policy, role, ServiceAccount and SecretStore for each new tenant is what turns this from a tool into a platform capability.