Kubernetes Secrets Management: From Leaky to Bulletproof

Co-founder & CPTO

On this page
- The Problem: Secrets Everywhere
- Kubernetes Native Secrets: The Starting Point
- Solution 1: Sealed Secrets - Git-Safe Encryption
- Solution 2: HashiCorp Vault - Enterprise Grade
- Development Workflow: Making It Developer-Friendly
- Security Best Practices
- Choosing the Right Solution
- Manual Secrets Management (Option 3)
- Integration with GitOps
- What's Next?
Secrets management is where most Kubernetes deployments go wrong. You've probably seen it: passwords hardcoded in YAML files, API keys committed to Git, or worse—production credentials shared in Slack channels.
This guide shows you how to build a secrets management system that's both secure and developer-friendly. No more choosing between security and productivity.
The Problem: Secrets Everywhere
Modern applications need secrets everywhere:
- Database connection strings
- API keys for third-party services
- TLS certificates
- OAuth client secrets
- Service account tokens
The challenge? These secrets need to be:
- Secure - Never exposed in plain text
- Accessible - Available to applications at runtime
- Rotatable - Easy to update without downtime
- Auditable - Track who accessed what and when
- Developer-friendly - Simple to use in development and CI/CD
Kubernetes Native Secrets: The Starting Point
Kubernetes has built-in Secret resources, but they're not actually secret:
apiVersion: v1kind: Secretmetadata: name: app-secretstype: Opaquedata: database-url: cG9zdGdyZXM6Ly91c2VyOnBhc3NAZGIuZXhhbXBsZS5jb20vbXlkYg== api-key: YWJjZGVmZ2hpams=The problem? Those values are just base64 encoded, not encrypted. Anyone with cluster access can decode them:
kubectl get secret app-secrets -o jsonpath='{.data.database-url}' | base64 -d# postgres://user:pass@db.example.com/mydbNever commit Kubernetes Secrets to Git. They're not encrypted and provide no real security.
Solution 1: Sealed Secrets - Git-Safe Encryption
Sealed Secrets solve the "secrets in Git" problem by encrypting secrets that only your cluster can decrypt.
🤖 Sealed Secrets Setup
First, ask me about my secrets management needs before generating any code:- What types of secrets do you need to manage (database URLs, API keys, certificates, etc.)?- How many environments do you have (dev, staging, prod)?- What operating system are you using for the kubeseal CLI?- Do you have existing secrets to migrate?- What is your preferred backup strategy for encryption keys?
Then set up sealed-secrets for Kubernetes secrets management. I need:
- Complete installation via Helm/Flux- Step-by-step process to encrypt secrets for Git storage- Examples of converting environment files to encrypted secrets- Integration with application deployments- Backup and recovery procedures for encryption keys- Rotation strategies for both secrets and encryption keys
Important: All Kubernetes manifests should go in the /k8s directory and use FluxCD HelmRelease format for GitOps deployment.Installing Sealed Secrets
apiVersion: helm.toolkit.fluxcd.io/v2beta1kind: HelmReleasemetadata: name: sealed-secrets namespace: kube-systemspec: interval: 30m chart: spec: chart: sealed-secrets version: '2.15.x' sourceRef: kind: HelmRepository name: sealed-secrets namespace: flux-system values: fullnameOverride: sealed-secrets-controllerCreating Encrypted Secrets
Install the kubeseal CLI:
# macOSbrew install kubeseal
# Linuxwget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gztar -xvzf kubeseal-0.24.0-linux-amd64.tar.gzsudo install -m 755 kubeseal /usr/local/bin/kubesealCreate secrets from environment files:
# Create a .env file (never commit this!)cat > .env.production << EOFDATABASE_URL=postgres://user:password@db.example.com/myappREDIS_URL=redis://redis.example.com:6379API_KEY=super-secret-api-keySTRIPE_SECRET_KEY=sk_live_...EOF
# Convert to Kubernetes secret and encryptkubectl create secret generic app-secrets \ --from-env-file=.env.production \ --dry-run=client -o yaml | \ kubeseal --format yaml > k8s/secrets/app-secrets-sealed.yaml
# Clean up the plain text filerm .env.productionThe result is a SealedSecret that's safe to commit:
apiVersion: bitnami.com/v1alpha1kind: SealedSecretmetadata: name: app-secrets namespace: defaultspec: encryptedData: DATABASE_URL: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQAx... REDIS_URL: AgAKAoiQm7QDhDqf0jRqm6oOHLwDeHBb16... API_KEY: AgAjklfmn2j3h4g5f6d7s8a9q0w1e2r3t4... template: metadata: name: app-secrets namespace: defaultUsing Encrypted Secrets in Deployments
apiVersion: apps/v1kind: Deploymentmetadata: name: my-appspec: template: spec: containers: - name: app image: my-app:latest envFrom: - secretRef: name: app-secrets # Or individual environment variables: env: - name: DATABASE_URL valueFrom: secretKeyRef: name: app-secrets key: DATABASE_URLKey Management and Rotation
Backup your encryption keys:
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-master-key.yamlStore this backup securely (password manager, encrypted storage, etc.). Without it, you can't decrypt your secrets if you rebuild your cluster.
Rotate secrets regularly:
# Update your .env file with new valuescat > .env.production << EOFDATABASE_URL=postgres://user:new-password@db.example.com/myappAPI_KEY=new-super-secret-api-keyEOF
# Re-encrypt and commitkubectl create secret generic app-secrets \ --from-env-file=.env.production \ --dry-run=client -o yaml | \ kubeseal --format yaml > k8s/secrets/app-secrets-sealed.yaml
git add k8s/secrets/app-secrets-sealed.yamlgit commit -m "rotate: update application secrets"git push
# Clean uprm .env.productionSolution 2: HashiCorp Vault - Enterprise Grade
For maximum security and compliance, Vault provides enterprise-grade secrets management with dynamic secrets, detailed auditing, and fine-grained access control.
🤖 HashiCorp Vault Setup with FluxCD
First, ask me about my Vault requirements before generating any code:- Do you need high availability (multiple Vault replicas)?- What storage backend do you prefer (integrated storage, Consul, etc.)?- What types of secrets will you store in Vault?- Do you have compliance requirements (audit logging, etc.)?- How many applications need to integrate with Vault?- What are your backup and disaster recovery requirements?
Then set up HashiCorp Vault in my Kubernetes cluster using FluxCD. I need:
- Complete Vault installation via Helm/Flux- Vault configuration for your availability requirements- Kubernetes authentication setup- Vault Secrets Operator for automatic secret injection- Examples of storing and retrieving secrets- Backup and disaster recovery procedures- Integration with existing applications
Important: All Kubernetes manifests should go in the /k8s directory and use FluxCD HelmRelease format for GitOps deployment.Installing Vault with FluxCD
apiVersion: v1kind: Namespacemetadata: name: vault---# k8s/vault-helm-repo.yamlapiVersion: source.toolkit.fluxcd.io/v1beta2kind: HelmRepositorymetadata: name: hashicorp namespace: vaultspec: interval: 1h url: https://helm.releases.hashicorp.com---# k8s/vault.yamlapiVersion: helm.toolkit.fluxcd.io/v2beta1kind: HelmReleasemetadata: name: vault namespace: vaultspec: interval: 30m chart: spec: chart: vault version: '0.27.x' sourceRef: kind: HelmRepository name: hashicorp namespace: vault values: server: ha: enabled: true replicas: 3 dataStorage: enabled: true size: 10Gi auditStorage: enabled: true size: 10Gi ui: enabled: true serviceType: ClusterIP injector: enabled: trueVault Secrets Operator
apiVersion: helm.toolkit.fluxcd.io/v2beta1kind: HelmReleasemetadata: name: vault-secrets-operator namespace: vaultspec: interval: 30m chart: spec: chart: vault-secrets-operator version: '0.4.x' sourceRef: kind: HelmRepository name: hashicorp namespace: vault values: defaultVaultConnection: enabled: true address: 'http://vault.vault.svc.cluster.local:8200'Using Vault Secrets in Applications
apiVersion: secrets.hashicorp.com/v1beta1kind: VaultStaticSecretmetadata: name: app-secretsspec: type: kv-v2 mount: secret path: myapp destination: name: app-secrets create: true refreshAfter: 30s vaultAuthRef: vault-auth---apiVersion: secrets.hashicorp.com/v1beta1kind: VaultAuthmetadata: name: vault-authspec: method: kubernetes mount: kubernetes kubernetes: role: myapp serviceAccount: defaultDevelopment Workflow: Making It Developer-Friendly
The best secrets management system is one developers actually use. Here's how to make it seamless:
Local Development
# .env.local (gitignored)DATABASE_URL=postgres://localhost:5432/myapp_devAPI_KEY=dev-api-key-not-secretStaging Environment
# Create staging secretskubectl create secret generic app-secrets \ --from-literal=DATABASE_URL="postgres://staging-db/myapp" \ --from-literal=API_KEY="staging-api-key" \ --namespace=staging \ --dry-run=client -o yaml | \ kubeseal --format yaml > k8s/overlays/staging/secrets.yamlProduction Environment
# Production secrets (extra careful here)kubectl create secret generic app-secrets \ --from-env-file=.env.production \ --namespace=production \ --dry-run=client -o yaml | \ kubeseal --format yaml > k8s/overlays/production/secrets.yamlSecurity Best Practices
1. Principle of Least Privilege
apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata: name: app-secrets-readerrules: - apiGroups: [''] resources: ['secrets'] resourceNames: ['app-secrets'] verbs: ['get']---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: name: app-secrets-bindingsubjects: - kind: ServiceAccount name: my-approleRef: kind: Role name: app-secrets-reader apiGroup: rbac.authorization.k8s.io2. Secret Rotation
Automate secret rotation with CronJobs:
apiVersion: batch/v1kind: CronJobmetadata: name: rotate-database-passwordspec: schedule: '0 2 * * 0' # Weekly at 2 AM Sunday jobTemplate: spec: template: spec: containers: - name: rotate-secrets image: my-secret-rotator:latest env: - name: SECRET_NAME value: 'app-secrets' - name: SECRET_KEY value: 'database-password' restartPolicy: OnFailure3. Audit and Monitoring
Monitor secret access:
apiVersion: monitoring.coreos.com/v1kind: PrometheusRulemetadata: name: secrets-monitoringspec: groups: - name: secrets.rules rules: - alert: SecretAccessFailure expr: increase(apiserver_audit_total{verb="get",objectRef_resource="secrets"}[5m]) > 10 for: 2m annotations: summary: 'High number of secret access failures'Choosing the Right Solution
Decision Matrix
Use this matrix to determine which secrets management approach fits your needs:
| Criteria | Manual Secrets | Sealed Secrets | HashiCorp Vault |
|---|---|---|---|
| Team Size | 1-3 developers | 3-20 developers | 20+ developers |
| Security Requirements | Basic | Medium | High/Enterprise |
| Compliance Needs | None | Basic | Advanced (SOC2, GDPR, etc.) |
| GitOps Workflow | ❌ Not compatible | ✅ Perfect fit | ✅ Compatible |
| Setup Complexity | ⭐ Very simple | ⭐⭐ Simple | ⭐⭐⭐⭐ Complex |
| Operational Overhead | ⭐ Minimal | ⭐⭐ Low | ⭐⭐⭐⭐ High |
| Secret Rotation | Manual process | Semi-automated | Fully automated |
| Audit Trail | None | Basic Git history | Comprehensive |
| Dynamic Secrets | ❌ Not supported | ❌ Not supported | ✅ Full support |
| Multi-Environment | Difficult | Good | Excellent |
| Cost | Free | Free | Free (self-hosted) |
When to Choose Each Solution
Choose Manual Secrets when:
- 🏠 Small team (1-3 developers)
- 🚀 Rapid prototyping or proof-of-concept
- 💰 Minimal budget for infrastructure
- 🔒 Low security requirements
- ⚡ Need to move fast without setup overhead
Example scenario: Weekend side project, startup MVP, internal tools
Choose Sealed Secrets when:
- 👥 Medium team (3-20 developers)
- 🔄 GitOps workflow is important
- 🛡️ Moderate security requirements
- 📝 Want secrets in Git (encrypted)
- 🎯 Balance of simplicity and security
Example scenario: Growing startup, SaaS product, team collaboration needed
Choose HashiCorp Vault when:
- 🏢 Large organization (20+ developers)
- 🔐 High security requirements
- 📋 Compliance mandates (SOC2, HIPAA, GDPR)
- 🔄 Dynamic secrets needed
- 📊 Detailed auditing required
- 🌍 Multi-environment complexity
Example scenario: Enterprise software, financial services, healthcare, regulated industries
Quick Decision Tree
Do you have compliance requirements?├─ Yes → Use Vault└─ No ├─ Do you use GitOps? │ ├─ Yes → Use Sealed Secrets │ └─ No → Manual or Sealed Secrets └─ Team size > 20? ├─ Yes → Consider Vault └─ No → Sealed Secrets or ManualMigration Path
Most teams follow this evolution:
- Start with Manual - Get to production quickly
- Move to Sealed Secrets - When team grows or GitOps is adopted
- Upgrade to Vault - When compliance or enterprise features are needed
Each step builds on the previous, so you're never starting from scratch.
Manual Secrets Management (Option 3)
For small teams or simple deployments, manual secrets management can be the right choice:
When Manual Makes Sense
- Small teams (1-3 developers) with direct cluster access
- Simple applications with few secrets
- Rapid prototyping where setup time matters more than automation
- Learning environments where you want to understand the basics first
Manual Secrets Workflow
# Create secrets directly in clusterkubectl create secret generic app-secrets \ --from-literal=DATABASE_URL="postgres://..." \ --from-literal=API_KEY="secret-key" \ --namespace=production
# Or from environment file (never commit the .env file!)kubectl create secret generic app-secrets \ --from-env-file=.env.production \ --namespace=productionManual Secrets Best Practices
- Never commit secrets to Git
- Use separate namespaces for different environments
- Document secret names and purposes
- Rotate secrets regularly with calendar reminders
- Use RBAC to limit who can access secrets
- Keep backups of critical secrets in a password manager
Manual Secrets Limitations
- ❌ No version control for secrets
- ❌ Difficult to share across team
- ❌ Manual rotation process
- ❌ No audit trail
- ❌ Doesn't work with GitOps
Integration with GitOps
Different solutions have different GitOps compatibility:
Sealed Secrets + GitOps ✅
# k8s/kustomization.yaml - Safe to commit!apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomization
resources: - deployment.yaml - service.yaml - secrets/app-secrets-sealed.yaml # Encrypted, safe in Git
namespace: productionVault + GitOps ✅
# k8s/kustomization.yaml - Safe to commit!apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomization
resources: - deployment.yaml - service.yaml - vault-secret.yaml # References Vault, no actual secrets
namespace: productionManual Secrets + GitOps ❌
Manual secrets don't work with GitOps since secrets can't be stored in Git repositories.
What's Next?
This secrets management foundation integrates perfectly with the GitOps infrastructure from our DevOps Holy Grail series:
With proper secrets management, you can build systems that are both secure and developer-friendly—no more choosing between the two.
🚀 Complete Secrets Management Setup
First, ask me about my specific setup and requirements before generating any code:- What type of Kubernetes cluster are you using?- What CI/CD system do you currently use?- What types of secrets do you need to manage (database credentials, API keys, certificates, etc.)?- Do you have compliance requirements that might need HashiCorp Vault?- How many environments do you need (dev/staging/prod)?- Do you have existing secrets management to migrate from?
Then help me implement a complete secrets management solution for my Kubernetes cluster:
1. Sealed Secrets controller installation via Flux2. Development workflow for creating and encrypting secrets3. Multi-environment secret management for your environments4. Secret rotation procedures and automation5. RBAC for least-privilege access6. Monitoring and alerting for secret access7. Backup and recovery procedures for encryption keys8. Integration with existing GitOps workflow9. HashiCorp Vault setup if needed for enterprise requirements
Important: All Kubernetes manifests should go in the /k8s directory and use FluxCD HelmRelease format for GitOps deployment.
Provide everything needed for production-ready secrets management.