Kubernetes Multi-Environment Deployments: From Single to Multiple Environments

Co-founder & CPTO

On this page
- Why Multiple Environments Matter
- The Problem with Single Environment Deployments
- The Kustomize Pattern: Base + Overlays
- Directory Structure: Organizing for Scale
- Step 1: Create Base Configuration
- Step 2: Create Staging Overlay
- Step 3: Create Production Overlay
- Step 4: Environment-Specific Secrets
- Step 5: FluxCD Configuration for Multiple Environments
- Step 6: Deployment Workflow
- Migration Strategy: Zero-Downtime Transition
- Best Practices for Multi-Environment Success
- Troubleshooting Common Issues
- What's Next?
A comprehensive guide to migrating from single environment to proper multi-environment deployments with Kubernetes and GitOps
Most teams start with a single environment and later realize they need separate staging and production environments. This transition is critical for maintaining stability while enabling rapid development, but it's also where many teams get stuck.
This guide shows you how to migrate from a single environment to a proper multi-environment setup using Kustomize and GitOps—without breaking your existing deployment or requiring a complete rewrite.
Why Multiple Environments Matter
Risk Reduction
Test changes in staging before they hit production. Catch integration issues, performance problems, and configuration errors early—when they're cheap to fix.
Parallel Development
Multiple teams can work on different features without blocking each other. Staging becomes your integration testing ground where features come together safely.
Compliance Requirements
Many industries require separate environments for audit trails and change management. Having proper environment separation isn't just good practice—it's often legally required.
Customer Confidence
Demonstrate new features to stakeholders in staging before production release. Show, don't tell, and build confidence in your releases.
Incident Response
When production breaks, having a working staging environment means you can test fixes before applying them to production. This dramatically reduces recovery time.
The Problem with Single Environment Deployments
Everything is Production
When you only have one environment, every change goes directly to production. There's no safety net, no place to test integrations, and no way to validate changes before users see them.
No Testing Ground
New features, dependency updates, and configuration changes all get tested in production. This leads to more outages and less confidence in deployments.
Difficult Collaboration
Multiple developers can't work on conflicting features simultaneously. Someone always has to wait, slowing down development velocity.
Poor Change Management
Without proper environments, it's hard to implement proper change management processes. Everything becomes a "hotfix" because there's nowhere else to test.
The Kustomize Pattern: Base + Overlays
Kustomize lets you define common configuration once and customize it per environment. This follows the DRY principle while maintaining environment-specific flexibility.
The key insight: Base configuration represents your "default" environment (typically staging-like), and overlays modify this base for specific environments.
🤖 Multi-Environment Migration Setup
First, ask me about my current setup and requirements before generating any code:- Do you currently have a single environment or multiple environments?- What environments do you need (staging, production, development, testing)?- What differences exist between your environments (resource limits, replicas, domains, etc.)?- Do you use different databases or external services per environment?- What is your current directory structure for Kubernetes manifests?- Do you need different secrets or configuration per environment?
Then help me migrate from single environment to multi-environment setup using Kustomize:
1. Restructure existing manifests into base configuration2. Create environment-specific overlays for each environment3. Set up proper namespace separation4. Configure environment-specific ingress and domains5. Handle secrets and configuration differences6. Update FluxCD configuration for multiple environments7. Create deployment workflows for each environment
Important: All Kubernetes manifests should go in the /k8s directory and use FluxCD HelmRelease format for GitOps deployment.
Provide a complete migration plan from single to multi-environment setup.Directory Structure: Organizing for Scale
Here's how to structure your repository for multiple environments:
k8s/├── base/ # Common resources (default: staging-like)│ ├── deployment.yaml # Base deployment configuration│ ├── service.yaml # Service definition│ ├── configmap.yaml # Common configuration│ └── kustomization.yaml # Base kustomization├── overlays/│ ├── staging/ # Staging environment (default)│ │ ├── kustomization.yaml│ │ ├── ingress.yaml # staging.example.com│ │ └── secrets.yaml # Staging secrets│ └── production/ # Production environment│ ├── kustomization.yaml│ ├── ingress.yaml # app.example.com│ ├── hpa.yaml # Production auto-scaling│ ├── secrets.yaml # Production secrets│ └── resources.yaml # Higher resource limits└── flux/ # FluxCD configurations ├── staging.yaml # Staging Kustomization └── production.yaml # Production KustomizationKey Principle: Base configuration represents your "default" environment (typically staging-like). Overlays modify this base for specific environments.
Step 1: Create Base Configuration
Start by moving your existing manifests to the base/ directory. This becomes your foundation that all environments build upon.
Base Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomization
resources: - deployment.yaml - service.yaml - configmap.yaml
commonLabels: app: my-service
# Default to staging-like configurationnamespace: staging
# Common configuration for all environmentsimages: - name: my-app newTag: latest
# Conservative defaultsreplicas: - name: my-service count: 2Base Deployment
apiVersion: apps/v1kind: Deploymentmetadata: name: my-servicespec: replicas: 2 # Conservative default selector: matchLabels: app: my-service template: metadata: labels: app: my-service spec: containers: - name: app image: my-app:latest ports: - containerPort: 3000 resources: requests: memory: '128Mi' cpu: '100m' limits: memory: '256Mi' # Staging-appropriate limits cpu: '200m' env: - name: ENVIRONMENT value: 'staging' # Default environment - name: LOG_LEVEL value: 'debug' # More verbose by default envFrom: - configMapRef: name: app-config - secretRef: name: app-secrets livenessProbe: httpGet: path: /health port: 3000 initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: httpGet: path: /ready port: 3000 initialDelaySeconds: 5 periodSeconds: 5Base Service
apiVersion: v1kind: Servicemetadata: name: my-servicespec: selector: app: my-service ports: - name: http port: 80 targetPort: 3000 type: ClusterIPBase ConfigMap
apiVersion: v1kind: ConfigMapmetadata: name: app-configdata: # Common configuration for all environments APP_NAME: 'my-service' PORT: '3000' # These can be overridden in overlays DATABASE_POOL_SIZE: '10' CACHE_TTL: '300'Step 2: Create Staging Overlay
Staging should be as close to production as possible, but with some differences for testing and debugging.
Staging Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomization
namespace: staging
resources: - ../../base - ingress.yaml - secrets.yaml
# Staging uses base configuration mostly as-ispatchesStrategicMerge: - deployment-patch.yaml
# Use staging-specific image tagsimages: - name: my-app newTag: staging-latest
# Staging-specific labelscommonLabels: environment: staging
# Override some base configurationconfigMapGenerator: - name: app-config behavior: merge literals: - LOG_LEVEL=debug - DEBUG_MODE=trueStaging Deployment Patch
apiVersion: apps/v1kind: Deploymentmetadata: name: my-servicespec: template: spec: containers: - name: app env: - name: ENVIRONMENT value: 'staging' - name: LOG_LEVEL value: 'debug' # More verbose logging in staging - name: DEBUG_MODE value: 'true' # Enable debug featuresStaging Ingress
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: my-service annotations: cert-manager.io/cluster-issuer: letsencrypt-prod external-dns.alpha.kubernetes.io/hostname: staging.example.com # Staging-specific annotations nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-authspec: tls: - hosts: - staging.example.com secretName: staging-tls rules: - host: staging.example.com http: paths: - path: / pathType: Prefix backend: service: name: my-service port: number: 80Step 3: Create Production Overlay
Production needs higher resource limits, more replicas, stricter security, and production-specific configuration.
Production Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomization
namespace: production
resources: - ../../base - ingress.yaml - hpa.yaml - secrets.yaml - monitoring.yaml - network-policy.yaml
patchesStrategicMerge: - deployment-patch.yaml
# Production uses specific image tags, not latestimages: - name: my-app newTag: v1.2.3
# Scale up for productionreplicas: - name: my-service count: 5
# Production-specific labelscommonLabels: environment: production
# Production configurationconfigMapGenerator: - name: app-config behavior: merge literals: - LOG_LEVEL=info - DEBUG_MODE=false - DATABASE_POOL_SIZE=20Production Deployment Patch
apiVersion: apps/v1kind: Deploymentmetadata: name: my-servicespec: template: spec: containers: - name: app resources: requests: memory: '256Mi' cpu: '200m' limits: memory: '512Mi' # Higher limits for production cpu: '500m' env: - name: ENVIRONMENT value: 'production' - name: LOG_LEVEL value: 'info' # Less verbose in production - name: DEBUG_MODE value: 'false' # No debug features in production # Production-specific security context securityContext: runAsNonRoot: true runAsUser: 1000 fsGroup: 2000Production Ingress
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: my-service annotations: cert-manager.io/cluster-issuer: letsencrypt-prod external-dns.alpha.kubernetes.io/hostname: app.example.com # Production-specific annotations nginx.ingress.kubernetes.io/rate-limit: '100' nginx.ingress.kubernetes.io/rate-limit-window: '1m' nginx.ingress.kubernetes.io/ssl-redirect: 'true'spec: tls: - hosts: - app.example.com secretName: production-tls rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: my-service port: number: 80Production Auto-Scaling
apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: my-service-hpaspec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: my-service minReplicas: 5 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 80 behavior: scaleUp: stabilizationWindowSeconds: 60 policies: - type: Percent value: 50 periodSeconds: 60 scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 10 periodSeconds: 60Step 4: Environment-Specific Secrets
Each environment needs its own secrets with appropriate values.
Staging Secrets
# Create staging secretskubectl create secret generic app-secrets \ --from-literal=DATABASE_URL="postgres://staging-user:staging-pass@staging-db.example.com/myapp" \ --from-literal=API_KEY="staging-api-key-12345" \ --from-literal=REDIS_URL="redis://staging-redis.example.com:6379" \ --namespace=staging \ --dry-run=client -o yaml | \ kubeseal --format yaml > k8s/overlays/staging/secrets.yamlProduction Secrets
# Create production secrets (be extra careful here!)kubectl create secret generic app-secrets \ --from-literal=DATABASE_URL="postgres://prod-user:secure-pass@prod-db.example.com/myapp" \ --from-literal=API_KEY="prod-api-key-67890" \ --from-literal=REDIS_URL="redis://prod-redis.example.com:6379" \ --namespace=production \ --dry-run=client -o yaml | \ kubeseal --format yaml > k8s/overlays/production/secrets.yamlStep 5: FluxCD Configuration for Multiple Environments
Configure FluxCD to manage both environments with different policies.
Staging FluxCD Configuration
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2kind: Kustomizationmetadata: name: my-service-staging namespace: flux-systemspec: interval: 5m # Frequent updates for staging path: './k8s/overlays/staging' prune: true sourceRef: kind: GitRepository name: my-service targetNamespace: staging # Staging can auto-deploy suspend: false # Health checks healthChecks: - apiVersion: apps/v1 kind: Deployment name: my-service namespace: staging # Retry configuration retryInterval: 2m timeout: 5mProduction FluxCD Configuration
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2kind: Kustomizationmetadata: name: my-service-production namespace: flux-systemspec: interval: 10m # Less frequent updates in production path: './k8s/overlays/production' prune: true sourceRef: kind: GitRepository name: my-service targetNamespace: production # Production deployments can be gated suspend: false # Depends on staging being healthy dependsOn: - name: my-service-staging # Health checks healthChecks: - apiVersion: apps/v1 kind: Deployment name: my-service namespace: production # More conservative retry policy retryInterval: 5m timeout: 10mStep 6: Deployment Workflow
Create a deployment workflow that promotes changes from staging to production.
GitHub Actions Workflow
name: Deployon: push: branches: [main] pull_request: branches: [main]
jobs: deploy-staging: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3
- name: Update staging image run: | cd k8s/overlays/staging kustomize edit set image my-app:staging-${{ github.sha }}
- name: Commit staging deployment run: | git config --local user.email "action@github.com" git config --local user.name "GitHub Action" git add k8s/overlays/staging/kustomization.yaml git commit -m "deploy: update staging to ${{ github.sha }}" || exit 0 git push
deploy-production: runs-on: ubuntu-latest needs: deploy-staging if: github.ref == 'refs/heads/main' environment: production # Requires approval in GitHub steps: - uses: actions/checkout@v3
- name: Update production image run: | cd k8s/overlays/production kustomize edit set image my-app:v${{ github.run_number }}
- name: Commit production deployment run: | git config --local user.email "action@github.com" git config --local user.name "GitHub Action" git add k8s/overlays/production/kustomization.yaml git commit -m "deploy: update production to v${{ github.run_number }}" || exit 0 git pushMigration Strategy: Zero-Downtime Transition
Here's how to migrate from single environment to multi-environment without downtime:
Phase 1: Prepare Base Configuration
- Create base directory: Move existing manifests to
k8s/base/ - Create base kustomization: Define common resources
- Test locally: Verify
kustomize build k8s/baseworks
Phase 2: Create Staging Environment
- Create staging overlay: Start with minimal changes from base
- Deploy to staging namespace: Test the new structure
- Verify staging works: Ensure all functionality works in staging
Phase 3: Migrate Production
- Create production overlay: Add production-specific configuration
- Update FluxCD: Point to new overlay structure
- Monitor deployment: Watch for any issues during migration
Phase 4: Clean Up
- Remove old manifests: Delete the old single-environment files
- Update documentation: Document the new structure
- Train team: Ensure everyone understands the new workflow
Best Practices for Multi-Environment Success
Environment Parity
Keep environments as similar as possible:
# Use the same base imageimages: - name: my-app newTag: v1.2.3 # Same version across environments
# Scale resources proportionally, not differently# Production: 4 CPUs, 8GB RAM# Staging: 1 CPU, 2GB RAM (1/4 of production)Configuration Management
Use environment variables for differences:
apiVersion: v1kind: ConfigMapmetadata: name: app-configdata: # Same across environments APP_NAME: 'my-service' # Different per environment (overridden in overlays) LOG_LEVEL: 'info' DEBUG_MODE: 'false'Secret Management
Keep secrets separate and secure:
# Never commit plain text secrets# Always use sealed-secrets or external secret managementkubeseal --format yaml < staging-secrets.yaml > k8s/overlays/staging/secrets.yamlkubeseal --format yaml < production-secrets.yaml > k8s/overlays/production/secrets.yamlMonitoring and Alerting
Monitor both environments differently:
# Staging: Warn on errors- alert: StagingHighErrorRate expr: rate(http_requests_total{status=~"5..",env="staging"}[5m]) > 0.1 for: 5m
# Production: Page on errors- alert: ProductionHighErrorRate expr: rate(http_requests_total{status=~"5..",env="production"}[5m]) > 0.01 for: 1mTroubleshooting Common Issues
Kustomize Build Failures
# Test your kustomization locallykustomize build k8s/overlays/stagingkustomize build k8s/overlays/production
# Common issues:# - Missing resources in base# - Incorrect patch syntax# - Namespace mismatchesFluxCD Sync Issues
# Check FluxCD statusflux get kustomizations
# Check specific environmentkubectl describe kustomization my-service-staging -n flux-system
# Force reconciliationflux reconcile kustomization my-service-stagingEnvironment Drift
# Compare environmentskubectl get deployment my-service -n staging -o yaml > staging-deployment.yamlkubectl get deployment my-service -n production -o yaml > production-deployment.yamldiff staging-deployment.yaml production-deployment.yamlWhat's Next?
With proper multi-environment deployments, you now have:
- Risk-free deployments through staging validation
- Parallel development without conflicts
- Proper change management with environment promotion
- Compliance-ready audit trails
- Scalable processes that work for any number of environments
This foundation integrates perfectly with the GitOps infrastructure from our DevOps Holy Grail series:
- Part 1 - Basic GitOps setup with CI/CD
- Part 2 - HTTPS and DNS automation
- Part 3 - Enterprise features and monitoring
With multi-environment deployments, you can ship faster while breaking less—the true goal of modern DevOps.
Ready to implement multi-environment deployments? Start with staging, validate your approach, then promote to production with confidence.