Environment Promotion

Environment promotion is the process of moving applications through different environments (development → staging → production) in a controlled, automated manner. GitOps provides several patterns for implementing promotion workflows.

Promotion Concepts

Promotion ensures:

Consistency - Same process for all environments
Traceability - Complete audit trail
Safety - Controlled, reviewed changes
Automation - Reduce manual errors

graph LR A[Development] --> B[Staging] B --> C[Production] D[Code Changes] --> A E[Testing] --> B F[Approval] --> C style A fill:#e1f5ff style B fill:#fff4e1 style C fill:#ffe1e1

Promotion Strategies

Strategy 1: Branch-Based Promotion

Each environment has its own Git branch.

graph TB A[main branch] --> B[dev branch] B --> C[staging branch] C --> D[production branch] E[Feature Branch] --> A A --> F[Merge to dev] F --> G[Promote to staging] G --> H[Promote to production] style A fill:#e1f5ff style B fill:#e8f5e9 style C fill:#fff4e1 style D fill:#ffe1e1

Structure:

repo/
├── apps/
│   └── web-app/
│       ├── deployment.yaml
│       └── service.yaml

ArgoCD Applications:

# Dev Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-dev
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: dev
    path: apps/web-app

---
# Staging Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-staging
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: staging
    path: apps/web-app

---
# Production Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-prod
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: production
    path: apps/web-app

Promotion Process:

# 1. Merge to dev
git checkout dev
git merge feature-branch
git push

# 2. Promote to staging
git checkout staging
git merge dev
git push

# 3. Promote to production
git checkout production
git merge staging
git push

Benefits:

Clear environment separation
Easy to see what’s in each environment
Simple promotion (merge branches)

Limitations:

Requires branch management
Can have merge conflicts
Multiple branches to maintain

Strategy 2: Directory-Based Promotion

Environments organized in directories, all in main branch.

graph TB A[main branch] --> B[dev/ directory] A --> C[staging/ directory] A --> D[production/ directory] E[Update dev/] --> B F[Copy to staging/] --> C G[Copy to production/] --> D style A fill:#e1f5ff style B fill:#e8f5e9 style C fill:#fff4e1 style D fill:#ffe1e1

Structure:

repo/
├── apps/
│   └── web-app/
│       ├── base/
│       └── overlays/
│           ├── dev/
│           ├── staging/
│           └── production/

Kustomize Overlays:

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: web-app
  newTag: dev-latest

---
# overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: web-app
  newTag: staging-v1.2.3

---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: web-app
  newTag: v1.2.3

ArgoCD Applications:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-dev
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: main
    path: apps/web-app/overlays/dev

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-prod
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: main
    path: apps/web-app/overlays/production

Promotion Process:

# Update image tag in production overlay
cd apps/web-app/overlays/production
# Edit kustomization.yaml to update image tag
git commit -m "Promote web-app v1.2.3 to production"
git push

Benefits:

Single branch (main)
Easy to compare environments
All changes in one place

Limitations:

Requires careful directory management
Can accidentally promote wrong version

Strategy 3: Tag-Based Promotion

Use Git tags to mark versions for promotion.

graph TB A[Commit] --> B[Tag: v1.2.3-dev] B --> C[Tag: v1.2.3-staging] C --> D[Tag: v1.2.3-prod] style A fill:#e1f5ff style B fill:#e8f5e9 style C fill:#fff4e1 style D fill:#ffe1e1

ArgoCD Applications:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-dev
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: v1.2.3-dev
    path: apps/web-app

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-prod
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: v1.2.3-prod
    path: apps/web-app

Promotion Process:

# Tag for dev
git tag v1.2.3-dev
git push --tags

# Tag same commit for staging
git tag v1.2.3-staging
git push --tags

# Tag for production
git tag v1.2.3-prod
git push --tags

Automated Promotion

Using ArgoCD ApplicationSets

Automate promotion with ApplicationSets:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web-app
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - env: dev
        namespace: dev
        imageTag: dev-latest
      - env: staging
        namespace: staging
        imageTag: staging-v1.2.3
      - env: production
        namespace: production
        imageTag: v1.2.3
  template:
    metadata:
      name: 'web-app-{{env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/repo
        targetRevision: main
        path: apps/web-app/overlays/{{env}}
        kustomize:
          images:
          - web-app:myregistry/web-app:{{imageTag}}
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Using Flux ImageUpdateAutomation

Automate image updates across environments:

# Dev ImageUpdateAutomation
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: web-app-dev
  namespace: flux-system
spec:
  interval: 5m0s
  sourceRef:
    kind: GitRepository
    name: manifests
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: Flux
        email: [email protected]
      messageTemplate: 'Update dev image: {{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: main
  update:
    path: ./apps/web-app/overlays/dev
    strategy: Setters

---
# Production ImageUpdateAutomation (manual trigger)
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: web-app-prod
  namespace: flux-system
spec:
  interval: 24h0m0s  # Less frequent
  sourceRef:
    kind: GitRepository
    name: manifests
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: Flux
        email: [email protected]
      messageTemplate: 'Promote to prod: {{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: main
  update:
    path: ./apps/web-app/overlays/production
    strategy: Setters

Using CI/CD Integration

Promote via CI/CD pipelines:

# GitHub Actions example
name: Promote to Production
on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Version to promote'
        required: true

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Update production image
      run: |
        cd apps/web-app/overlays/production
        sed -i "s/newTag: .*/newTag: ${{ github.event.inputs.version }}/" kustomization.yaml
    - name: Create PR
      uses: peter-evans/create-pull-request@v5
      with:
        title: "Promote web-app ${{ github.event.inputs.version }} to production"
        branch: promote-prod

Promotion Gates and Approvals

Manual Approval Gates

Require approval before promotion:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-prod
  namespace: argocd
spec:
  syncPolicy:
    syncWindows:
    - kind: allow
      schedule: '0 2 * * 1-5'
      duration: 2h
      manualSync: true
    - kind: deny
      schedule: '* * * * *'
      manualSync: false

ArgoCD Sync Windows

Control when promotions can happen:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  syncWindows:
  - kind: allow
    schedule: '0 2 * * 1-5'  # Weekdays 2 AM
    duration: 2h
    applications:
    - 'production-*'
    manualSync: true
  - kind: deny
    schedule: '* * * * *'
    applications:
    - 'production-*'

CI/CD Approval Gates

Use CI/CD for approval workflows:

# GitLab CI example
promote_to_prod:
  stage: promote
  only:
    - main
  when: manual  # Requires manual trigger
  script:
    - ./scripts/promote-to-prod.sh
  environment:
    name: production

Rollback Strategies

Git Revert

Revert the promotion commit:

# Revert last promotion
git revert HEAD
git push

# GitOps controller automatically applies rollback

Tag Rollback

Point to previous tag:

# Update Application to previous tag
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-prod
spec:
  source:
    targetRevision: v1.2.2  # Previous version

Kustomize Rollback

Revert image tag in overlay:

# Revert to previous image tag
cd apps/web-app/overlays/production
# Edit kustomization.yaml
git commit -m "Rollback to v1.2.2"
git push

Best Practices

1. Use Semantic Versioning

Tag versions consistently:

# Format: v<major>.<minor>.<patch>
git tag v1.2.3
git tag v1.2.3-dev
git tag v1.2.3-staging
git tag v1.2.3-prod

2. Automate Testing

Run tests before promotion:

# CI/CD pipeline
stages:
  - test
  - promote-dev
  - test-staging
  - promote-staging
  - test-production
  - promote-production

3. Require Approvals for Production

Always require manual approval for production:

spec:
  syncPolicy:
    # No automated sync for production
    syncWindows:
    - kind: allow
      schedule: '0 2 * * 1-5'
      manualSync: true

4. Document Promotion Process

Create runbooks:

# Promotion Process

1. Merge to dev branch
2. Verify in dev environment
3. Create PR to staging
4. Review and merge
5. Verify in staging
6. Create PR to production
7. Get approval
8. Merge to production

5. Monitor Promotions

Set up alerts for:

Failed promotions
Promotion approvals
Rollbacks

Troubleshooting

Promotion Not Happening

# Check Application status
argocd app get web-app-prod

# Check sync status
argocd app sync web-app-prod

# Check sync windows
argocd app get web-app-prod --show-operation

Wrong Version Promoted

# Check current version
argocd app get web-app-prod

# Rollback
git revert <commit-hash>
git push

Summary

Environment promotion in GitOps:

Branch-based - Separate branches per environment
Directory-based - Overlays in same branch
Tag-based - Git tags mark versions
Automated - Use ApplicationSets or ImageUpdateAutomation
Gated - Require approvals for production
Traceable - All changes in Git

Choose a strategy based on your needs:

Simple teams - Directory-based
Complex workflows - Branch-based
Version control - Tag-based

Always require approvals for production and maintain clear promotion processes.