Flux Multi-Tenancy

Multi-tenancy in Flux allows you to isolate applications and resources for different teams, projects, or customers within a single Kubernetes cluster. Flux provides several mechanisms to implement tenant isolation and access control.

Multi-Tenancy Concepts

Multi-tenancy enables:

Isolation - Separate resources per tenant
Access Control - Limit what each tenant can do
Resource Quotas - Control resource usage per tenant
Policy Enforcement - Apply different policies per tenant

graph TB A[Cluster] --> B[Tenant A] A --> C[Tenant B] A --> D[Tenant C] B --> E[Namespace A1] B --> F[Namespace A2] C --> G[Namespace B1] D --> H[Namespace C1] D --> I[Namespace C2] J[Flux Controllers] --> B J --> C J --> D style A fill:#e1f5ff style B fill:#e8f5e9 style C fill:#fff4e1 style D fill:#f3e5f5

Tenant Isolation Strategies

1. Namespace-Based Isolation

The simplest approach: each tenant gets one or more namespaces.

# Tenant A resources
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a-apps
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./tenants/tenant-a
  prune: true
  sourceRef:
    kind: GitRepository
    name: manifests
  targetNamespace: tenant-a

Benefits:

Simple to implement
Native Kubernetes isolation
Easy to manage

Limitations:

Namespace-level isolation only
Requires RBAC configuration
No automatic tenant management

2. Repository-Based Isolation

Each tenant has a separate Git repository.

# Tenant A GitRepository
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: tenant-a-repo
  namespace: flux-system
spec:
  interval: 1m0s
  url: https://github.com/org/tenant-a-manifests
  secretRef:
    name: tenant-a-credentials

---
# Tenant A Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./
  sourceRef:
    kind: GitRepository
    name: tenant-a-repo
  targetNamespace: tenant-a

Benefits:

Complete isolation
Independent version control
Separate access control

Limitations:

More repositories to manage
Requires separate credentials

3. Directory-Based Isolation

Multiple tenants in a single repository with directory separation.

manifests/
├── tenants/
│   ├── tenant-a/
│   │   ├── apps/
│   │   └── infrastructure/
│   ├── tenant-b/
│   │   ├── apps/
│   │   └── infrastructure/
│   └── tenant-c/
│       ├── apps/
│       └── infrastructure/

# Tenant A Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./tenants/tenant-a
  sourceRef:
    kind: GitRepository
    name: shared-repo
  targetNamespace: tenant-a

Benefits:

Single repository
Easy to manage
Shared infrastructure code

Limitations:

Less isolation
Requires careful access control

Flux Tenant Model

Flux doesn’t have a built-in Tenant CRD, but you can model tenants using:

Tenant Structure

# Tenant definition (custom resource or ConfigMap)
apiVersion: v1
kind: ConfigMap
metadata:
  name: tenant-a
  namespace: flux-system
  labels:
    tenant: tenant-a
data:
  name: "Tenant A"
  namespace: "tenant-a"
  repository: "https://github.com/org/tenant-a"

Tenant Kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a
  namespace: flux-system
  labels:
    tenant: tenant-a
spec:
  interval: 5m0s
  path: ./
  prune: true
  sourceRef:
    kind: GitRepository
    name: tenant-a-repo
  targetNamespace: tenant-a
  dependsOn:
  - name: tenant-a-infra

RBAC Integration

Combine Flux with Kubernetes RBAC for access control.

Service Account per Tenant

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tenant-a-controller
  namespace: flux-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tenant-a-role
rules:
- apiGroups: [""]
  resources: ["*"]
  resourceNames: ["tenant-a"]
  verbs: ["*"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tenant-a-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tenant-a-role
subjects:
- kind: ServiceAccount
  name: tenant-a-controller
  namespace: flux-system

Kustomization with Service Account

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a
  namespace: flux-system
spec:
  serviceAccountName: tenant-a-controller
  interval: 5m0s
  path: ./
  sourceRef:
    kind: GitRepository
    name: tenant-a-repo
  targetNamespace: tenant-a

Resource Quotas

Limit resources per tenant using ResourceQuota.

Namespace ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"

LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-a-limits
  namespace: tenant-a
spec:
  limits:
  - default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    type: Container

Network Policies

Isolate tenant networks using NetworkPolicy.

Tenant Network Isolation

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-a-isolation
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: tenant-a
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: tenant-a

Tenant Configuration

Complete Tenant Setup

# 1. Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    tenant: tenant-a
---
# 2. ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
---
# 3. LimitRange
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-a-limits
  namespace: tenant-a
spec:
  limits:
  - default:
      cpu: "500m"
      memory: "512Mi"
    type: Container
---
# 4. NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-a-isolation
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
# 5. GitRepository
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: tenant-a-repo
  namespace: flux-system
spec:
  interval: 1m0s
  url: https://github.com/org/tenant-a
  secretRef:
    name: tenant-a-credentials
---
# 6. Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./
  prune: true
  sourceRef:
    kind: GitRepository
    name: tenant-a-repo
  targetNamespace: tenant-a

Multi-Tenant Repository Structure

Recommended Structure

manifests/
├── tenants/
│   ├── tenant-a/
│   │   ├── base/
│   │   │   ├── namespace.yaml
│   │   │   ├── resourcequota.yaml
│   │   │   └── networkpolicy.yaml
│   │   └── apps/
│   │       ├── app1/
│   │       └── app2/
│   ├── tenant-b/
│   │   └── ...
│   └── tenant-c/
│       └── ...
└── shared/
    ├── monitoring/
    └── infrastructure/

Best Practices

1. Use Namespace Labels

Label namespaces for easy identification:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    tenant: tenant-a
    environment: production

2. Separate Service Accounts

Use different service accounts per tenant for better isolation.

3. Enforce Resource Quotas

Always set ResourceQuota and LimitRange for tenants.

4. Use Network Policies

Implement network policies for network isolation.

5. Separate Git Repositories

For high isolation, use separate repositories per tenant.

6. Monitor Tenant Resources

Set up monitoring and alerts per tenant:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tenant-a-metrics
  namespace: tenant-a
spec:
  selector:
    matchLabels:
      tenant: tenant-a

Security Considerations

1. Credential Management

Use separate credentials per tenant
Store in external secret management
Rotate regularly

2. Access Control

Implement RBAC per tenant
Use least privilege principle
Audit access regularly

3. Resource Limits

Set appropriate quotas
Monitor usage
Alert on violations

4. Network Isolation

Use NetworkPolicy
Limit cross-tenant communication
Monitor network traffic

Troubleshooting

Tenant Resources Not Applying

# Check Kustomization status
kubectl get kustomization -n flux-system -l tenant=tenant-a

# Check GitRepository
kubectl get gitrepository -n flux-system

# Check service account permissions
kubectl auth can-i create deployments --as=system:serviceaccount:flux-system:tenant-a-controller -n tenant-a

Resource Quota Issues

# Check quota usage
kubectl describe resourcequota -n tenant-a

# Check limit range
kubectl describe limitrange -n tenant-a

# Check pod status
kubectl get pods -n tenant-a

Network Policy Issues

# Check network policies
kubectl get networkpolicy -n tenant-a

# Test connectivity
kubectl run test -n tenant-a --image=busybox --rm -it -- wget -O- http://service:port

Summary

Multi-tenancy with Flux involves:

Isolation - Namespace, repository, or directory-based
RBAC - Service accounts and role bindings per tenant
Resource Quotas - Limit resources per tenant
Network Policies - Isolate network traffic
Git Structure - Organize manifests per tenant

Choose the isolation strategy based on your requirements:

Namespace-based - Simple, good for most cases
Repository-based - Maximum isolation
Directory-based - Single repository, shared code

Combine with Kubernetes RBAC, ResourceQuota, and NetworkPolicy for complete tenant isolation.