Cilium Service Mesh: Native Mesh Integration Without Sidecars

Introduction

In November 2021, Cilium introduced native service mesh capabilities, fundamentally changing how service meshes work in Kubernetes. Instead of injecting sidecar proxies into every pod, Cilium’s service mesh leverages eBPF programs in the Linux kernel to provide service-to-service communication, security, and observability—all without sidecars.

Traditional service meshes like Istio and Linkerd require sidecar proxies (Envoy or Linkerd-proxy) in every pod, adding latency, resource overhead, and operational complexity. Cilium’s approach eliminates sidecars entirely by moving mesh functionality into the kernel via eBPF, providing the same capabilities with better performance and simpler operations.

The Sidecar Problem

• Resource Overhead: Sidecars consume CPU and memory in every pod.
• Latency: Extra network hop through sidecar adds latency to every request.
• Operational Complexity: Managing sidecar injection, upgrades, and configuration.
• Debugging Difficulty: Troubleshooting requires understanding both application and proxy behavior.

Cilium’s Approach

• Kernel-Level Mesh: eBPF programs in the kernel handle service mesh functionality.
• No Sidecars: Pods communicate directly; mesh features are transparent.
• Better Performance: Kernel-level processing is faster than user-space proxies.
• Simpler Operations: No sidecar injection, upgrades, or proxy configuration needed.

Key Features

1. Transparent mTLS: Automatic mutual TLS encryption without sidecar configuration.
2. Layer 7 Policies: HTTP/gRPC-aware policies enforced at the kernel level.
3. Traffic Management: Load balancing, retries, and circuit breaking in eBPF.
4. Observability: Service dependency mapping and flow visibility via Hubble.
5. Multi-Cluster: Mesh capabilities span multiple Kubernetes clusters.

Architecture

• eBPF Programs: Kernel-level programs handle service mesh functionality.
• Cilium Agent: Manages eBPF programs and mesh configuration.
• Envoy Integration: Optional Envoy can be used for advanced L7 features when needed.
• Hubble: Provides observability into mesh traffic and service dependencies.

Getting Started

Enable Cilium service mesh:

helm install cilium cilium/cilium --version 1.11.0 \
  --namespace kube-system \
  --set mesh.enabled=true \
  --set mesh.loadBalancing.mode=dsr

Deploy a service with mesh features:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    io.cilium/service-mesh: "enabled"
spec:
  selector:
    app: my-app
  ports:
  - port: 80

Comparison: Sidecar vs. Kernel Mesh

Use Cases

• High-Performance Workloads: Applications that can’t tolerate sidecar latency.
• Resource-Constrained Environments: Edge computing or IoT deployments with limited resources.
• Simplified Operations: Teams wanting mesh features without sidecar complexity.
• Multi-Cluster Meshes: Spanning service mesh across multiple Kubernetes clusters.

Layer 7 Capabilities

While Cilium’s kernel mesh handles L3/L4 efficiently, advanced L7 features can use Envoy:

• HTTP Routing: Advanced path-based routing and header manipulation.
• gRPC Load Balancing: Protocol-aware load balancing for gRPC services.
• Rate Limiting: Sophisticated rate limiting policies.
• WAF Integration: Web application firewall capabilities.

Cilium can selectively use Envoy for specific services that need advanced L7 features while keeping most traffic in the kernel.

Operational Benefits

• No Sidecar Injection: Eliminates mutating webhook complexity and injection failures.
• Simpler Upgrades: Mesh upgrades don’t require pod restarts or sidecar updates.
• Better Debugging: Fewer moving parts make troubleshooting easier.
• Resource Efficiency: Shared kernel programs reduce overall resource consumption.

Limitations

• Kernel Requirements: Requires Linux kernel 4.19+ with eBPF support.
• L7 Features: Advanced L7 features may still require Envoy for some use cases.
• Ecosystem: Fewer third-party integrations compared to established sidecar meshes.
• Learning Curve: Teams need to understand eBPF concepts for advanced troubleshooting.

Migration from Sidecar Meshes

• Gradual Migration: Run Cilium mesh alongside existing sidecar meshes during transition.
• Feature Parity: Evaluate which mesh features are needed and ensure Cilium supports them.
• Performance Testing: Validate that kernel mesh performance meets requirements.
• Team Training: Ensure operations teams understand eBPF-based mesh architecture.

Common Patterns

• Zero-Trust Networking: Use Cilium’s transparent mTLS for automatic encryption.
• Service Discovery: Leverage Kubernetes Services with Cilium’s service mesh integration.
• Traffic Splitting: Use Cilium’s load balancing for canary deployments.
• Multi-Cluster: Extend mesh capabilities across multiple clusters with Cilium Cluster Mesh.

Looking Ahead

Cilium’s service mesh approach would influence:

• Mesh Architecture: Other projects exploring kernel-level mesh capabilities.
• Performance Standards: Setting new benchmarks for service mesh performance.
• Hybrid Models: Combining kernel mesh with selective sidecar usage for advanced features.
• CNCF Evolution: Contributing to service mesh standardization efforts.

Summary

Cilium’s service mesh demonstrated that service mesh functionality didn’t require sidecar proxies. By moving mesh capabilities into the kernel via eBPF, it provided the same features with better performance, lower resource usage, and simpler operations. This approach challenged the sidecar model and influenced how the industry thinks about service mesh architecture.