Multus CNI: Multi-Network Interface Support for Kubernetes

Introduction

On March 15, 2019, the Kubernetes community released Multus CNI, a meta-plugin that enables Kubernetes pods to attach to multiple network interfaces. While standard CNI plugins provide a single network interface per pod, Multus allows pods to connect to multiple networks simultaneously, enabling advanced use cases like SR-IOV, secondary networks, and specialized workload requirements.

Traditional Kubernetes networking assumes one network interface per pod. But real-world scenarios often require multiple networks: a primary network for service communication, a secondary network for storage, or a dedicated network for high-performance workloads. Multus solves this by acting as a CNI meta-plugin that orchestrates multiple CNI plugins.

Why Multus?

• Multiple Networks: Attach pods to multiple network interfaces for different purposes.
• SR-IOV Support: Enable high-performance networking with SR-IOV virtual functions.
• Secondary Networks: Connect pods to specialized networks (storage, management, etc.).
• Workload Flexibility: Support diverse workload networking requirements.

Core Architecture

• Multus DaemonSet: Runs on each node, managing multiple CNI plugin invocations.
• NetworkAttachmentDefinition CRD: Defines secondary networks that pods can attach to.
• CNI Plugin Orchestration: Multus calls multiple CNI plugins to configure pod networking.
• Primary Network: Standard CNI plugin (Calico, Cilium, etc.) provides primary interface.
• Secondary Networks: Additional CNI plugins provide secondary interfaces.

Getting Started

Install Multus:

kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml

Create a NetworkAttachmentDefinition:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: secondary-network
  namespace: default
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "br1",
      "ipam": {
        "type": "host-local",
        "subnet": "10.56.0.0/16"
      }
    }

Attach a pod to multiple networks:

apiVersion: v1
kind: Pod
metadata:
  name: multi-net-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: secondary-network
spec:
  containers:
  - name: app
    image: nginx

Key Features

1. Multiple Interfaces: Pods can have multiple network interfaces with different CNI plugins.
2. NetworkAttachmentDefinition: Kubernetes-native way to define secondary networks.
3. CNI Plugin Agnostic: Works with any CNI plugin for primary or secondary networks.
4. SR-IOV Support: Integrates with SR-IOV CNI for high-performance networking.
5. Namespace Isolation: NetworkAttachmentDefinitions are namespace-scoped.

Use Cases

SR-IOV High-Performance Networking

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-network
spec:
  config: |
    {
      "type": "sriov",
      "cniVersion": "0.3.1",
      "vlan": 100,
      "ipam": {
        "type": "host-local",
        "subnet": "10.10.0.0/16"
      }
    }

Storage Networks

Connect pods to dedicated storage networks for high-performance data access.

Management Networks

Separate management traffic from application traffic using secondary networks.

Multi-Tenant Isolation

Use secondary networks for tenant-specific networking requirements.

Comparison with Single-Network CNI

Operational Considerations

• Complexity: Managing multiple networks adds operational complexity.
• Resource Usage: Multiple interfaces consume more resources.
• Troubleshooting: Debugging multi-network setups requires understanding all CNI plugins.
• Network Policies: Kubernetes NetworkPolicy applies to primary interface; secondary networks need separate policies.

Common Patterns

• Primary + Secondary: Use primary network for service communication, secondary for specialized needs.
• SR-IOV Workloads: Attach high-performance workloads to SR-IOV networks.
• Storage Networks: Connect pods to dedicated storage networks.
• Multi-Tenant: Use secondary networks for tenant isolation.

Limitations

• Complexity: Multi-network setups are more complex than single-network.
• Policy Support: NetworkPolicy applies to primary interface only.
• CNI Plugin Compatibility: Not all CNI plugins work well as secondary networks.
• Documentation: Multi-network scenarios require careful planning and testing.

Looking Ahead

Multus established the foundation for:

• CNCF Contribution: Multus would later be contributed to CNCF.
• Enhanced Features: Better integration with Kubernetes networking features.
• Performance Improvements: Optimizations for high-performance networking scenarios.
• Ecosystem Growth: More CNI plugins supporting Multus multi-network patterns.

Summary

Multus CNI demonstrated that Kubernetes networking could support complex, multi-network scenarios. By enabling pods to attach to multiple network interfaces, it opened the door for high-performance networking, specialized workloads, and advanced use cases that couldn’t be addressed with single-network CNI plugins.