Falco: Runtime Security Monitoring for Kubernetes

Introduction

In mid-2018, Falco emerged as a powerful runtime security monitoring tool for Kubernetes, providing real-time threat detection through system call monitoring and policy enforcement. Originally developed by Sysdig, Falco’s acceptance into the CNCF sandbox marked its recognition as a critical security tool for cloud-native environments.

This mattered because runtime security had become essential for production Kubernetes deployments. While network policies and RBAC provided preventive controls, teams needed visibility into what was actually happening inside their clusters. Falco filled this gap by monitoring system calls, detecting anomalous behavior, and alerting on security violations in real-time.

Historical note: Falco was originally developed by Sysdig and accepted into the CNCF sandbox in 2018. It would become one of the most widely adopted runtime security tools for Kubernetes, eventually graduating to CNCF incubation status.

Falco Architecture

System Call Monitoring

• eBPF/Kernel Module: Falco uses eBPF (or kernel modules) to monitor system calls at the kernel level.
• Real-Time Detection: Analyzes system calls in real-time to detect security violations.
• Policy Engine: Uses rules to define what constitutes suspicious or malicious behavior.
• Alerting: Generates alerts when policy violations are detected.

Key Components

• Falco Engine: Core detection engine that monitors system calls and evaluates rules.
• Falco Rules: YAML-based rules that define security policies and detection patterns.
• Falco Sidekick: Optional component for forwarding alerts to external systems.
• Falco Sidekick-UI: Web UI for visualizing Falco alerts.

Falco vs Traditional Security Monitoring

Common Detection Scenarios

Suspicious Process Execution

Falco can detect when unexpected processes are executed:

# Falco rule example
- rule: Run shell in container
  desc: Notice shell activity within a container
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
  output: >
    Shell spawned in container (user=%user.name %container.info
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline
    terminal=%proc.tty container_id=%container.id image=%container.image.repository)
  priority: WARNING

File System Access

Detect unauthorized file system access:

- rule: Write below binary dir
  desc: Detect writes to binary directories
  condition: >
    bin_dir and evt.dir = < and open_write
    and not package_mgmt_procs
    and not exe_running_docker_save
    and not python_running_deny_list
  output: >
    File below a known binary directory opened for writing
    (user=%user.name command=%proc.cmdline file=%fd.name
    parent=%proc.pname pcmdline=%proc.pcmdline gparent=%proc.aname[2])
  priority: ERROR

Network Anomalies

Detect suspicious network activity:

- rule: Outbound connection to C2 servers
  desc: Detect outbound connections to known C2 servers
  condition: >
    outbound and evt.type = connect
    and fd.sip != "127.0.0.1"
    and c2_server
  output: >
    Outbound connection to C2 server (command=%proc.cmdline
    connection=%fd.name user=%user.name container_id=%container.id)
  priority: CRITICAL

Kubernetes Integration

DaemonSet Deployment

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app: falco
  template:
    metadata:
      labels:
        app: falco
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: falco
        image: docker.io/falcosecurity/falco:latest
        securityContext:
          privileged: true
        volumeMounts:
        - name: falco-config
          mountPath: /etc/falco
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: falco-config
        configMap:
          name: falco-config
      - name: sys
        hostPath:
          path: /sys

Custom Rules

# custom-rules.yaml
- rule: Unauthorized pod creation
  desc: Detect pod creation outside approved namespaces
  condition: >
    k8s_psp.pod.name exists
    and not k8s_psp.pod.namespace.name in (production, staging)
  output: >
    Unauthorized pod creation (pod=%k8s_psp.pod.name
    namespace=%k8s_psp.pod.namespace.name user=%user.name)
  priority: WARNING

Integration with Kubernetes Audit Logs

Falco can integrate with Kubernetes audit logs for additional context:

# Falco configuration
json_output: true
json_include_output_property: true
json_include_tags_property: true

# Kubernetes audit log integration
k8s_audit_endpoint: http://kube-apiserver:8080/audit

Alerting and Response

Falco Sidekick Integration

# Forward alerts to Slack
apiVersion: v1
kind: ConfigMap
metadata:
  name: falco-sidekick-config
data:
  config.yaml: |
    slack:
      webhookurl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
      minimumpriority: "warning"

Automated Response

Falco can trigger automated responses:

# Example: Scale down deployment on critical alert
- rule: Critical security violation
  desc: Detect critical security violations
  condition: >
    evt.type = execve and suspicious_process
  output: >
    Critical security violation detected (process=%proc.name
    container=%container.name)
  priority: CRITICAL
  actions:
    - scale_down_deployment
    - notify_security_team

Comparison: Falco vs Traditional Security Monitoring

Advantages of Falco

• Real-Time Detection: System call monitoring provides immediate threat detection.
• Container Awareness: Understands Kubernetes context (pods, namespaces, labels).
• Low Overhead: eBPF-based monitoring has minimal performance impact.
• Policy Flexibility: Custom rules enable organization-specific security policies.

Advantages of Traditional Monitoring

• Mature Ecosystem: Integration with existing SIEM and security tools.
• Log Analysis: Can analyze historical logs for forensic analysis.
• Network Monitoring: Better visibility into network-level threats.
• Compliance: May be required for compliance with specific standards.

Practical Considerations

Performance Impact

Falco’s eBPF-based monitoring has minimal performance impact:

• CPU Overhead: Typically < 5% CPU per node.
• Memory Usage: ~100-200MB per Falco instance.
• Network Impact: Minimal (alerts only, no data exfiltration).

Rule Management

• Default Rules: Falco ships with comprehensive default rules.
• Custom Rules: Create custom rules for organization-specific requirements.
• Rule Testing: Test rules in non-production environments first.
• Rule Maintenance: Update rules as threats evolve.

Alert Fatigue

Too many alerts can lead to alert fatigue:

• Tune Rules: Adjust rule priorities and conditions to reduce false positives.
• Alert Aggregation: Use Falco Sidekick to aggregate and filter alerts.
• Response Automation: Automate responses to common, low-risk alerts.

Getting Started

# Install Falco using Helm
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco

# Or deploy using DaemonSet
kubectl apply -f https://raw.githubusercontent.com/falcosecurity/falco/master/deploy/kubernetes/falco-daemonset.yaml

# View Falco logs
kubectl logs -f -l app=falco

Caveats & Lessons Learned

• Privileged Access: Falco requires privileged access to monitor system calls.
• Rule Complexity: Complex rules can impact performance; optimize carefully.
• False Positives: Tune rules to reduce false positives while maintaining security coverage.
• Kubernetes Version: Some Falco features require specific Kubernetes versions.

Common Failure Modes

• “Too many alerts”: Overly sensitive rules generate alert fatigue; tune rules.
• “Missing detections”: Rules not covering all threat scenarios; review and update rules.
• “Performance issues”: Complex rules impacting cluster performance; optimize rules.

Conclusion

Falco’s introduction in 2018 marked a significant advancement in Kubernetes runtime security. It provided teams with real-time visibility into cluster activity, enabling proactive threat detection and response. While network policies and RBAC provided preventive controls, Falco added detective controls that were essential for comprehensive security.

For organizations deploying Kubernetes in production, Falco became an essential tool for runtime security monitoring. It demonstrated that security wasn’t just about preventing attacks—it was also about detecting and responding to threats in real-time. Falco’s CNCF acceptance validated its importance in the cloud-native security ecosystem.

The patterns and practices established with Falco would influence the development of other runtime security tools (KubeArmor, Tracee) and set the standard for cloud-native threat detection. Falco proved that Kubernetes security could be both comprehensive and performant, enabling teams to secure their clusters without sacrificing operational efficiency.