Lab 6: Cilium eBPF Kubernetes Networking + Hubble Observability

Week: 5-6 -- eBPF/XDP + NSM at Scale
Points: 15
Time estimate: 60 min lab + 2.5 hr independent
Deliverable: lab-6-report.md + policy YAML files

Objectives

1. Deploy a local Kubernetes cluster with Cilium as the CNI plugin using kind.
2. Configure a CiliumNetworkPolicy that enforces L7 HTTP path filtering between two microservices.
3. Use Hubble CLI to observe traffic flows and confirm policy enforcement.
4. Deploy Tetragon and observe a sys_execve system-call event from inside a running pod.

Prerequisites

Install the toolchain (covered in SETUP.md):

# kind cluster manager
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/

# Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

# Hubble CLI (Cilium observability layer)
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin

# Helm (for Cilium/Tetragon install)
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Part A: Cilium Cluster Bring-Up (20 min)

Create a kind cluster configured for Cilium:

# kind config -- no default CNI so Cilium can install its own
cat > lab6/kind-cilium.yaml << 'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "10.244.0.0/16"
EOF

kind create cluster --name lab6 --config lab6/kind-cilium.yaml

# Install Cilium with Hubble enabled
cilium install \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# Wait for Cilium to be ready
cilium status --wait

# Verify all nodes are Ready
kubectl get nodes

Expected output: all 3 nodes in Ready state; cilium status showing green across datapath, controller, hubble.

Deploy the two microservices for policy testing:

# Create the test namespace
kubectl create namespace lab6

# Deploy a simple HTTP server (the "backend")
cat > lab6/backend.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: lab6
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
        tier: api
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: lab6
spec:
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 80
EOF

# Deploy a client pod (the "frontend")
cat > lab6/frontend.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: frontend
  namespace: lab6
  labels:
    app: frontend
    tier: web
spec:
  containers:
  - name: curl
    image: curlimages/curl:latest
    command: ["sleep", "infinity"]
EOF

kubectl apply -f lab6/backend.yaml
kubectl apply -f lab6/frontend.yaml

# Wait for pods to be running
kubectl wait --for=condition=Ready pod -l app=backend -n lab6 --timeout=120s
kubectl wait --for=condition=Ready pod frontend -n lab6 --timeout=60s

# Confirm baseline connectivity (before policy)
kubectl exec -n lab6 frontend -- curl -s http://backend/get | head -5
kubectl exec -n lab6 frontend -- curl -s http://backend/post -X POST | head -5

Record: do both /get and /post return HTTP 200 before any policy is applied?

Part B: L7 CiliumNetworkPolicy (20 min)

Apply a policy that permits GET to /get and /status/* but blocks POST and all other paths:

cat > lab6/l7-policy.yaml << 'EOF'
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: backend-l7-policy
  namespace: lab6
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/get"
        - method: "GET"
          path: "/status/[0-9]+"
EOF

kubectl apply -f lab6/l7-policy.yaml

# Wait for policy to propagate
sleep 5

Test policy enforcement:

# Allowed: GET /get
kubectl exec -n lab6 frontend -- curl -s -o /dev/null -w "%{http_code}" http://backend/get
# Expected: 200

# Allowed: GET /status/200
kubectl exec -n lab6 frontend -- curl -s -o /dev/null -w "%{http_code}" http://backend/status/200
# Expected: 200

# Blocked: POST to /post (method not in policy)
kubectl exec -n lab6 frontend -- curl -s -o /dev/null -w "%{http_code}" -X POST http://backend/post
# Expected: 403

# Blocked: GET /anything (path not in policy)
kubectl exec -n lab6 frontend -- curl -s -o /dev/null -w "%{http_code}" http://backend/anything
# Expected: 403

Record the HTTP response codes for all four tests.

Part C: Hubble Flow Observation (15 min)

Enable Hubble port-forward and use the CLI to observe flows:

# Forward Hubble relay port
cilium hubble port-forward &
sleep 3

# Check Hubble status
hubble status

# Observe all flows in the lab6 namespace (run in background)
hubble observe --namespace lab6 --follow &
HUBBLE_PID=$!

# Trigger some traffic
kubectl exec -n lab6 frontend -- curl -s http://backend/get > /dev/null
kubectl exec -n lab6 frontend -- curl -s -X POST http://backend/post > /dev/null
kubectl exec -n lab6 frontend -- curl -s http://backend/anything > /dev/null

sleep 3
kill $HUBBLE_PID

From the Hubble output, identify:

1. The flow entries for the allowed GET /get request: what verdict does Hubble report?
2. The flow entries for the blocked POST /post request: what verdict does Hubble report?
3. Does Hubble show the full HTTP method and URL path for L7 flows?

# Query specific flows after the fact
hubble observe --namespace lab6 \
  --verdict DROPPED \
  --last 20

hubble observe --namespace lab6 \
  --verdict FORWARDED \
  --last 20

Part D: Tetragon System-Call Tracing (5 min)

Deploy Tetragon and observe execve events from inside a pod:

# Install Tetragon via Helm
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system

# Wait for Tetragon daemonset
kubectl rollout status -n kube-system ds/tetragon --timeout=60s

# Apply the TracingPolicy for execve events
cat > lab6/execve-tracing.yaml << 'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: execve-tracing
spec:
  kprobes:
  - call: "sys_execve"
    syscall: true
    args:
    - index: 0
      type: "string"
EOF

kubectl apply -f lab6/execve-tracing.yaml

# Watch Tetragon events
kubectl exec -n kube-system ds/tetragon -c export-stdout -- \
  tetra getevents --namespaces lab6 &
TETRA_PID=$!

# Trigger an execve inside the frontend pod
kubectl exec -n lab6 frontend -- sh -c "ls /tmp"

sleep 3
kill $TETRA_PID 2>/dev/null

Record: what process_kprobe event does Tetragon report for the ls /tmp command? What fields are visible (namespace, pod name, binary path, arguments)?

Lab Report

Create lab-6-report.md with:

1. Cluster bring-up: cilium status output confirming all green
2. Baseline connectivity: HTTP codes for GET /get and POST /post before policy
3. Policy enforcement table:

4. Hubble observations: copy 3-4 representative flow lines (one FORWARDED, one DROPPED); note whether L7 HTTP method/path is visible
5. Tetragon event: paste the JSON event for the ls /tmp execve; identify the pod, binary, and arguments fields
6. One-paragraph analysis: "Cilium enforces L7 policies without a sidecar proxy. Explain how eBPF achieves this compared to a service-mesh model like Istio/Envoy. Name one operational advantage and one limitation."

Grading