darrylcauldwell.com On a journey around the datacenter and public cloud.

Introduction To Kubernetes Cluster Networking with Antrea

What is Antrea

Antrea is Container Network Interface (CNI) plugin for Kubernetes. The Antrea CNI leverages Open vSwitch (OVS) to provides an overlay network and security services for a Kubernetes cluster. The overlay encapsulation uses Virtual Extensible LAN (VXLAN) by default, although Generic Network Virtualization Encapsulation (GENEVE), Generic Routing Encapsulation (GRE) or Stateless Transport Tunneling (STT) encapsulation can be configured. Antrea is an open source project hosted on GitHub here. Using OVS also introduces other standard network management capabilities such NetFlow, sFlow, IP Flow Information Export (IPFIX) and Remote SPAN (RSPAN).

Architecture

Antrea is composed of several components deployed as Pods in the kube-system namespace of the cluster. The Antrea Conroller monitors the addition and deletion of objects by watching the Kubernetes API. A daemonset is deployed which druns Antrea Agent on each node the Agent applys flow configuration to Open vSwitch (OVS).

The network configration required by Kubernetes is configured in a collection of Open vSwitch (OVS) flow tables. To make it easy to interpret each type of network traffic is classified and each traffic classification is stored in its own flow table.

Creating A Simple Antrea Lab

We can create a lab to look at Antrea anywhere we can run some VMs so on either a public cloud or a homelab. Create three Ubuntu 19.10 VMs with 2x CPU, 4GB RAM and 50GB vHDD, use Netplan to configure NIC to static IP addressing like.

cat /etc/netplan/01-netcfg.yaml 

network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      addresses: 
      - 192.168.1.27/24
      gateway4: 192.168.1.254
      nameservers:
          search:
          - darrylcauldwell.com
          addresses: 
          - 192.168.1.10

Antrea requires Kubernetes 1.16 or later. Install Docker, Open vSwitch and Kubernetes by running the following.

sudo apt update -y
sudo apt upgrade -y
sudo apt install docker.io python apt-transport-https openvswitch-switch -y
sudo gpasswd -a $USER docker
sudo systemctl start docker
sudo systemctl enable docker
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt-get update
sudo swapoff -a 
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo apt-get install -y kubelet=1.16.4-00 kubeadm=1.16.4-00 kubectl=1.16.4-00

To form the overlay network Antrea requires an IP address block to allocate IP addresses (–pod-network-cidr). We can use kubeadm to configure the IP address block and bootstrap the cluster by running the following:

sudo kubeadm init --pod-network-cidr=172.16.0.0/16

In order to easily run kubectl from the master we can copy the kube config to our user profile by running the following.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

With the control plane initiatlised we can now get the cluster token from the master we can use this to add the two worker Nodes to the cluster.

sudo kubeadm join 192.168.1.27:6443 --token 4bp2f0.ppfjk7mfee89j2kd \
>     --discovery-token-ca-cert-hash sha256:5e79610f28840c15be46895340c4a5535b9a0d003741ed657961891a05987ccd 

All the Antrea components are containerized and can be installed using the Kubernetes deployment manifest. We can apply the default Antrea manifest supplied in GitHub repository which will deploy all necessary components.

kubectl apply -f https://raw.githubusercontent.com/vmware-tanzu/antrea/master/build/yamls/antrea.yml

The Antrea components are deployed to the kube-system Namespace. The components include Deploymentm, Pods, Daemonset and ConfigMap.

kubectl get all --namespace kube-system
kubectl get configmap --namespace kube-system

The ConfigMap defines antrea-agent.conf this is used by Daemonset on each Node to configure Open vSwitch for use with CNI we can see some key information like.

kubectl describe configmap antrea-config-tm7bht9mg6

Integration Bridge name (default: br-int)
DataPath type (default: system)
Interface name to communicate with host (default: gw0)
Tunnel (Encapsulation) type (default: vxlan)
MTU value (default: 1450)
Service CIDR (default 10.96.0.0/12)

Exploring The Open vSwitch Overlay Network

With Antrea in place we can look at exploring the overlay network it has put in place. We create a Kubernetes Namespace and deploy an simple stateless application like guestbook.

kubectl create -f https://k8s.io/examples/admin/namespace-dev.json
kubectl config set-context --current --namespace=development
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-deployment.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-service.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-slave-deployment.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-slave-service.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml

When running we can view the Pods IP addressing.

kubectl get pods --output wide

NAME                            READY   STATUS    RESTARTS   AGE     IP           NODE              NOMINATED NODE   READINESS GATES
frontend-6cb7f8bd65-25qv4       1/1     Running   0          5m47s   172.16.2.4   antrea-worker-2   <none>           <none>
frontend-6cb7f8bd65-mn5jk       1/1     Running   0          5m47s   172.16.1.5   antrea-worker-1   <none>           <none>
frontend-6cb7f8bd65-nr9zk       1/1     Running   0          5m47s   172.16.2.5   antrea-worker-2   <none>           <none>
redis-master-7db7f6579f-4wdnj   1/1     Running   0          5m51s   172.16.2.2   antrea-worker-2   <none>           <none>
redis-slave-7664787fbc-5xrvj    1/1     Running   0          5m49s   172.16.1.4   antrea-worker-1   <none>           <none>
redis-slave-7664787fbc-nj6m5    1/1     Running   0          5m49s   172.16.2.3   antrea-worker-2   <none>           <none>

We can see that IPs from 172.16.1.0/24 are issued to Pods running on Node antrea-worker-1 and 172.16.2.0/24 are issued to Pods running on Node antrea-worker-2. To facilitate communications between Pods the antrea-agent configures flows on Open vSwitch on each Node. If we connect to a Antrea Agent container we can see that an OVS bridge is created called br-int and the bridge has a vxlan tunnel port called tun0 and a gateway port called gw0. The internal port gw0 is allocated the role of gateway of the Node’s subnet and is allocated the first IP address in subnet.

kubectl config set-context --current --namespace=kube-system
kubectl get pods --selector=component=antrea-agent --output wide

NAME                 READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
antrea-agent-czksb   2/2     Running   0          14m   192.168.1.28   antrea-worker-1   <none>           <none>
antrea-agent-gwkmr   2/2     Running   0          14m   192.168.1.27   antrea-master     <none>           <none>
antrea-agent-x9xjk   2/2     Running   0          14m   192.168.1.29   antrea-worker-2   <none>           <none>

kubectl exec -it antrea-agent-czksb -c antrea-ovs bash

ovs-vsctl list-br
ovs-vsctl show

3cf39eec-f64c-49ed-ad86-48f203b215a4
    Bridge br-int
        Port "tun0"
            Interface "tun0"
                type: vxlan
                options: {key=flow, remote_ip=flow}
        Port "coredns--886217"
            Interface "coredns--886217"
        Port "redis-sl-b32512"
            Interface "redis-sl-b32512"
        Port "coredns--3d5851"
            Interface "coredns--3d5851"
        Port "gw0"
            Interface "gw0"
                type: internal
        Port "frontend-1ee3a9"
            Interface "frontend-1ee3a9"
    ovs_version: "2.11.1"

We can also see that ArpResponderTable (20) and L3ForwardingTable (70) have flow records relating to the pod network.

kubectl config set-context --current --namespace=kube-system
kubectl exec -it antrea-agent-czksb -c antrea-ovs bash
ovs-ofctl dump-flows br-int | grep 172.16.2

 cookie=0xf70e020000000000, duration=820.733s, table=20, n_packets=1, n_bytes=42, idle_age=585, priority=200,arp,arp_tpa=172.16.2.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:aa:bb:cc:dd:ee:ff,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],load:0xaabbccddeeff->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xac100201->NXM_OF_ARP_SPA[],IN_PORT

 cookie=0xf70e020000000000, duration=820.733s, table=70, n_packets=649, n_bytes=63996, idle_age=0, priority=200,ip,nw_dst=172.16.2.0/24 actions=dec_ttl,mod_dl_src:06:d1:bb:d3:bc:fa,mod_dl_dst:aa:bb:cc:dd:ee:ff,load:0x1->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[16],load:0xc0a8011d->NXM_NX_TUN_IPV4_DST[],resubmit(,105)

We can look to test the IP routing and connectivity between Pods on same Node and also between Nodes,

kubectl config set-context --current --namespace=development
kubectl get pods --output wide | grep frontend

frontend-6cb7f8bd65-25qv4       1/1     Running   0          12m   172.16.2.4   antrea-worker-2   <none>           <none>
frontend-6cb7f8bd65-mn5jk       1/1     Running   0          12m   172.16.1.5   antrea-worker-1   <none>           <none>
frontend-6cb7f8bd65-nr9zk       1/1     Running   0          12m   172.16.2.5   antrea-worker-2   <none>           <none>

kubectl exec -it frontend-6cb7f8bd65-25qv4 -- ping -c 1 172.16.2.5
kubectl exec -it frontend-6cb7f8bd65-25qv4 -- ping -c 1 172.16.1.5

Exploring Network Policy

Antrea implements NetworkPolicy using OVS Flows. Flows are organized in tables, and they are applied on each node by the Antrea agent. Before applying a network policy, we can check flow table IngressDefault (100) and IngressRuleTable (90)

kubectl config set-context --current --namespace=kube-system
kubectl get pods | grep antrea-agent

kubectl exec -it antrea-agent-czksb -c antrea-agent ovs-ofctl dump-flows br-int | grep table=100

cookie=0xcac6000000000000, duration=9103.902s, table=100, n_packets=102, n_bytes=9696, priority=0 actions=resubmit(,105)
kubectl exec -it antrea-agent-czksb -c antrea-agent ovs-ofctl dump-flows br-int | grep table=90

cookie=0xcac6000000000000, duration=4059.159s, table=90, n_packets=32493, n_bytes=6556290, priority=210,ct_state=-new+est,ip actions=resubmit(,105)

cookie=0xcac6000000000000, duration=4059.159s, table=90, n_packets=1634, n_bytes=122104, priority=210,ip,nw_src=172.16.1.1 actions=resubmit(,105)

cookie=0xcac6000000000000, duration=4059.159s, table=90, n_packets=37, n_bytes=3768, priority=0 actions=resubmit(,100)

We can confirm that frontend can ping the backend.

kubectl config set-context --current --namespace=development
kubectl get pods -o wide
kubectl exec -it frontend-6cb7f8bd65-25qv4 -- ping -c 1 172.16.2.27

We can create a network policy to deny all ingress.

cat <<EOF >/tmp/deny-all.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
EOF

kubectl create -f /tmp/deny-all.yaml

If we test again we can test and ensure that the policy is applied and the frontend can no longer ping backend.

kubectl exec -it frontend-6cb7f8bd65-25qv4 -- ping -c 1 172.16.2.27

We can then check IngressDefault (100) flow table and see that our network policy has added action to drop.

kubectl config set-context --current --namespace=kube-system

kubectl exec -it antrea-agent-czksb -c antrea-agent ovs-ofctl dump-flows br-int | grep table=100
 cookie=0xcac6000000000000, duration=3.427s, table=100, n_packets=0, n_bytes=0, priority=200,ip,reg1=0xa actions=drop

 cookie=0xcac6000000000000, duration=3.427s, table=100, n_packets=0, n_bytes=0, priority=200,ip,reg1=0x8 actions=drop

 cookie=0xcac6000000000000, duration=9226.746s, table=100, n_packets=137, n_bytes=12966, priority=0 actions=resubmit(,105)

Be social and share this post!