Implementing Kubernetes and Docker on OCI

Introduction

This was an assignment for Cloud Security class in semester 1, 2022. The requirements asked me to pack a web application into Docker and then deploy the service on the cloud using Kubernetes. The entire lab environment was built on OCI (Oracle Cloud Infrastructure). Here are my configuration notes and the pitfalls I stepped through to help take the easy way out when deploying services to OCI in the future.

This diary focuses on the deployment phase. Since OCI has some pre-defined iptables rules that can block communications between nodes, security lists also affect internal communication between VMs, even if they are in the same subnet. I needed to modify the iptables and security lists as below:

Pre-Configuration

iptables (controller & nodes)

## save existing rules
sudo iptables-save > ~/iptables-rules

## modify rules, remove drop and reject lines
grep -v "DROP" iptables-rules > tmpfile && mv tmpfile iptables-rules-mod
grep -v "REJECT" iptables-rules-mod > tmpfile && mv tmpfile iptables-rules-mod

## apply the modifications
sudo iptables-restore < ~/iwptables-rules-mod

## check
sudo iptables -L

## save the changes
sudo netfilter-persistent save
sudo systemctl restart iptables

## letting iptables see bridged traffic
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sudo sysctl --system

Security Lists

For easy configuration, I opened everything, so security lists ended up with only three rules:

• 0.0.0.0/0; TCP; All ports
• 0.0.0.0/0; UDP; All ports
• 0.0.0.0/0; ICMP

But make sure that you refer to Kubernetes documentation and related CNI documentation to narrow down the open ports and IP ranges after implementation.

Configuration

After the pre-configuration part, we can start installing Kubernetes and Docker on all the VMs.

System updates (controller & nodes)

sudo apt update

Docker (controller & nodes)

sudo apt -y install docker.io
sudo nano /etc/docker/daemon.json

In daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

Then run

sudo systemctl restart docker

To check if Cgroup is systemd

sudo docker info | grep group

K8S Installation (controller & nodes)

curl -s <https://packages.cloud.google.com/apt/doc/apt-key.gpg> | sudo apt-key add
sudo apt-add-repository "deb <http://apt.kubernetes.io/> kubernetes-xenial main"
sudo apt-get -y install kubeadm kubelet kubectl
sudo apt-mark hold kubeadm kubelet kubectl

Post-Configuration

K8S init (controller)

sudo kubeadm init --pod-network-cidr=[CNI required range] --apiserver-advertise-address=[local IP address]
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

--pod-network-cidr:

• flannel: 10.244.0.0/16
• weave: 10.32.0.0/12
• calico: 192.168.0.0/16

apply CNI

# flannel
kubectl apply -f <https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml>

# weave
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

# calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico-typha.yaml

K8S join (nodes)

sudo kubeadm join 10.0.0.145:6443 --token rehcz9.67jf80xbe86oj9j1 \\
        --discovery-token-ca-cert-hash sha256:c4275404b872ce55ccb2a9e7df48a9b8d40010c1bc35e4ec19278a449de9bad5

deployement.yml

kubectl apply -f deployement.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: [this is a name]-deployment
spec:
  selector:
    matchLabels:
      app: [this is a name]
  replicas: 1
  template:
    metadata:
      labels:
        app: [this is a name]
    spec:
      containers:
      - name: [this is a name]
        image: [this is docker image]
        resources:
          limits:
            cpu: "0.5"
            memory: "512Mi"
          requests:
            cpu: "0.5"
            memory: "512Mi"
        ports:
        - containerPort: 5000

service.yml

kubectl apply -f service.yml

apiVersion: v1
kind: Service
metadata:
  name: [this is a name]-service
spec:
  type: NodePort
  selector:
    app: [this is a name]
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 5000
    nodePort: 30007

Check Docker and Kubelet Status

sudo systemctl status docker
sudo systemctl status kubelet

Monitor K8S Resources

watch kubectl get deployment,svc,pods,nodes -o wide --all-namespaces

Command to Revert

K8S

sudo kubeadm reset
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
sudo apt-get autoremove
sudo rm -rf ~/.kube
sudo rm -rf /etc/kubernetes
sudo rm -rf /etc/cni/net.d
sudo rm -rf /var/lib/kubelet
sudo rm -rf /var/lib/etcd

Docker

sudo docker system prune -a
sudo apt remove docker.io
sudo apt autoremove

!!! Pitfalls !!!

1. Everything is well configured; I can curl localhost:30033 from nodes or curl [node_ip]:30033 from the controller, but not curl localhost:30033 from the controller.

   This is because the iptables and security lists block the communication between kube-proxies. Refers to the Pre-Configuration part.

2. Timeout while joining controller from nodes.

   This is highly due to improper configurations on security lists. We have to open TCP port 6443 for both ways.

3. Pods show running, but the READY state is 0/1 for the DNS pod.

   This happens when I first implement flannel as my OCI, I finally figured out this issue is also related to iptables. The traffics are DROPPED by rules.

4. dial tcp 127.0.0.1:10248: connect: connection refused when I run kubeadm init.

   This is because Docker cgroup is not systemd.

5. failed to get config map: Unauthorised while joining nodes to the controller.

   The token is invalid, run sudo kubeadm token create --print-join-command on the controller to create a new one.

6. Port 10250 is in use

   This is because kubelet is using this port, run sudo systemctl restart kubelet or sudo kubeadm reset

References

1. https://faun.pub/free-ha-multi-architecture-kubernetes-cluster-from-oracle-c66b8ce7cc37
2. https://medium.com/platformer-blog/running-a-kubernetes-cluster-on-ubuntu-with-calico-9e372fb9175e
3. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
4. https://github.com/flannel-io/flannel
5. https://stackoverflow.com/a/60725618
6. https://kubernetes.io/docs/reference/ports-and-protocols/
7. https://kubernetes.io/docs/concepts/services-networking/service/
8. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
9. https://docs.docker.com/engine/reference/commandline/dockerd/
10. http://www.dest-unreach.org/socat/doc/socat.html#EXAMPLES
11. https://blog.csdn.net/haveanybody/article/details/86494063
12. https://stackoverflow.com/a/58491408