The KKK Kubernetes Cluster
Seriously guys, were you not using your brains when you named all of this? Today we’ll focus on the three K’s: Kubectl, Kubeadm and Kubelet.
Basic Theories
Let’s get your head around how this whole thing works. Firstly, there’s an awesome comic made by Scott McCloud that can be seen here. This will help you wrap your brain around how powerful it is, however you’ll likely just stare at a blank terminal, paralyzed under the weight of freedom, unable to progress the first time that you try to pick it up.
You can transfer a little bit of your hypervisor knowledge forward onto Kubernetes. At a basic level, you will be creating three hosts which will run a bunch of containers. I’m just going to assume that you know what a container is by this point.
The KKK suite helps us delegate which host will run which container, and how each container will work. It works just fine with Docker, so a lot of the images propping Docker up are transferrable into a Kubernetes cluster. Firstly, you’re going to need to configure your three starter hosts, configure one as a master (giggity) and the other two as slaves.
The idea behind what we want to do is have three machines configured to run containers, and let the KKK suite control what runs where. We want to configure things so that it doesn’t matter what host the container runs on, but I’ll also be running you through how to use labels to assign certain containers to specific hosts.
Configuring your Hosts (CentOS)
I swear, I’m getting de-ja-vu on this documentation.
I haven’t tried this out on Debian hosts yet, although I probably should. I wrote an Ansible role that will setup the host for us. The steps boil down to:
- Ensure all hosts can communicate with each other over hostname (I just used a hosts file)
- Configure Docker and Kubernetes repos
- Configure kernel for k8s
- Configure Docker to use systemd for it’s cgroup driver instead of systemd
- Disable swap
- Run a system update
- Configure the firewall for the k8s plane
- Install & Configure the KKK
- Ensure that it is all running
- Register the worker nodes with the master.
The steps to do this are acurately documented in the role, so I recommend reading through that if you’re not sure.
The link to the Ansible role: https://github.com/surprise-automation/ansible-first-k8s-cluster
1. Ensure Host Communication
Personally, I just did the following:
$ cat << EOF >> /etc/hosts
10.0.0.2 ed
10.0.0.3 edd
10.0.0.4 eddie
EOF
You can handle this with a properly configured DNS server though.
2. Configure Docker and Kubernetes Repos
The below is solely to configure the repos, as per steps above.
Docker
You can check the official installation guide here: https://docs.docker.com/engine/install/centos/
If you’re running on a fresh system, here’s the copypasta:
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
yum -y install yum-utils
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
# yum -y install docker-ce docker-ce-cli containerd.io
# systemctl enable --now docker
Kubernetes
To install Kubernetes using official repos, check out the official installation guide here: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management
Again, on a fresh system:
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
# yum --disableexcludes=kubernetes -y install kubectl kubelet kubeadm
The “disable excludes” flag is included because we have excluded the KKK trio from the kubernetes repo. There’s a warning on the kubeadm install page which explains this however the same warning isn’t present for kubectl and kubelet:
Warning: These instructions exclude all Kubernetes packages from any system upgrades. This is because kubeadm and Kubernetes require special attention to upgrade.
If you review the playbook that I have posted, you’ll see that we also include the trio in our .repo file and disableexcludes when installing in the roles tasks/main.yml file.
3. Configure Kernel for k8s
Your kernel should include the bridge module by default and you should be pretty aware if you have manually disabled it, or if it’s a feature of your distro. Regardless, RedHat flavors have the bridge module installed by default however you still need to the bridge to send packets to iptables prior to sending them to your containers.
My understanding on this is foggy, and I welcome ammendments via email, however I’ve sourced my information from the libvirt wiki.
I believe this comes from the fact that Docker/Kubernetes will leverage and control your hosts iptables installation to route traffic around. As such, traffic needs to pass through the iptables firewall in order to get to the containers correctly. The way to do this is pretty easy; drop the following into /etc/sysctl.d/k8s.conf:
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
You additionally need to enable the br_netfilter module for your kernel. Direct from Oracle, “This module is required to enable transparent masquerading and to facilitate Virtual Extensible LAN (VxLAN) traffic for communication between Kubernetes pods across the cluster. [source]”.
Create the file /etc/modules-load.d/k8s.conf and populate with:
br_netfilter
I’m fuzzy on loading kernels as it’s been a while since I ran Gentoo or Arch, but at this point a reboot wouldn’t hurt you. You can hot-load the kernel module with modprobe if you don’t want to reboot, but you’ll still need the above to load on boot:
modprobe br_netfilter
4. Configure Docker to use systemd for it’s cgroup driver instead of systemd
This is really important. If you skip this like I did, you’ll run into dramas. I did not document the error I received, however if you run into any cgroup driver errors - and you will if you don’t do this - then this is the solution.
cat << EOF > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
5. Disable Swap
To permanent disable swaps on your system, comment out or delete the lines in /etc/fstab pertaining to swap. A swap line will look like this:
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx swap swap defaults 0 0
This will not take effect until a reboot, or maybe a mount -a however I have not tested the latter. To avoid a reboot at this point but still immediately disable swap, run the following to disable all swapping on the system:
swapoff -a
6. Run a system update
Naturally, you’ll want to run a full system update. You can do this at any point but I prefer to do it before installing KKK.
yum -y update
7. Configure the firewall for the k8s plane
The final piece of the puzzle is the firewall. You need to open all the management ports that Kubernetes uses.
Here’s some firewalld copy pasta for your enjoyment:
firewall-cmd --permanent --zone=public --add-port=179/tcp
firewall-cmd --permanent --zone=public --add-port=6443/tcp
firewall-cmd --permanent --zone=public --add-port=2379/tcp
firewall-cmd --permanent --zone=public --add-port=2380/tcp
firewall-cmd --permanent --zone=public --add-port=10250/tcp
firewall-cmd --permanent --zone=public --add-port=10251/tcp
firewall-cmd --permanent --zone=public --add-port=10252/tcp
firewall-cmd --permanent --zone=public --add-port=30000-32767/tcp
firewall-cmd --reload
8. Install & Configure the KKK
Finally, the moment we have all been waiting for. Installing KKK.
Install Kubeadm, Kubelet & Kubectl
yum --disableexcludes=kubernetes -y install kubelet kubeadm kubectl
Again, note that we are disabling excludes on the kubernetes repo.
Configuration
I have added installing the network prior to creating the control plane as it makes logical sense but it’s entirely theoretical. It goes against the docs but if I recall my experience correctly, I think it should have been done in this order.
If you run the network install prior to any control plane initialization and run into errors, please contact me.
Install Network: Project Calico
You need to install your network, which is a weird concept to me personally. I opt for Project Calico as it’s easy to get started with.
If you read their official quickstart guide, you will issues. They don’t tell you that you need to edit their manifest prior to ingesting with kubectl.
The best way to do this and use a custom CIDR is to download the manifest, make your edits, then ingest it:
curl https://docs.projectcalico.org/manifests/calico.yaml -O calico.yaml
sed -i 's/# - name: CALICO_IPV4POOL_CIDR/ - name: CALICO_IPV4POOL_CIDR/' calico.yaml
sed -i 's/# value: "192.168.0.0/16"/ value: "10.8.0.0/16"/' calico.yaml
You can then apply it with kubectl:
kubectl apply -f calico.yaml
To check if it’s applying to all your nodes:
kubectl get pods -n kube-system
You should eventually see the STATUS turn to Running
Create Control Plane
On the host that is to be the master, run the following:
kubeadm init --pod-network-cidr=10.8.0.0/16
Note the join command it gives you, as it’ll make your life easier when you join the workers:
kubeadm join <masters ip>:6443 --token <token> \
--discovery-token-ca-cert-hash <some hash>
9. Ensure that it is all running
To verify that everything is talking, you can run the following to get details of all pods:
kubectl get pods
10. Register the worker nodes with the master.
When you build your master node, you’ll have been given a command for binding worker nodes to the master. All that you need to do is run this command on the worker node, presuming that it can connect to the master, and let kubeadm do it’s thing.
You can check pod status via:
kubectl get pods
Your join command will look something like this:
kubeadm join <masters ip>:6443 --token <token> \
--discovery-token-ca-cert-hash <some hash>
Run the above on the workers to join them to the master. You can run the following to get there join status:
kubectl get pods -n kube-system
Extras
Something that you should be aware of; kubectl requires that you have ~/.kube/config created and configured in order for kubectl to function correctly.
kubectl looks at ~/.kube/config to know how to connect to the api. If this is not configured, you’ll get a no response from localhost:8080 type of message, which appears to be the default port kubectl uses.
When running things as root, like the total badass that I am:
mkdir ~/.kube
ln -s /etc/kubernetes/admin.conf ~/.kube/config
If you’re running as a general user, I’d recommend just copying it:
[root@host ~]# mkdir /home/user/.kube/
[root@host ~]# cp /etc/kubernetes/admin.conf /home/user/.kube/config
[root@host ~]# chown -R user:group /home/user/.kube
[user@host ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcd69978-k5sbq 0/1 Pending 0 95m
coredns-78fcd69978-xzqhv 0/1 Pending 0 95m
etcd-edd 1/1 Running 1 95m
kube-apiserver-edd 1/1 Running 1 95m
kube-controller-manager-edd 1/1 Running 1 95m
kube-proxy-82xhn 1/1 Running 0 56m
kube-proxy-8t4kg 1/1 Running 0 95m
kube-proxy-dz6hp 1/1 Running 0 56m
kube-scheduler-edd 1/1 Running 1 95m
[user@host ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
edd NotReady control-plane,master 105m v1.22.1
That’s it. Thanks for coming to my TED Talk.
