How to Setup Kubernetes Cluster on Ubuntu 20.04 LTS with kubeadm

Kubeadm is a tool to build Kubernetes clusters. It’s responsible for cluster bootstrapping and support upgrades, downgrades, and managing bootstrap tokens. First of all Kubeadm runs a series of prechecks to ensure that the machine is ready to run Kubernetes, bootstrapping the cluster kubeadm is downloading and installing the cluster control plane components and configuring all necessary cluster resources.

AWS Security Group Inbound Traffic Ports for Kubernetes Cluster

Control-plane node(s)

Worker node(s)

sudo apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release

Install Docker

sudo apt-get install docker.io
sudo usermod -aG docker $USER
sudo chmod 666 /var/run/docker.sock
sudo systemctl start docker.service
sudo systemctl status docker.service
sudo systemctl enable docker.service
sudo systemctl restart docker

Add Kubernetes GPG Key on All node
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Add Kubernetes apt repository on all node for Ubuntu.
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main

update the system packages
sudo apt-get update

Install Kubeadm, Kubelet, and Kubectl on All Nodes

sudo apt-get install -y kubelet=1.21.4-00 kubeadm=1.21.4-00 kubectl=1.21.4-00 kubernetes-cni

How to Setup Kubernetes Cluster on Ubuntu 20.04 LTS with kubeadm

$ sudo kubeadm init --pod-network-cidr 10.0.0.0/16

I0318 09:50:06.413354   20536 version.go:254] remote version is much newer: v1.23.5; falling back to: stable-1.21
[init] Using Kubernetes version: v1.21.11
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ip-172-31-21-67 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.21.67]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ip-172-31-21-67 localhost] and IPs [172.31.21.67 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ip-172-31-21-67 localhost] and IPs [172.31.21.67 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.005780 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ip-172-31-21-67 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node ip-172-31-21-67 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: bo6g2v.1g5np9bs5c2yvg87
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

Create new '.kube' configuration directory and copy the configuration 'admin.conf' from '/etc/kubernetes' directory.

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.31.21.67:6443 --token bo6g2v.1g5np9bs5c2yvg87 \
        --discovery-token-ca-cert-hash sha256:c095ea351cdff6eabbec851dced588052090b2595758a70d5d4772547f13571c

Worker Node

$ sudo kubeadm join 172.31.21.67:6443 --token bo6g2v.1g5np9bs5c2yvg87 \
>         --discovery-token-ca-cert-hash sha256:c095ea351cdff6eabbec851dced588052090b2595758a70d5d4772547f13571c
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

$ sudo kubectl get nodes
NAME              STATUS     ROLES                  AGE    VERSION
ip-172-31-21-67   NotReady   control-plane,master   4m6s   v1.21.4

$ sudo kubectl get nodes
NAME              STATUS   ROLES                  AGE   VERSION
ip-172-31-20-59   Ready    <none>                7m2s   v1.21.4
ip-172-31-21-67   Ready   control-plane,master   6m6s   v1.21.4

Turn off swap
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a

Enable kernel modules and configure sysctl.

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
clusterrole.rbac.authorization.k8s.io/flannel configured
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-558bd4d5db-dfq22                  1/1     Running   0          6m46s
kube-system   coredns-558bd4d5db-dtzwx                  1/1     Running   0          6m46s
kube-system   etcd-ip-172-31-5-181                      1/1     Running   0          6m59s
kube-system   kube-apiserver-ip-172-31-5-181            1/1     Running   0          6m59s
kube-system   kube-controller-manager-ip-172-31-5-181   1/1     Running   0          6m59s
kube-system   kube-flannel-ds-cfd9f                     1/1     Running   0          86s
kube-system   kube-flannel-ds-mqdqp                     1/1     Running   0          86s
kube-system   kube-proxy-jwmrf                          1/1     Running   0          6m47s
kube-system   kube-proxy-rp6jd                          1/1     Running   0          4m46s
kube-system   kube-scheduler-ip-172-31-5-181            1/1     Running   0          6m59s

$ kubectl get serviceaccounts
NAME      SECRETS   AGE
default   1         7m5s

$ kubectl -n kube-system get pods -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE              NOMINATED NODE   READINESS GATES
coredns-558bd4d5db-dfq22                  1/1     Running   0          7m14s   10.0.0.3       ip-172-31-5-181   <none>           <none>
coredns-558bd4d5db-dtzwx                  1/1     Running   0          7m14s   10.0.0.2       ip-172-31-5-181   <none>           <none>
etcd-ip-172-31-5-181                      1/1     Running   0          7m27s   172.31.5.181   ip-172-31-5-181   <none>           <none>
kube-apiserver-ip-172-31-5-181            1/1     Running   0          7m27s   172.31.5.181   ip-172-31-5-181   <none>           <none>
kube-controller-manager-ip-172-31-5-181   1/1     Running   0          7m27s   172.31.5.181   ip-172-31-5-181   <none>           <none>
kube-flannel-ds-cfd9f                     1/1     Running   0          114s    172.31.20.59   ip-172-31-20-59   <none>           <none>
kube-flannel-ds-mqdqp                     1/1     Running   0          114s    172.31.5.181   ip-172-31-5-181   <none>           <none>
kube-proxy-jwmrf                          1/1     Running   0          7m15s   172.31.5.181   ip-172-31-5-181   <none>           <none>
kube-proxy-rp6jd                          1/1     Running   0          5m14s   172.31.20.59   ip-172-31-20-59   <none>           <none>
kube-scheduler-ip-172-31-5-181            1/1     Running   0          7m27s   172.31.5.181   ip-172-31-5-181   <none>           <none>

$ kubectl -n kube-system get deployments
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
coredns   2/2     2            2           7m38s

$ kubectl -n kube-system get configmap coredns -oyaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2022-03-25T10:12:11Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "241"
  uid: 8213dc7e-2026-4428-bfe7-a2c4a49e4084

$ kubectl get nodes
NAME              STATUS   ROLES                  AGE    VERSION
ip-172-31-20-59   Ready    <none>                 8m6s   v1.21.4
ip-172-31-5-181   Ready    control-plane,master   10m    v1.21.4

kubectl get serviceaccounts
kubectl get pods --all-namespaces
kubectl -n kube-system get pods -o wide
kubectl -n kube-system get deployments
kubectl -n kube-system get configmap coredns -o yaml

Deploy Nginx microservice on Kubernetes

$ vim nginx-web.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx-app
    type: front-end
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30011
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx-app
    type: front-end
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-app
      type: front-end
  template:
    metadata:
      labels:
        app: nginx-app
        type: front-end
    spec:
      containers:
        - name: nginx-container
          image: nginx:latest

Let's create the nginx web server on kubeadm.

$ kubectl apply --filename nginx-web.yaml
service/nginx-service created
deployment.apps/nginx-deployment created

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-56cbd5d774-7n9z2   1/1     Running   0          9s
nginx-deployment-56cbd5d774-97hzg   1/1     Running   0          9s
nginx-deployment-56cbd5d774-kpkh2   1/1     Running   0          9s

$ kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP            NODE              NOMINATED NODE   READINESS GATES
nginx-deployment-56cbd5d774-7n9z2   1/1     Running   0          6m2s   10.244.2.15   ip-172-31-20-59   <none>           <none>
nginx-deployment-56cbd5d774-97hzg   1/1     Running   0          6m2s   10.244.2.14   ip-172-31-20-59   <none>           <none>
nginx-deployment-56cbd5d774-kpkh2   1/1     Running   0          6m2s   10.244.2.16   ip-172-31-20-59   <none>           <none>

$ kubectl get deployments
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           22s

$ kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP        27d
nginx-service   NodePort    10.110.241.41   <none>        80:30011/TCP   17s

Verify Deployed Services

$ kubectl exec nginx-deployment-56cbd5d774-7n9z2  -- curl http://localhost

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 <!DOCTYPE html>0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
  615  100   615    0     0   150k      0 --:--:-- --:--:-- --:--:--  150k

Remove the Nginx App from kubeadm

$ kubectl delete --filename nginx-web.yaml
service "nginx-service" deleted
deployment.apps "nginx-deployment" deleted

[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

Add the Docker Daemon configurations to use systemd as the cgroup driver.

cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

How to Remove Worker Node from Cluster:

Find the node with
kubectl get nodes

Drain it with 
kubectl drain mynode (Specify node name)

Delete it with 
kubectl delete node mynode