Renew Certificates

To renew the Kubernetes internal pods mTLS certificates, perform the following steps:

1. Stop kubelet

    sudo systemctl stop kubelet
   

2. Renew all internal K8s certs

    sudo kubeadm alpha certs renew all
   

3. Update the config for kubectl

    mv .kube/config .kube/config.old
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    sudo chmod 777 $HOME/.kube/config
    export KUBECONFIG=.kube/config
   

4. Reboot the node

    sudo shutdown now -r
   

(Hacky) Option - Turn back system clock

1. Stop kubelet

    sudo systemctl stop kubelet
   

2. Set back the Controller system clock to August 1 2021

    sudo timedatectl set-ntp off
    sudo timedatectl set-time 2021-08-01
   

3. Start kubelet

    sudo systemctl start kubelet
   

4. Monitor the lazy kubelet certificate rotation process

    sudo ls -l /var/lib/kubelet/pki
   

   My observations are that this process took approximately 5 minutes and it is randomly variable and described as a lazy process.

5. Set the date back to ‘today’ before re-generating the K8s internal mTLS certificates

    sudo timedatectl set-time 2021-08-23 
   

6. Verify that the Kubernetes mTLS internal certificates are expired and need to be renewed. This can be validated with observing all internal kubernetes certificates:

    find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs
    -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After' or sudo
    kubeadm alpha certs check-expiration