Kubernetes TLS Handshake Error After Renewing Certificates

Question

I have run into an issue getting logs since I renewed the certificates that expired on Kubernetes. The steps I took to renew are:

1. Remove symlink for /var/lib/kubelet/pki/kubelet-client-current.conf.
2. Remove /etc/kubernetes/kubelet.conf.
3. Run 'kubeadm init phase kubeconfig kubelet'.
4. Restart Kubelet.
5. Run 'kubeadm init phase kubelet-finalize all'.
6. Restart kubelet.
7. Run 'kubeadm certs renew all'.
8. Reboot.
9. Copy /etc/kubernetes/admin.conf to .kube/config.

I performed this on all three nodes.

The kubectl commands are working fine, the kubernetes dashboard is coming up fine, but the logs show:

Get "https://192.168.200.3:10250/containerLogs/ninegold-platform-build/jenkins-0/jenkins?tailLines=5000&timestamps=true": remote error: tls: internal error 

All logs for all deployments are failing not just the Jenkins stateful set.

When I look at the logs using 'journalctl -u kubelet -n 100 --no-pager' I can see the following error consistently.

TLS handshake error from 127.0.0.1:56232: no serving certificate available for the kubelet

Also, Jenkins is failing to run with error:

io.fabric8.kubernetes.client.http.WebSocketHandshakeException

Any advice on any steps I have missed in renewing the certificates that would fix the handshake issue would be greatly appreciated.

Answer 1

Found the issue here

The steps I took to fix it.

1. Run 'kubectl get csr --sort-by=.metadata.creationTimestamp'.
2. Run 'kubectl certificate approve csr-xyz' on the latest one for each node.

Then in Jenkins, I updated the .kube/config file in the Jenkins secrets referenced by the deploy step of the Jenkins file.