KUBERNETES: How to Change ETCD Leader From One Control Node To Other Using ETCDCTL Commands?

 1. Identify which Kubernetes Control Node has the leader. For his run below etcdctl command. In below command for —endpoints give the https client address endpoint of each control node.

sudo ETCDCTL_API=3 etcdctl --endpoints=https://10.0.0.182:2379,https://10.0.0.236:2379,https://10.0.0.4:2379,https://10.0.0.155:2379,https://10.0.0.96:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table

Below is the sample output.

+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |

+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

| https://10.0.0.182:2379 |  d5907367f2bc466 |   3.5.3 |  9.7 MB |     false |      false |         9 |     110761 |             110761 |        |

| https://10.0.0.236:2379 | e1c82ddedc10c490 |   3.5.3 |  9.7 MB |     false |      false |         9 |     110761 |             110761 |        |

|   https://10.0.0.4:2379 | 86d10b822621f9cf |   3.5.3 |  9.7 MB |      true |      false |         9 |     110761 |             110761 |        |

| https://10.0.0.155:2379 | 9869473a238b9297 |   3.5.3 |  9.6 MB |     false |      false |         9 |     110761 |             110761 |        |

|  https://10.0.0.96:2379 | c4438cde3e1e5536 |   3.5.3 |  9.5 MB |     false |      false |         9 |     110761 |             110761 |        |

+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

2. Run below etcdctl command to change the leader to new control node.

sudo ETCDCTL_API=3 etcdctl --endpoints=<end-points> --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key move-leader <member ID>

For example, below is the example command.

In below command for —endpoints give the https client address endpoint of each control node, Replace the leader ID in this case d5907367f2bc466  with the leader ID of the Control Node you wish to change the Leader to.

sudo ETCDCTL_API=3 etcdctl --endpoints=https://10.0.0.182:2379,https://10.0.0.236:2379,https://10.0.0.4:2379,https://10.0.0.155:2379,https://10.0.0.96:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key move-leader d5907367f2bc466

Below is the sample console output you would see.

Leadership transferred from 86d10b822621f9cf to d5907367f2bc466

3. Run above command in step (1) to check if the leader moved to new control node.

Below is example output. As you can see leader is changed now to member ID d5907367f2bc466

+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |

+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

| https://10.0.0.182:2379 |  d5907367f2bc466 |   3.5.3 |  9.7 MB |      true |      false |        10 |     111728 |             111728 |        |

| https://10.0.0.236:2379 | e1c82ddedc10c490 |   3.5.3 |  9.7 MB |     false |      false |        10 |     111728 |             111728 |        |

|   https://10.0.0.4:2379 | 86d10b822621f9cf |   3.5.3 |  9.7 MB |     false |      false |        10 |     111728 |             111728 |        |

| https://10.0.0.155:2379 | 9869473a238b9297 |   3.5.3 |  9.6 MB |     false |      false |        10 |     111728 |             111728 |        |

|  https://10.0.0.96:2379 | c4438cde3e1e5536 |   3.5.3 |  9.5 MB |     false |      false |        10 |     111728 |             111728 |        |

+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Read more "KUBERNETES: How to Change ETCD Leader From One Control Node To Other Using ETCDCTL Commands?"

KUBERNETES (K8S): ETCDCTL Command To Delete Pod Registry Entry From ETCD Database

NOTE: Deleting the Pod registry entry from etcd database will delete the pod from Kubernetes cluster. This action should never be performed and may cause issues with Kubernetes cluster. This action needs to be taken with lot of precaution, thorough analysis and guidance.

Below is the command.

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key del <pod registry entry>

For e.g. command can look as follows:

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key del /registry/pods/default/nginx-static-pod-control1

To List the Pod registry entries in etcd you can run below command.

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/pods --prefix --keys-only

Read more "KUBERNETES (K8S): ETCDCTL Command To Delete Pod Registry Entry From ETCD Database"

KUBERNETES: ETCDCTL Command To List All the Keys Of all Pods In Kubernetes Cluster From ETCD Database

KUBERNETES: ETCDCTL Command To List All the Keys Of all Pods In Kubernetes Cluster From ETCD Database

Below command can be used.

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/pods --prefix --keys-only

Below is snippet of above command.

$ sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/pods --prefix --keys-only | more

/registry/pods/default/nginx-deployment-6595874d85-hhhrs

/registry/pods/default/nginx-deployment-6595874d85-n6rww

/registry/pods/externalip-validation-system/externalip-validation-webhook-64f866b

87c-sx7x6

/registry/pods/kube-system/coredns-679794b79f-qcspd

/registry/pods/kube-system/coredns-679794b79f-zt8gz

/registry/pods/kube-system/etcd-cne14-control1

/registry/pods/kube-system/kube-apiserver-cne14-control1

Read more "KUBERNETES: ETCDCTL Command To List All the Keys Of all Pods In Kubernetes Cluster From ETCD Database"

KUBERNETES: ETCDCTL Command To List All the Keys Of Kubernetes resources in ETCD Database

KUBERNETES: ETCDCTL Command To List All the Keys Of Kubernetes resources in ETCD Database

Below command can be used.

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only

Below is snippet of above command.

$ sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only | more

/registry/apiregistration.k8s.io/apiservices/v1.

/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.apps

/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io

Read more "KUBERNETES: ETCDCTL Command To List All the Keys Of Kubernetes resources in ETCD Database"

KUBERNETES (K8S): ETCDCTL Command To Check Health of ETCD Member Endpoints

Below is command to use. Replace 10.XX.XX.219 and 10.XX.XX.86 endpoint IPs with the client address IPs of the ETCD member's

ETCDCTL_API=3 etcdctl --endpoints=https://10.XX.XX.219:2379,https://10.XX.XX.86:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health -w table

  

You can gather client address IPs for etcd member nodes using etcdctl member list command. Details can be found in https://www.tarbots.com/2024/01/tarun-boyella-kubernetes-k8s-etcdctl-command-to-list-the-members-of-etcd-and-their-details.html

Read more "KUBERNETES (K8S): ETCDCTL Command To Check Health of ETCD Member Endpoints"

KUBERNETES (K8S): ETCDCTL Command To Check the Performance and Health of etcd

Below command can be used.

sudo ETCDCTL_API=3 etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key check perf

Above command will run for a minute and then report the response time and health status of etcd whether it is pass. Below is sample output.

sudo ETCDCTL_API=3 etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key check perf

 59 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom  !  98.33%PASS: Throughput is 150 writes/s

PASS: Slowest request took 0.015323s

PASS: Stddev is 0.000739s

PASS

Read more "KUBERNETES (K8S): ETCDCTL Command To Check the Performance and Health of etcd"

KUBERNETES (K8S): ETCDCTL Command To List the Members of ETCD And Their Details

Below etcdctl command can be used to list the members of etcd and their detials.

sudo ETCDCTL_API=3 etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key member list -w table

Below is sample output of the command.

+------------------+---------+----------------+-------------------------+-------------------------+------------+

|        ID        | STATUS  |      NAME      |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |

+------------------+---------+----------------+-------------------------+-------------------------+------------+

| 1581f4841997c696 | started | cne14-control1 | https://10.XX.XX.219:2380 | https://10.XX.XX.219:2379 |      false |

| d79b8841ab930268 | started | cne14-control2 |  https://10.XX.XX.86:2380 |  https://10.XX.XX.86:2379 |      false |

+------------------+---------+----------------+-------------------------+-------------------------+------------+

Read more "KUBERNETES (K8S): ETCDCTL Command To List the Members of ETCD And Their Details"

KUBERNETES (K8S): ETCDCTL Command To Update Member of ETCD

Below etcdctl command can be used.

ETCDCTL_API=3 etcdctl \

  --endpoints=https://127.0.0.1:2379 \

  --cacert=/etc/kubernetes/pki/etcd/ca.crt \

  --cert=/etc/kubernetes/pki/etcd/server.crt \

  --key=/etc/kubernetes/pki/etcd/server.key \

  member update 1581f4841997c696 --peer-urls=https://10.XX.XX.219:2380

In above command replace 1581f4841997c696 ID with member ID of the node you want to replace and --peer-urls with the peer URL of the member ID that you are updating.

For getting the member ID and peer URL you can use below ETCDCTL Command.

sudo ETCDCTL_API=3 etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key member list -w table

Below is sample output of the command.

+------------------+---------+----------------+-------------------------+-------------------------+------------+

|        ID        | STATUS  |      NAME      |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |

+------------------+---------+----------------+-------------------------+-------------------------+------------+

| 1581f4841997c696 | started | cne14-control1 | https://10.XX.XX.219:2380 | https://10.XX.XX.219:2379 |      false |

| d79b8841ab930268 | started | cne14-control2 |  https://10.XX.XX.86:2380 |  https://10.XX.XX.86:2379 |      false |

+------------------+---------+----------------+-------------------------+-------------------------+------------+

Read more "KUBERNETES (K8S): ETCDCTL Command To Update Member of ETCD"