Keywords:
k8s kubernetes troubleshooting, debugging, debug, troubleshoot, useful, command, commands, kubectl, problem, problems
Showing posts with label Kubectl Commands. Show all posts
Showing posts with label Kubectl Commands. Show all posts
KUBERNETES TROUBLESHOOTING: COMMANDS: Pod, Services and PVC Issues
Keywords:
KUBERNETES: Kubectl Command To List All Pods And Nodes On Which They are Running
Below Kubectl command can be used to list all pods and nodes on which they are running.
kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name --all-namespaces
Below is sample output.
NODE NAME
worker2 grafana-78ccdc896d-rsntv
worker2 istio-egressgateway-688d8db5f6-4k57n
worker3 istio-egressgateway-688d8db5f6-lmk42
worker2 istio-ingressgateway-97bd4fb86-2l4xx
worker1 istio-ingressgateway-97bd4fb86-ljjfr
worker3 istiod-6945f949c8-m9tsx
worker1 istiod-6945f949c8-x9mnm
worker3 prometheus-server-85577f7c59-nxlv9
Key words:
commands run running placed pod node how to docs identify identifying
KUBERNETES: Kubectl Command to Decode Grafana Admin User Password
Below Kubectl Command can be used to Decode Grafana Admin User Password.
kubectl get secret --namespace $NAMESPACE grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
KUBERNETES: Grafana Configuration .ini File Which Has Admin User Password Credentials
/etc/grafana/grafana.ini is the file which has grafana Admin User Password Credentials
You will see below block if user credentials are specific in Grafana /etc/grafana/grafana.ini file.
[security]
admin_user = admin
admin_password = admin
KUBERNETES How to Start / Run Temporary Pod On Specific Node In Kubernetes
You can start a pod temporarily on specific node for testing by using kubectl command as follows:
kubectl run <name> --image=<image> --overrides='{"apiVersion": "v1", "spec": {"nodeSelector": {"kubernetes.io/hostname": "<hostname>"}}}' --restart=NeverWith above command a temporary pod with specified name and image will be started on the node specified. The pod will not start once you delete the pod since restart is specified as "Never".
For e.g. to start nginx pod using nginx image in Oracle container registry on worker1 node, your command will look as follows:
kubectl run nginx --image=container-registry.oracle.com/olcne/nginx:1.17.7 --overrides='{"apiVersion": "v1", "spec": {"nodeSelector": {"kubernetes.io/hostname": "worker1"}}}' --restart=NeverKeywords:
temporarily starting running pod pods test testing without deployment deployments yaml file files pull pulliing particular desired choice
KUBERNETES: How to Start / Run Temporary Pod For Testing In Kubernetes
You can start a pod temporarily for testing by using kubectl command as follows:
kubectl run <name> --image=<image> --restart=Never
With above command a temporay pod with specified name and image will be deployed. The pod will not start once you delete the pod since restart is specified as "Never"
For e.g. to start nginx pod using nginx image in Oracle container registry, your command will look as follows:
kubectl run nginx --image=container-registry.oracle.com/olcne/nginx:1.17.7 --restart=NeverKeywords:
temporarily starting running pod pods test testing without deployment deployments yaml file files pull pulliing
KUBERNETES: KUBECTL Command To List Pods And Their Associated Container Listening Ports
Below command can be used.
kubectl get pods -A -o custom-columns="POD:.metadata.name,PORTS:.spec.containers[*].ports[*].containerPort"
Below is sample output of above command.
POD PORTS
nginx-6f4cfc8479-22zlh 80
nginx-6f4cfc8479-2trg9 80
nginx-6f4cfc8479-7sfpj 80
nginx-6f4cfc8479-mkfng 80
nginx-6f4cfc8479-nffq6 80
nginx-6f4cfc8479-vcmx8 80
coredns-5859f68d4-7p95k 53,53,9153
coredns-5859f68d4-f7hnb 53,53,9153
KUBERNETES: KUBECTL Command To Check Health Status and Liveliness Probe (livez check) Of Kubernetes Components
Below kubectl command can be used on the Kubernetes Control plane node.
kubectl get --raw='/readyz?verbose'
Alternatively below curl command can also be used instead.
curl -k https://localhost:6443/livez?verbose
Below is sample output of above command.
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/start-service-ip-repair-controllers ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-system-namespaces-controller ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok
[+]poststarthook/start-legacy-token-tracking-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
[+]poststarthook/apiservice-discovery-controller ok
livez check passed
KUBERNETES: KUBECTL Command To Check UnHealthy Status Pods
Below command can be used. Below command will list all the pods in all the namespaces which are in unhealthy status.
kubectl get pods -o wide --all-namespaces | grep -vE 'Running|Completed'
Above command will list all the pods which are in other status than the running or completed status
KUBERNETES: KUBECTL Command To Check Kubernetes Components Status
Below command can be used to check the Kubernetes components status.
kubectl get cs
Below is sample output.
#kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0
KUBERNETES: KUBECTL Command To List The Pods and Their CPU & Memory Resource Limits
Below command can be used. Below command will list all the pods in all the namespaces.
kubectl get pods -A -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[].resources.limits.cpu}{"\t"}{.spec.containers[].resources.limits.memory}{"\n"}{end}'
For listing the pods and their limits in specific namespace below command can be used.
kubectl get pods -n <namespace> -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[].resources.limits.cpu}{"\t"}{.spec.containers[].resources.limits.memory}{"\n"}{end}'
Below is sample output of above command. In below output memory is listed as 50Mi which is 50MB and CPU is 100m which is means 1/10th of CPU which is 0.1 CPU.
kube-flannel-ds-4w6lq 100m 50Mi
kube-flannel-ds-69xn6 100m 50Mi
kube-flannel-ds-d7drw 100m 50Mi
kube-flannel-ds-f87dw 100m 50Mi
kube-flannel-ds-rwcjl 100m 50Mi
kube-flannel-ds-xlnh7 100m 50Mi
KUBERNETES: KUBECTL Command To List All Pod Disruption Budgets (PDB) Set In Cluster
Below command can be used.
kubectl get pdb -A
KUBERNETES: KUBECTL Command To Force Delete A Pod
Below command can be used.
kubectl delete pod -n <name-space> <pod-name> --force --grace-period=0
KUBERNETES: Kubectl Config View To Display KubeConfig Settings
Below kubectl config view command can be used to view KubeConfig Settings Without Certificate Chain.
$ kubectl config view --minify.
Below is sample output.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://XX.XX.XX.XX:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: DATA+OMITTED
client-key-data: DATA+OMITTED
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://XX.XX.XX.XX:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: DATA+OMITTED
client-key-data: DATA+OMITTED
Below kubectl config view command can be used to view KubeConfig Settings With Certificate Chain. It will be long output.
$ kubectl config view --raw
KUBERNETES: Kubectl Command To View All Pods And List Containers & Images Part Of the All Pods (How To)
Below command can be used on Control node. In below command, we are listing pods part of all namespaces, you can change the command and list pods/container/images for the namespace you want using -n flag.
kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{"pod: "}{.metadata.name}{"\n"}{range .status.containerStatuses[*]}{"\tname: "}{.containerID}{"\n\timage: "}{.image}{"\n"}{end}'
Below is sameple output.
pod: details-v1-cf74bb974-8jfx5
name: cri-o://5630266da8cc049448888d722ea52859cb5fe0bbf9053222db9440ab889173c6
image: docker.io/istio/examples-bookinfo-details-v1:1.19.1
pod: productpage-v1-87d54dd59-2xqm2
name: cri-o://b8a374635c35e6725f9dceac159acf8ea65aba481fcb6d9fb9ef5c3f3182a635
image: docker.io/istio/examples-bookinfo-productpage-v1:1.19.1
pod: ratings-v1-7c4bbf97db-hcxgt
name: cri-o://5d32cd9f25e0f6189c903ff840239bc7b5459427cf861c0c40df3ddfd9041fb4
image: docker.io/istio/examples-bookinfo-ratings-v1:1.19.1
KUBERNETES: Kubectl Command To Tail CoreDNS Pod Service Logs (Kube-Dns)
Below command can be used.
kubectl logs --follow -n kube-system --selector 'k8s-app=kube-dns'
KUBERNETES: How To Setup Kubectl Environment Variables On Control Node To Run Kubectl Commands?
Login to Kubernetes control node as user using which you want to run kubectl commands. Execute below commands to setup kubectl command
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config
echo 'export KUBECONFIG=$HOME/.kube/config' >> $HOME/.bashrc
Validate that the kubectl commands are working. For this you can run below simple command to list the pods.
kubectl get pods -owide
KUBERNETES: How To Quickly Test App Pod Without A Service & Not Exposed Externally Whether It is Accessible And Working?
KUBERNETES: How To Quickly Test App Pod Without A Service & Not Exposed Externally Whether It is Accessible And Working?
The quickest and easiest way to verify whether the app pod that is not exposed externally and without a service, is to do following
- Identify the port on which app is listening/running in the pod.
- One of the control nodes, use kubectl port-forward command to forward requests coming into IP address/Port on the control node to app pod and port on which app is listening on the pod.
- Access the application via curl, wget or from browser using the IP address and port of the control node on which kubectl port-forward command is being executed.
Following is the syntax for kubectl port-forward command that has to be executed on the control node.
kubectl port-forward --address <local machine address> <target pod name> -n <namespace> <local machine port>:<target pod app listening port>
Once above kubectl port-forward command is executed, command will be running and control node will be listening on the port waiting for connects. For e.g.On one of the Kubernetes control nodes, You can do port forwarding to forward requests on port 8080 on that kubernetes control node to port 80 on one of the nginx pods as shown below.
kubectl port-forward --address 10.10.10.10 nginx-deployment-6595874d85-n6rww 8080:80
You can then try to access the application using the IP of the Kubernetes nodes and port in this case using http://10.10.10.10:8080 for accessing the nginx app. Other way to test if BUI access is not available is to do wget test as follows in this example.
wget -p 10.10.10.10:8080
You will see output as follows which indicates nginx index.html file is accessible on port 8080 (forwarded port)
wget -p 10.10.10.10:8080
--2024-03-21 18:07:29-- http://127.0.0.1:8080/
Connecting to 10.10.10.10:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: ‘10.10.10.10:8080/index.html’
100%[=========================================>] 612 --.-K/s in 0s
2024-03-21 18:07:29 (1.23 MB/s) - ‘10.10.10.10:8080/index.html’ saved [612/612]
FINISHED --2024-03-21 18:07:29--
KUBERNETES: How To Untaint / Taint Control Nodes (Master Nodes) So That Application Pods (App / User Pods) Are Started On Them (Kubectl Command)
KUBERNETES: How To Untaint / Taint Control Nodes (Master Nodes) So That Application Pods (App / User Pods) Are Started On Them (Kubectl Command)
Untaint the Control Node
Below are commands to untaint the Control Nodes so that application / user pods are started on them.
kubectl taint nodes control1 node-role.kubernetes.io/control-plane-
kubectl taint nodes control1 node-role.kubernetes.io/control-plane-
When running above command, in the console output you would see "<node> untainted" message.
Taint back Control Node
To taint back the control node and revert back changes you can run below command.
kubectl taint nodes control1 node-role.kubernetes.io/control-plane:NoSchedule
kubectl taint nodes control1 node-role.kubernetes.io/control-plane:NoSchedule
When running above command, in the console output you would see "<node> tainted" message.
Subscribe to:
Posts (Atom)