KUBERNETES: How To Enable & Capture Coredumps From Crashing/Failing Pods At Linux Level?

 KUBERNETES: How To Enable & Capture Coredumps From Crashing/Failing Pods At Linux Level?

Follow below steps:

1) Make a note of the default core pattern setting using the below command.

     cat /proc/sys/kernel/core_pattern

Default value will be core.

2) Change the core pattern as follows:

    sysctl -w "kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h"

3) Verify that the core handler changed to systemd core handler by running below command.

    cat /proc/sys/kernel/core_pattern

You should see output as follows:

    |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

4) Restart the kubernetes pods which are having issues. Wait for the pods to fail/crash.

5) Once the pods are in crashloopbackoff or other failed or crashed state, check for pod corefile PID using below command.

    coredumpctl list

Below is sample output of the above command, which shows test under EXE column. In your case, look for pods which is having issues under EXE column and identify the PID for it.

# coredumpctl list

TIME    PID   UID   GID SIG PRESENT EXE

Tue XXXX-XX-XX XX:XX:56 GMT    1358     0     0   3 * /usr/local/bin/test

Once the PID is identified, dump the core to a file using below command. In below command, replace  PID-NUMBER with the identified pod pid.

    coredumpctl dump PID-NUMBER > dump.out

To revert the core_pattern to default value at Linux level, you can run below command.

    sysctl -w "kernel.core_pattern=core"