For troubleshooting network communication issues between two Linux hosts. We have to check and verify following on how the tcp/ip packets are transmitted.
- If the source is sending the packets
- destination is receiving the packet and sending the reply back.
- If source is receving the reply back
- If there are any dropped packets at source during communication.
- If therea are any dropped packets at destination during communication.
For checking above things we have to do ping tests between Linux hosts which have communication issue and capture following data.
- tcpdumps capture on ICMP ping protocol- which shows how the tcp/ip packets are transmitted.
- "ip -s -s -s link" command outputs before and after capturing tcpdumps to see if there is any increase in packet loss.
Network Troubleshooting Data Gathering Steps
Below are steps to capture tcpdumps and ip link command outputs for troubleshooting network issues.
1. Identify two Linux hosts (VM's or Physical hosts) - source and destination VM's which have communication issues. Make a Note of the Linux Hostnames.
2. Identify the network bond/interface name for both source and destination Linux Hosts over which they have communication issues. For e.g. bond1 or eth1 etc.
3. Run below command on the source Linux Host which has issues and capture the ip link command output before the tcpdump test.
ip -s -s -s link>>/tmp/iplink_before_`hostname -s`.out
Above command captures the iplink command output inside /tmp/iplink_before_<hostname>.out file.
4. Run below command on the Destination Linux Host which has issues and capture the ip link command output before the tcpdump test.
ip -s -s -s link>>/tmp/iplink_before_`hostname -s`.out
Above command captures the iplink command output inside /tmp/iplink_before_<hostname>.out file.
5. Now open 2 SSH sessions for each source and destination Linux Hosts.
6. Now in one of the SSH sessions for the source Linux Host run below tcpdump command.
tcpdump -n -i bondX icmp -w /tmp/tcpdumps__$(date +%F'-'%T)_`hostname`.pcap
In above command bondX will be the bond interface of source Linux host. This command saves the tcpdumps inside /tmp with filename as tcpdump_<date/timestamps>_<hostname>. Leave the tcpdump command running and do not do Ctrl C
For e.g. if source Linux Host network is bond1, above command looks like below.
tcpdump -n -i bond1 icmp -w /tmp/tcpdumps__$(date +%F'-'%T)_`hostname`.pcap
7. Now in one of the SSH sessions for the destination Linux Host run below tcpdump command.
tcpdump -n -i bondX icmp -w /tmp/tcpdumps__$(date +%F'-'%T)_`hostname`.pcap
In above command bondX will be the bond interface of destination VM. This command saves the tcpdumps inside /tmp with filename as tcpdump_<date/timestamps>_<hostname>. Leave the tcpdump command running and do not do Ctrl C
For e.g. if destination Linux Host is bond1, above command looks like below.
tcpdump -n -i bond1 icmp -w /tmp/tcpdumps__$(date +%F'-'%T)_`hostname`.pcap
8. Now send 5 pings from another SSH session of source Linux Host to destination Linux Host IP as follows
ping -c 5 <destination Linux Host IP>
9. Now send 5 pings from another SSH session of destination Linux Host to source Linux Host IP as follows
ping -c 5 <source Linux Host IP>
10. Now go back to the SSH sessions for source and destination Linux Host's where tcpdump is running in above steps 6 & 7 and then do CTRL C to stop tcpdumps.
11. Wait for about 2 minutes at this time.
12.Run below command on the source Linux Host which has issues and capture the ip link command output after the tcpdump test.
ip -s -s -s link>>/tmp/iplink_after_`hostname -s`.out
Above command captures the iplink command output inside /tmp/iplink_after_<hostname>.out file.
13. Run below command on the Destination Linux Host which has issues and capture the ip link command output after the tcpdump test.
ip -s -s -s link>>/tmp/iplink_after_`hostname -s`.out
Above command captures the iplink command output inside /tmp/iplink_after_<hostname>.out file.
14. Capture below data from above steps from both source and destination Linux hosts.
/tmp/tcpdump_*
/tmp/iplink_*
From above data collection steps we should have below data
From source Linux Host
- tcpdumps which were captured
- iplink command output before tcpdump capture
- iplink command output after tcpdump capture
From Destination Linux Host
- tcpdumps which were captured
- iplink command output before tcpdump capture
- iplink command output after tcpdump capture
Analyzing Data Gathered For Network Issues
TCPDUMPS review
tcpdumps can either be reviewed using wireshark on local machine or they can reviewed on the Linux host itself where they were captured using below tcpdump command.
tcpdump -r <tcpdump filename>
Check the tcpdumps from the source Linux host to see if the packets are going out to destination Linux Host and reply is coming back.
For example in this case lets assume source Linux Host IP is 10.10.10.11 and Destination Linux Host IP is 10.10.10.22, when we look at the tcpdumps on source Linux Host (with .11 IP) we should be seeing requests going out to .22 IP and reply coming back from .22 IP as follows
13:11:13.017013 IP 10.10.10.11 > 10.10.10.22: ICMP echo request, id 6384, seq 28126, length 64
13:11:13.017049 IP 10.10.10.22 > 10.10.10.11: ICMP echo reply, id 6384, seq 28126, length 64
13:11:14.016897 IP 10.10.10.11 > 10.10.10.22: ICMP echo request, id 6384, seq 28127, length 64
13:11:14.016918 IP 10.10.10.22 > 10.10.10.11: ICMP echo reply, id 6384, seq 28127, length 64
Same way when we check the tcpdumps on destination Linux host (with .22 IP), we should see requests coming in from source .11 IP and reply sent back to .11 IP. For example below is the snippet.
13:11:57.195307 IP 10.10.10.11 > 10.10.10.22: ICMP echo request, id 28322, seq 1, length 64
13:11:57.195343 IP 10.10.10.22 > 10.10.10.11: ICMP echo reply, id 28322, seq 1, length 64
13:11:58.185332 IP 10.10.10.11 > 10.10.10.22: ICMP echo request, id 28322, seq 2, length 64
13:11:58.185353 IP 10.10.10.22 > 10.10.10.11: ICMP echo reply, id 28322, seq 2, length 64
"ip -s -s -s link" Output Review
For both source and destination Linux hosts when we review "ip -s -s -s link" in a ideal working scenario there should not be increase in dropped packets on the network interface (where troubleshooting is done) before or after the tcpdumps capture, there should NOT be increase in RX or TX packets drop.
For example below is sample output of "ip -s -s -s link" command. We should not see increase in dropped packet numbers for RX (received) and TX (transmitted).
bond1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 1000
link/ether 00:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
28070237 308525 0 0 0 4486
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
179835757 350942 0 0 0 0
TX errors: aborted fifo window heartbeat transns
0 0 0 0 2
Above data capture and review will give enough information where the packets are getting dropped when troubleshooting network communication related information.
Products to which Article Applies
All Linux environments
