"rds-ping" & "rds-stress" commands can be used for troubleshooting Infiniband network issues on Linux hosts. By running tests using "rds-ping" & "rds-stress" commands we can narrow down whether the networking problem is at the Linux OS networking layer or whether the issue is at Infiniband Networking level itself. Because "rds-ping" & "rds-stress" commands will test the reachability of the IP addresses by sending packets over RDS sockets and bypassing Linux OS level tcp rules & route tables.
If actual ping to the hosts/IPs is not working and "rds-ping" & "rds-stress" commands works, then the issue may be at Linux OS network level. If ping and "rds-ping" & "rds-stress" commands both are not working then the issue may be at the Infiniband network layer.
Below are detailed steps for doing "rds-ping" & "rds-stress" tests.
"rds-ping" Tests
1. Identify the IP address of target Linux host to which you want to do rds-ping test.
2. Login to source Linux host as root user and identify interface IP address (in same subnet as target linux host) using which rds-ping test has to be done to the target Linux host.
3. On Linux source host run rds-ping command as follows to do rds test to target host.
For e.g. if source IP address is 192.168.2.96 and target IP address is 192.168.2.114, number of ping counts is 3, then your rds-ping command on source Linux host looks as follows:
Below is example snippet of above command.
As we can see from above command we see response coming back to the rds-ping.
"rds-stress" Tests
1. Login to the target List host as root user to which you want to do rds tests from source Linux host. Run below command.
You will see output as follows that the target host is listening for connection on port 4000
2. Login to source Linux host and connect to target Linux host in above step 1 using rds-stress command as follows:
In above command replace IP address with the IP address of target Linux host in above step 1. For e.g. if you ran rds-stress between target host with IP 192.168.2.96 and source host with IP 192.168.2.240 and sent 500000 rdma packets from source, your rds commands and outputs on source and target will look as follows:
source:
target:
As you can see from above "rds-stress" command outputs the communication between both the Linux hosts is happening without any issues.
https://linux.die.net/man/1/rds-stress
If actual ping to the hosts/IPs is not working and "rds-ping" & "rds-stress" commands works, then the issue may be at Linux OS network level. If ping and "rds-ping" & "rds-stress" commands both are not working then the issue may be at the Infiniband network layer.
Below are detailed steps for doing "rds-ping" & "rds-stress" tests.
"rds-ping" Tests
1. Identify the IP address of target Linux host to which you want to do rds-ping test.
2. Login to source Linux host as root user and identify interface IP address (in same subnet as target linux host) using which rds-ping test has to be done to the target Linux host.
3. On Linux source host run rds-ping command as follows to do rds test to target host.
rds-ping -c <count> -I <source IP address> <target Linux host IP address>
For e.g. if source IP address is 192.168.2.96 and target IP address is 192.168.2.114, number of ping counts is 3, then your rds-ping command on source Linux host looks as follows:
# rds-ping -c 3 -I 192.168.2.96 192.168.2.114
Below is example snippet of above command.
# rds-ping -c 3 -I 192.168.2.96 192.168.2.114
1: 142 usec
2: 167 usec
3: 110 usec
1: 142 usec
2: 167 usec
3: 110 usec
As we can see from above command we see response coming back to the rds-ping.
"rds-stress" Tests
1. Login to the target List host as root user to which you want to do rds tests from source Linux host. Run below command.
rds-stress
You will see output as follows that the target host is listening for connection on port 4000
# rds-stress
waiting for incoming connection on 0.0.0.0:4000
waiting for incoming connection on 0.0.0.0:4000
2. Login to source Linux host and connect to target Linux host in above step 1 using rds-stress command as follows:
rds-stress -s <IP address of target VM> -p 4000 -t 1 -D <number of bytes>
In above command replace IP address with the IP address of target Linux host in above step 1. For e.g. if you ran rds-stress between target host with IP 192.168.2.96 and source host with IP 192.168.2.240 and sent 500000 rdma packets from source, your rds commands and outputs on source and target will look as follows:
source:
~]# rds-stress -s 192.168.2.96 -p 4000 -t 1 -D 500000
connecting to 192.168.2.96:4000
negotiated options, tasks will start in 2 seconds
Starting up....
tsks tx/s rx/s tx+rx K/s mbi K/s mbo K/s tx us/c rtt us cpu %
1 3434 3436 7292.05 1677105.22 1677105.22 37.75 286.29 -1.00
1 3436 3428 7285.58 1675617.04 1675617.04 38.20 286.30 -1.00
1 3451 3444 7318.36 1683398.48 1682910.25 37.18 285.21 -1.00
1 3441 3441 7304.68 1680009.46 1680009.46 37.42 286.42 -1.00
connecting to 192.168.2.96:4000
negotiated options, tasks will start in 2 seconds
Starting up....
tsks tx/s rx/s tx+rx K/s mbi K/s mbo K/s tx us/c rtt us cpu %
1 3434 3436 7292.05 1677105.22 1677105.22 37.75 286.29 -1.00
1 3436 3428 7285.58 1675617.04 1675617.04 38.20 286.30 -1.00
1 3451 3444 7318.36 1683398.48 1682910.25 37.18 285.21 -1.00
1 3441 3441 7304.68 1680009.46 1680009.46 37.42 286.42 -1.00
target:
~]# rds-stress
waiting for incoming connection on 0.0.0.0:4000
accepted connection from 192.168.2.240:61630 on 192.168.2.96:4000
negotiated options, tasks will start in 2 seconds
Starting up....
tsks tx/s rx/s tx+rx K/s mbi K/s mbo K/s tx us/c rtt us cpu %
1 3439 3437 7291.51 1677468.07 1676492.52 42.69 285.65 -1.00
1 3432 3440 7287.16 1675979.62 1675979.62 41.86 286.46 -1.00
1 3447 3454 7317.88 1682801.69 1683289.46 41.40 285.27 -1.00
1 3444 3444 7304.10 1679876.75 1679876.75 41.20 285.69 -1.00
---------------------------------------------
1 3436 3440 7300.16 1679031.54 1678909.60 41.83 286.06 -1.00 (average)
waiting for incoming connection on 0.0.0.0:4000
accepted connection from 192.168.2.240:61630 on 192.168.2.96:4000
negotiated options, tasks will start in 2 seconds
Starting up....
tsks tx/s rx/s tx+rx K/s mbi K/s mbo K/s tx us/c rtt us cpu %
1 3439 3437 7291.51 1677468.07 1676492.52 42.69 285.65 -1.00
1 3432 3440 7287.16 1675979.62 1675979.62 41.86 286.46 -1.00
1 3447 3454 7317.88 1682801.69 1683289.46 41.40 285.27 -1.00
1 3444 3444 7304.10 1679876.75 1679876.75 41.20 285.69 -1.00
---------------------------------------------
1 3436 3440 7300.16 1679031.54 1678909.60 41.83 286.06 -1.00 (average)
As you can see from above "rds-stress" command outputs the communication between both the Linux hosts is happening without any issues.
Products to which Article Applies
All Linux hosts using Infiniband.Additional Reference
https://linux.die.net/man/1/rds-pinghttps://linux.die.net/man/1/rds-stress