After run tcpdump.py in Rhel8.2 with Avocado(version: 80.0), it suspends. #1863

Gene-Lo · 2020-08-19T08:38:27Z

After run tcpdump.py in Rhel8.2 with Avocado(version: 80.0), it suspends. Take below results for example:

【Example 1】
※Network Card: Marvell_QUAD E'NET (2X1 + 2X10 10Gb), PCIe Gen 2 X8/SHORT LP CAPABLE (SHINER SFP+ SR COPPER)
※SOL log:

※job-log:
20200816-Network9-tcpdump-job.log
※Configuration:
20200816-Network9-Configuration.zip

【Example 2】
※Network Card: Marvell_2-PORT E'NET (2X10 10Gb), PCIe Gen 2 X8/SHORT LP CAPABLE (SHINER 10GBase-T)
※SOL log:

※job-log:
20200817-Network7-tcpdump-job.log
※Configuration:
20200817-Network7-Configuration.zip

【Example 3】
※Network Card: Mellanox_2-PORT 25/10Gb NIC&ROCE SR/Cu PCIe 3.0 (25/10Gb EVERGLADES EN)
※SOL log:

※job-log:
20200818-Network6-tcdump-job.log
※Configuration:
20200818-Network6-Configuration.zip

In the past, we also use tcpdump.py to test these Network Cards in Rhel7.6 and Rhel8.1, and all results were PASS. Please kindly check if it is script error in the latest version of tcpdump.py. Many thanks!!

vaishnavibhat · 2020-08-26T10:45:20Z

Hi,
From the logs attached, I see that nping_count value is not set . We need to provide around 15-20 count more than than the count value. count is set to 100 by default. Can you please set the nping_value to 115 and try out the test?

2020-08-16 21:31:21,237 parameters       L0143 DEBUG| PARAMS (key=count, path=*, default=500) => 100
2020-08-16 21:31:21,237 parameters       L0143 DEBUG| PARAMS (key=nping_count, path=*, default=) => ''
2020-08-16 21:31:21,237 parameters       L0143 DEBUG| PARAMS (key=peer_ip, path=*, default=) => '192.168.10.2'
2020-08-16 21:31:21,237 parameters       L0143 DEBUG| PARAMS (key=peer_public_ip, path=*, default=) => '192.168.10.2'
2020-08-16 21:31:21,237 parameters       L0143 DEBUG| PARAMS (key=drop_accepted, path=*, default=10) => 10
2020-08-16 21:31:21,238 parameters       L0143 DEBUG| PARAMS (key=host_ip, path=*, default=) => '192.168.10.1'
2020-08-16 21:31:21,238 parameters       L0143 DEBUG| PARAMS (key=option, path=*, default=) => ''
2020-08-16 21:31:21,239 parameters       L0143 DEBUG| PARAMS (key=netmask, path=*, default=) => '255.255.255.0'

From the different screenshots attached, this is my explanation. I would recommend you to retry the tests with nping_count and see if its recreated.
Example 1) I see tcpdump with tcp option has succeeded and a hang is seen when udp option was tested. There might already be some tcp traffic in the machine
and we see this option has succeeded. When it came to udp, there was a hang as there is no udp traffic in the machine.
Example 2) From the log of Example 2, the test was actually running but very slow. When the test was aborted there were 64 packets captured which otherwise would have been zero packets captured.

2020-08-17 23:28:26,401 process          L0437 DEBUG| [stderr] tcpdump: listening on enP1p9s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
2020-08-17 23:28:57,745 process          L0437 DEBUG| [stderr] 64 packets captured```

abdhaleegit · 2020-08-27T08:39:12Z

I also seeing similar issues, this has happen after recent enhancement with tcpdump .. the test never completed after around 80 scenarios of 333 scenarios.. it is hung here

[36611.768878] bnx2x 0010:01:00.0 enP16p1s0f0: NIC Link is Up, 1000 Mbps full duplex, Flow control: none
(079/333) avocado-misc-tests/io/net/tcpdump.py:TcpdumpTest.test;run-mtu-7000-options-IPv4_numeric-0292: PASS (81.79 s)
2020-08-27 04:23:27,047:op-test.common.OpTestUtil:try_sendcontrol:WARNING:OpTestSystem detected something, working on recovery
(080/333) avocado-misc-tests/io/net/tcpdump.py:TcpdumpTest.test;run-mtu-8000-options-IPv4_numeric-9255: ^C^C
Interrupt requested. Waiting 2 seconds for test to finish (ignoring new Ctrl+C until then)
^M~.
2020-08-27 04:23:39,300:op-test.common.OpTestUtil:try_recover:WARNING:OpTestSystem detected something, working on recovery
command-line line 0: Unsupported option "afstokenpassing"^M
This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Activate the web console with: systemctl enable --now cockpit.socket

Gene-Lo · 2020-09-21T02:17:10Z

Dear @vaishnavibhat

After we edit the yaml file, and then run tcpdump.py again in Rhel8.2 with Avocado(version: 82.0), it doesn't suspend. However, some items show "ERROR: Could not get operational link state".

【Test Step】
Step 1. Prepare 2 terminals while each one is equipped with 1 Network-Card. Then connect the 2 Network-Cards with each other.
※Network-Card: Broadcom 5719 QP 1G (1G/100M/10M) Network Interface Card PCIe x4 LP

Step 2. Edit yaml file: /root/tests/tests/avocado-misc-tests/io/net/tcpdump.py.data/tcpdump.yaml

Step 3. Run tcpdump.py via command: avocado run tcpdump.py -m tcpdump.py.data/tcpdump.yaml
↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
JOB ID : fddfeb5f01f284c237bd009e27f3123a6ace79e8
JOB LOG : /root/tests/results/job-2020-09-18T21.34-fddfeb5/job.log tcpdump.py:TcpdumpTest.test;run-mtu-9000-options-protocol_icmp-860d: PASS (75.87 s)
RESULTS : PASS 37 | ERROR 26 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML : /root/tests/results/job-2020-09-18T21.34-fddfeb5/results.html
JOB TIME : 3479.48 s
↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

【Test log】: job log & Manual-Test-log
Test log.zip

【Configuration】
《SUT6》
[Rhel8.2 Kernel]
4.18.0-193.14.3.el8_2.ppc64le

[FW config]
BMC: op940.00.mih-5-0-g86f9791c2
PNOR: OP9-v2.4-4.37-prod

[HW config]
CPU DD2.3 20core *2
Micron (MTA18ASF2G72PZ-2G9E1) 16G *16
Samsung PM985 960GB *1
PSU ACBEL 2000w *2
Slot1: Network2 - Mellanox 2-PORT EDR 100Gb IB CONNECTX-5 GEN4 PCIe x16 CAPI CAPABLE LP ADAPTER
Slot2: Network7 - Marvell 2-PORT E'NET (2X10 10Gb), PCIe Gen 2 X8/SHORT LP CAPABLE (SHINER 10GBase-T)
Slot3: Network5 - Mellanox 2-PORT 10Gb NIC&ROCE ConnectX-4Lx SR/Cu PCIe 3.0 LP CAPABLE ADAPTER
Slot4: Network6 - Mellanox 2-PORT 25/10Gb NIC&ROCE SR/Cu PCIe 3.0 (25/10Gb EVERGLADES EN)
Slot5: Network10 - Broadcom 5719 QP 1G (1G/100M/10M) Network Interface Card PCIe x4 LP
Slot6: Network3 - Mellanox 2-PORT 100Gb ROCE EN CONNECTX-5 GEN4 PCIe x16 LP CAPABLE ADAPTER
Slot7: Network9 - Marvell QUAD E'NET (2X1 + 2X10 10Gb), PCIe Gen 2 X8/SHORT LP CAPABLE (SHINER SFP+ SR COPPER)

《SUT8》
[Rhel8.2 Kernel]
4.18.0-193.19.1.el8_2.ppc64le

[FW config]
BMC: op940.00.mih-5-0-g86f9791c2
PNOR: OP9-v2.4-4.37-prod

[HW config]
CPU DD2.3 12core *2
Micron (MTA18ASF2G72PZ-2G9E1) 16G *16
Samsung PM985 960GB *1
PSU ACBEL 2000w *2
Slot1: Network2 - Mellanox 2-PORT EDR 100Gb IB CONNECTX-5 GEN4 PCIe x16 CAPI CAPABLE LP ADAPTER
Slot2: Network7 - Marvell 2-PORT E'NET (2X10 10Gb), PCIe Gen 2 X8/SHORT LP CAPABLE (SHINER 10GBase-T)
Slot3: Network5 - Mellanox 2-PORT 10Gb NIC&ROCE ConnectX-4Lx SR/Cu PCIe 3.0 LP CAPABLE ADAPTER
Slot4: Network6 - Mellanox 2-PORT 25/10Gb NIC&ROCE SR/Cu PCIe 3.0 (25/10Gb EVERGLADES EN)
Slot5: Network10 - Broadcom 5719 QP 1G (1G/100M/10M) Network Interface Card PCIe x4 LP
Slot6: Network3 - Mellanox 2-PORT 100Gb ROCE EN CONNECTX-5 GEN4 PCIe x16 LP CAPABLE ADAPTER
Slot7: Network9 - Marvell QUAD E'NET (2X1 + 2X10 10Gb), PCIe Gen 2 X8/SHORT LP CAPABLE (SHINER SFP+ SR COPPER)

【Note】
Although there are many types of Network Cards equipped in the terminal, the Network Card we test is "Broadcom 5719 QP 1G (1G/100M/10M) Network Interface Card PCIe x4 LP", and we only add static IP in "Broadcom 5719 QP 1G (1G/100M/10M) Network Interface Card PCIe x4 LP".

vaishnavibhat · 2020-09-29T06:11:06Z

Hi,

This mostly looks like a timing issue during MTU change . When a MTU change occurs, there is a link down - link up happening subsequently. Currently there is a timeout of 30s for this process. Here it looks like its taking a longer time for the same. I have posted this patch to increase the timeout. Addressing the review comments and waiting to get accepted.
avocado-framework/avocado#4252

Gene-Lo · 2020-10-22T08:58:17Z

Tag [email protected]
Sachin, please kindly track issue.

vaishnavibhat · 2020-11-05T05:02:31Z

Using the yaml parameter mtu_timeout to extend wait for cases where we see timing issue with the adapter. Currently there is no specific wait time for drivers and each of them are designed and behaved differently.

Gene-Lo · 2020-11-11T07:29:43Z

Dears,
Here are the results that we change mtu_timeout to different values, and then run tcpdump.py in Rhel8.2 with Avocado(version: 82.0)

【job.log & Manual-Test-log】
job.log
Manual-Test-log for INIT 02.txt

【tcpdump.yaml】

【job.log & Manual-Test-log】
job.log
Manual-Test-log for INIT 03.txt

【job.log & Manual-Test-log】
job.log
job_debug for INIT 02.txt

【job.log】
job.log

【job.log】
job.log

【job.log】
job.log

【tcpdump.yaml】

From above results, mtu_timeout should not be the root cause of the issue. Please kindly check. Many thanks!!

Gene-Lo · 2020-11-11T10:26:43Z

In addition, in the past, when we we used the same Network-Card (Broadcom 5719 QP 1G (1G/100M/10M) Network Interface Card PCIe x4 LP), to run tcpdump.py in Rhel8.1 with Avocado(Version: 75.1), the results were all PASS.

【job.log】
20200302_Network10_tcpdump_job.txt

【tcpdump.yaml】
We only set below parameters, while other parameters were default values:

Please kindly check. Many thanks!!

narasimhan-v assigned vaishnavibhat Aug 24, 2020

vaishnavibhat mentioned this issue Nov 2, 2020

mtu_timeout param #1946

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After run tcpdump.py in Rhel8.2 with Avocado(version: 80.0), it suspends. #1863

After run tcpdump.py in Rhel8.2 with Avocado(version: 80.0), it suspends. #1863

Gene-Lo commented Aug 19, 2020

vaishnavibhat commented Aug 26, 2020

abdhaleegit commented Aug 27, 2020

Gene-Lo commented Sep 21, 2020

vaishnavibhat commented Sep 29, 2020

Gene-Lo commented Oct 22, 2020

vaishnavibhat commented Nov 5, 2020

Gene-Lo commented Nov 11, 2020

Gene-Lo commented Nov 11, 2020

After run tcpdump.py in Rhel8.2 with Avocado(version: 80.0), it suspends. #1863

After run tcpdump.py in Rhel8.2 with Avocado(version: 80.0), it suspends. #1863

Comments

Gene-Lo commented Aug 19, 2020

vaishnavibhat commented Aug 26, 2020

abdhaleegit commented Aug 27, 2020

Gene-Lo commented Sep 21, 2020

vaishnavibhat commented Sep 29, 2020

Gene-Lo commented Oct 22, 2020

vaishnavibhat commented Nov 5, 2020

Gene-Lo commented Nov 11, 2020

Gene-Lo commented Nov 11, 2020