X310 gNB consistently crashing after 2-3 minutes

[sebastiangrosvik]

Issue Description

I have an issue with my X310-based SRS gNB consistently crashing after 2-3 minutes of operation.
The crash occurs both with 40 MHz and 80 MHz bandwidth.
The 40 MHz config runs fine with the B210.

Setup Details

Host specs:
Ubuntu 22.04.3 LTS
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
4x 8GiB DIMM DDR4 Synchronous 2133 MHz (0.5 ns)
UHD version: UHD_4.7.0.HEAD-0-ga5ed1872

Device specific configuration:
ru_sdr:
device_driver: uhd
device_args: type=x300,addr=192.168.40.2,product=X310,master_clock_rate=184.32e6,dboard_clck_rate=92.16e6,send_frame_size=8000,recv_frame_size=8000
clock: internal
sync: internal
srate: 46.08
freq_offset: 0
clock_ppm: 0
lo_offset: 0
tx_gain: 31.5
rx_gain: 20
time_alignment_calibration: auto

Actual Behaviour

The gNB runs fine during and after startup. Our UEs connect.
Then after some time the radio link disconnects all of the sudden, UEs are not able to reconnect.
The console trace stops printing.

Two example console traces:
|--------------------DL---------------------|-------------------------UL------------------------------
pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4602 | 15 1.0 27 256k 248 0 0% 0 | -3.7 -4.8 0 3.9M 600 0 0% 300k 56n 21
1 4602 | 15 1.0 27 223k 216 0 0% 0 | -3.7 -4.9 0 3.9M 600 0 0% 300k 97n 21
1 4602 | 15 1.0 27 211k 205 0 0% 0 | -3.7 -4.9 0 3.9M 600 0 0% 300k 89n 21
1 4602 | 15 1.0 27 214k 208 0 0% 58 | -3.7 -4.8 0 3.9M 600 0 0% 300k 89n 21
1 4602 | 15 1.0 24 64k 68 33 32% 0 | -31.4 -30.7 0 656k 102 498 83% 300k 284n 21
1 4602 | n/a n/a 0 0 0 0 0% 0 | -35.6 -37.6 0 0 0 600 100% 300k -512n 21
1 4602 | n/a n/a 0 0 0 0 0% 0 | -37.1 -34.6 0 0 0 600 100% 300k 773n 21
1 4602 | n/a n/a 0 0 0 0 0% 0 | -36.9 -34.9 0 0 0 600 100% 300k -569n 21
^CStopping ..

      |--------------------DL---------------------|-------------------------UL------------------------------

pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4602 | 15 1.0 26 268M 1395 5 0% 30.3k | 36.1 ovl 27 243k 50 0 0% 0 -146n 29
1 4602 | 15 1.0 26 265M 1393 7 0% 30.5k | 35.8 ovl 27 243k 50 0 0% 0 -187n 29
1 4602 | 15 1.0 26 267M 1394 6 0% 26.1k | 36.0 ovl 27 243k 50 0 0% 0 -170n 29
1 4602 | 15 1.0 26 268M 1395 5 0% 17.2k | 35.9 ovl 27 243k 50 0 0% 0 -195n 29
1 4602 | 15 1.0 26 267M 1392 8 0% 26.3k | 35.7 ovl 27 243k 50 0 0% 0 -179n 29
1 4602 | 15 1.0 26 267M 1393 7 0% 26.3k | 35.9 ovl 27 243k 50 0 0% 0 -170n 29
1 4602 | 15 1.0 25 225M 1177 215 15% 4.46M | 11.4 -5.1 20 204k 42 16 27% 0 -602n 29
1 4602 | n/a n/a 19 9.9M 74 1318 94% 5.64M | -58.1 -14.1 0 0 0 100 100% 0 553n 29
1 4602 | n/a n/a 19 10M 71 1327 94% 5.59M | -57.1 -14.4 0 0 0 15 100% 0 1u 29
1 4602 | 2 1.0 10 5.1M 64 1317 95% 5.66M | -59.1 -12.9 0 0 0 10 100% 0 -537n 29
1 4602 | n/a n/a 0 504k 78 1309 94% 5.66M | -57.4 -13.5 0 0 0 15 100% 0 919n 29
^CStopping ..

Additional Information

Ran the SRSran performance script. No change.

[pgawlowicz]

could you send your gnb logs?

[sebastiangrosvik]

gnb_x310.log

[pgawlowicz]

what about CPU utilization? could you first test with smaller BW like 20mhz?

[sebastiangrosvik]

Did a rough CPU comparison of the two SDRs at 40 MHz for 100 seconds using top:
X310: CPU usage is stable at 139%, mem 7.3%
B210: CPU usage is stable at 170%, mem 7.3%

[pgawlowicz]

does it work correctly with b210?

[sebastiangrosvik]

Yep, the b210 is stable. Never had this issue with it.

[pgawlowicz]

Could you pull the newest code base and try again?

[sebastiangrosvik]

Looks like the new version has solved the problem for us.
Tested the newest code base with the X310 at 40 MHz bandwidth: Performance is stable, no crashing.
At 80 MHz our channel quality is not stable, but at least we are not experiencing any crashing.

[pgawlowicz]

Probably, you will need to optimize the CPU affinities and number of threads. Please see expert_execution in gnb config: https://docs.srsran.com/projects/project/en/latest/user_manuals/source/config_ref.html

[pgawlowicz]

@sebastiangrosvik did you try tuning the CPU affinities and number of threads?