Kernel crash

[bernhardschmidt]

Together with Gert we have been stress testing OpenVPN+DCO on one of our eduVPN nodes. It has been running for approximately 18 hours serving 10000 connections, before this kernel crash happened

Debian Bullseye, OpenVPN 2.6.0_rc1, ovpn-dco as of Oct 22th.

Jan  4 04:15:38 eduvpn-n09 kernel: [64024.293002] tun257: tun257: deleting peer with id 0, reason 2
Jan  4 04:15:38 eduvpn-n09 openvpn[8213]: read UDPv6 [ECONNREFUSED]: Connection refused (fd=6,code=111)
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.324986] list_del corruption. next->prev should be ffff9646a3de9460, but was dead000000000122
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.325046] ------------[ cut here ]------------
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.325048] kernel BUG at lib/list_debug.c:62!
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.326592] invalid opcode: 0000 [#1] SMP PTI
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.327252] CPU: 1 PID: 209725 Comm: kworker/1:1 Tainted: G        W  OE     5.10.0-20-amd64 #1 Debian 5.10.158-2
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.327926] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.329175] Workqueue: ovpn-event-wq-tun259 ovpn_peer_delete_work [ovpn_dco]
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.329843] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x69
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.330438] Code: c7 c7 a0 08 f2 85 e8 59 12 ff ff 0f 0b 48 89 fe 48 c7 c7 b0 09 f2 85 e8 48 12 ff ff 0f 0b 48 c7 c7 c0 0a f2 85 e8 3a 12 ff ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7
 80 0a f2 85 e8 26 12 ff ff 0f 0b
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.331755] RSP: 0018:ffffa97dc52b3e40 EFLAGS: 00010246
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.332422] RAX: 0000000000000054 RBX: ffff9646a3de9000 RCX: 0000000000000000
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.333107] RDX: 0000000000000000 RSI: ffff9647b9e60900 RDI: ffff9647b9e60900
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.333804] RBP: ffff9646a3de9300 R08: 0000000000000000 R09: ffffa97dc52b3c68
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.334552] R10: ffffa97dc52b3c60 R11: ffff9647bfe3e058 R12: ffff9646a3de9300
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.335259] R13: 0000000000000000 R14: ffffc97dbde50700 R15: 0000000000000000
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.335958] FS:  0000000000000000(0000) GS:ffff9647b9e40000(0000) knlGS:0000000000000000
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.336682] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.337419] CR2: 00005618e71db918 CR3: 00000001053ec006 CR4: 00000000007706e0
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.338202] PKRU: 55555554
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.338929] Call Trace:
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.339765]  __netif_napi_del+0x70/0x110
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.340537]  ovpn_peer_release+0x21/0x80 [ovpn_dco]
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.341304]  ovpn_peer_delete_work+0x15/0x20 [ovpn_dco]
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.342100]  process_one_work+0x1b3/0x350
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.342890]  worker_thread+0x53/0x3e0
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.343692]  ? process_one_work+0x350/0x350
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.344498]  kthread+0x118/0x140
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.345311]  ? __kthread_bind_mask+0x60/0x60
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.346139]  ret_from_fork+0x1f/0x30
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.346928] Modules linked in: binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_ssc fscache dm_mod ovpn_dco(OE) ip6_udp_tunnel udp_tunnel vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock ip6t_REJECT nf_reject_ipv6 nft_chain_nat nf_nat nft_counter ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables intel_rapl_msr intel_rapl_common nfnetlink nfit libnvdimm ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd glue_helper nls_ascii rapl nls_cp437 vfat fat vmwgfx joydev vmw_balloon evdev serio_raw ttm sg efi_pstore pcspkr vmw_vmci drm_kms_helper cec button ac drm sunrpc fuse configfs efivarfs ip_tables x_tables autofs4 xfs libcrc32c crc32c_generic sr_mod cdrom sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ata_generic ahci crc32_pclmul crc32c_intel psmouse libahci ata_piix vmxnet3 libata vmw_pvscsi scsi_mod i2c_piix4
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.351583] ---[ end trace 0e4217e449975ea8 ]---
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.559003] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x69
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.559853] Code: c7 c7 a0 08 f2 85 e8 59 12 ff ff 0f 0b 48 89 fe 48 c7 c7 b0 09 f2 85 e8 48 12 ff ff 0f 0b 48 c7 c7 c0 0a f2 85 e8 3a 12 ff ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 80 0a f2 85 e8 26 12 ff ff 0f 0b
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.561316] RSP: 0018:ffffa97dc52b3e40 EFLAGS: 00010246
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.561976] RAX: 0000000000000054 RBX: ffff9646a3de9000 RCX: 0000000000000000
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.562627] RDX: 0000000000000000 RSI: ffff9647b9e60900 RDI: ffff9647b9e60900
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.563313] RBP: ffff9646a3de9300 R08: 0000000000000000 R09: ffffa97dc52b3c68
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.563982] R10: ffffa97dc52b3c60 R11: ffff9647bfe3e058 R12: ffff9646a3de9300
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.564661] R13: 0000000000000000 R14: ffffc97dbde50700 R15: 0000000000000000
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.565352] FS:  0000000000000000(0000) GS:ffff9647b9e40000(0000) knlGS:0000000000000000
Jan  4 04:15:40 eduvpn-n09 openvpn[8213]: read UDPv6 [ECONNREFUSED]: Connection refused (fd=6,code=111)
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.566002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.566645] CR2: 00005618e71db918 CR3: 00000001053ec006 CR4: 00000000007706e0
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.567326] PKRU: 55555554
Jan  4 04:15:40 eduvpn-n09 openvpn[8328]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8185]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8427]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8397]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8363]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8213]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8155]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8242]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8270]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Jan  4 04:15:40 eduvpn-n09 openvpn[8300]: 067345860c8d009e790a261a41d06ddb/2a02:810d:e80:4b4f:e8a1:e485:517:812a SIGTERM[soft,ovpn-dco: ping expired] received, client-instance exiting
Jan  4 04:15:40 eduvpn-n09 kernel: [64026.572943] tun259: tun259: deleting peer with id 0, reason 2

[xevilstar]

please take a look to #11 and #12 it doesn't even compile

[cron2]

Hi,

On Wed, Jan 04, 2023 at 01:46:36AM -0800, xevilstar wrote: please take a look to #11 and #12 it doesn't even compile

Please do not mix issues. Issue #13 is about a crash, on a Debian-supported kernel. "Not compiling on a bleeding-edge kernel" is a different thing - which needs to be handled as well, of course, but "not in this issue". gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany ***@***.***

[mr-liusg]

May 13 12:41:26 XXXXXX kernel: [2856677.508346] tun5: deleting peer with id 2, reason 2
May 13 12:42:15 XXXXXX kernel: [2856726.661330] tun5: deleting peer with id 4, reason 2
May 13 12:42:15 XXXXXX kernel: [2856726.665323] tun5: deleting peer with id 0, reason 2
May 13 12:43:05 XXXXXX kernel: [2856775.798317] list_del corruption. next->prev should be ffff97c382cb8c60, but was 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856775.804950] ------------[ cut here ]------------
May 13 12:43:05 XXXXXX kernel: [2856775.808452] kernel BUG at lib/list_debug.c:54!
May 13 12:43:05 XXXXXX kernel: [2856775.811914] invalid opcode: 0000 [#1] SMP NOPTI
May 13 12:43:05 XXXXXX kernel: [2856775.815371] CPU: 3 PID: 194680 Comm: kworker/3:2 Tainted: G           OE     5.10.0-14-cloud-amd64 #1 Debian 5.10.113-1
May 13 12:43:05 XXXXXX kernel: [2856775.822689] Hardware name: Amazon EC2 c6i.2xlarge/, BIOS 1.0 10/16/2017
May 13 12:43:05 XXXXXX kernel: [2856775.826949] Workqueue: ovpn-event-wq-tun5 ovpn_peer_delete_work [ovpn_dco_v2]
May 13 12:43:05 XXXXXX kernel: [2856775.832791] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47
May 13 12:43:05 XXXXXX kernel: [2856775.836600] Code: c7 c7 70 d7 8d b4 e8 0c 15 ff ff 0f 0b 48 89 fe 48 c7 c7 00 d8 8d b4 e8 fb 14 ff ff 0f 0b 48 c7 c7 b0 d8 8d b4 e8 ed 14 ff ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 70 d8 8d b4 e8 d9 14 ff ff 0f 0b
May 13 12:43:05 XXXXXX kernel: [2856775.848432] RSP: 0018:ffffb4ca86013e40 EFLAGS: 00010246
May 13 12:43:05 XXXXXX kernel: [2856775.852145] RAX: 0000000000000054 RBX: ffff97c382cb8800 RCX: 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856775.858043] RDX: 0000000000000000 RSI: ffff97c65f0dc380 RDI: ffff97c65f0dc380
May 13 12:43:05 XXXXXX kernel: [2856775.863954] RBP: ffff97c382cb8b00 R08: 0000000000000000 R09: ffffb4ca86013c68
May 13 12:43:05 XXXXXX kernel: [2856775.869842] R10: ffffb4ca86013c60 R11: ffffffffb4eb3650 R12: ffff97c382cb8b00
May 13 12:43:05 XXXXXX kernel: [2856775.875734] R13: 0000000000000000 R14: ffffd4ca7fcda800 R15: 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856775.881641] FS:  0000000000000000(0000) GS:ffff97c65f0c0000(0000) knlGS:0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856775.887897] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 13 12:43:05 XXXXXX kernel: [2856775.891797] CR2: 00007ffda50df328 CR3: 000000010b72e003 CR4: 00000000007706a0
May 13 12:43:05 XXXXXX kernel: [2856775.897761] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856775.903693] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 13 12:43:05 XXXXXX kernel: [2856775.909578] PKRU: 55555554
May 13 12:43:05 XXXXXX kernel: [2856775.912329] Call Trace:
May 13 12:43:05 XXXXXX kernel: [2856775.914962]  __netif_napi_del+0x6c/0x100
May 13 12:43:05 XXXXXX kernel: [2856775.918129]  ovpn_peer_release+0x21/0x80 [ovpn_dco_v2]
May 13 12:43:05 XXXXXX kernel: [2856775.921814]  ovpn_peer_delete_work+0x15/0x20 [ovpn_dco_v2]
May 13 12:43:05 XXXXXX kernel: [2856775.925618]  process_one_work+0x1b6/0x350
May 13 12:43:05 XXXXXX kernel: [2856775.928850]  worker_thread+0x53/0x3e0
May 13 12:43:05 XXXXXX kernel: [2856775.931950]  ? process_one_work+0x350/0x350
May 13 12:43:05 XXXXXX kernel: [2856775.935254]  kthread+0x11b/0x140
May 13 12:43:05 XXXXXX kernel: [2856775.938168]  ? __kthread_bind_mask+0x60/0x60
May 13 12:43:05 XXXXXX kernel: [2856775.941487]  ret_from_fork+0x1f/0x30
May 13 12:43:05 XXXXXX kernel: [2856775.944561] Modules linked in: ovpn_dco_v2(OE) tun sctp xt_MASQUERADE msr ipt_REJECT nf_reject_ipv4 xt_set iptable_security iptable_nat nf_nat iptable_raw iptable_mangle ip_set_hash_net ip_set udp_diag tcp_diag inet_diag binfmt_misc ip6_udp_tunnel udp_tunnel nf_conntrack_netlink nfnetlink xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter intel_rapl_msr intel_rapl_common iosf_mbi crct10dif_pclmul crc32_pclmul nls_ascii nls_cp437 ghash_clmulni_intel vfat fat aesni_intel crypto_simd cryptd glue_helper rapl evdev serio_raw button tcp_bbr uslk(OE) fuse configfs ip_tables x_tables autofs4 ena(OE) crc32c_intel [last unloaded: ovpn_dco]
May 13 12:43:05 XXXXXX kernel: [2856775.981196] ---[ end trace 5710b115da3c68b5 ]---
May 13 12:43:05 XXXXXX kernel: [2856775.984716] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47
May 13 12:43:05 XXXXXX kernel: [2856775.988556] Code: c7 c7 70 d7 8d b4 e8 0c 15 ff ff 0f 0b 48 89 fe 48 c7 c7 00 d8 8d b4 e8 fb 14 ff ff 0f 0b 48 c7 c7 b0 d8 8d b4 e8 ed 14 ff ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 70 d8 8d b4 e8 d9 14 ff ff 0f 0b
May 13 12:43:05 XXXXXX kernel: [2856776.000441] RSP: 0018:ffffb4ca86013e40 EFLAGS: 00010246
May 13 12:43:05 XXXXXX kernel: [2856776.004154] RAX: 0000000000000054 RBX: ffff97c382cb8800 RCX: 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856776.010081] RDX: 0000000000000000 RSI: ffff97c65f0dc380 RDI: ffff97c65f0dc380
May 13 12:43:05 XXXXXX kernel: [2856776.016014] RBP: ffff97c382cb8b00 R08: 0000000000000000 R09: ffffb4ca86013c68
May 13 12:43:05 XXXXXX kernel: [2856776.021946] R10: ffffb4ca86013c60 R11: ffffffffb4eb3650 R12: ffff97c382cb8b00
May 13 12:43:05 XXXXXX kernel: [2856776.027920] R13: 0000000000000000 R14: ffffd4ca7fcda800 R15: 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856776.033818] FS:  0000000000000000(0000) GS:ffff97c65f0c0000(0000) knlGS:0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856776.040116] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 13 12:43:05 XXXXXX kernel: [2856776.044041] CR2: 00007ffda50df328 CR3: 000000010b72e003 CR4: 00000000007706a0
May 13 12:43:05 XXXXXX kernel: [2856776.049967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 13 12:43:05 XXXXXX kernel: [2856776.055926] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 13 12:43:05 XXXXXX kernel: [2856776.061855] PKRU: 55555554
May 13 12:43:54 XXXXXX kernel: [2856824.967287] tun5: deleting peer with id 6, reason 2
May 13 12:44:59 XXXXXX kernel: [2856890.504602] tun5: deleting peer with id 7, reason 2

I have experienced the same problem. However, there is no stable way to reproduce it, and it needs to be served continuously for some time before it appears.

openvpn 2.6.3 and dco 0.2.20230323

[cron2]

@ordex so maybe this is a race between "userland trying to close the tun device" and "a peer exiting this moment"? Or "userland trying to close the tun device, kernel trying to clean up all peers, and packets come by in the wrong moment"?

We might be able to reproduce this by

• start server instance with DCO
• have 20 clients connect, and send pings
• stop server instance
• repeat

[ordex]

@cron2 reference counting + RCU should really take care of all these cases, therefore if the bug is in this area, it is definitely well hidden..

[mr-liusg]

Can I do anything to help pinpoint this problem? This issue is causing DCO to be unavailable in my scenario

[ordex]

@mr-liusg should you have any way to easily reproduce the issue, please let me know the steps. This is what would help the most

[mr-liusg]

some error in dmesg

[һ 10��  9 10:52:48 2023] tun5: deleting peer with id 1, reason 2
[һ 10��  9 10:54:24 2023] list_add corruption. next->prev should be prev (ffff9d22a89c0058), but was 0000000000000000. (next=ffff9d22a806cc60).
[һ 10��  9 10:54:24 2023] ------------[ cut here ]------------
[һ 10��  9 10:54:24 2023] kernel BUG at lib/list_debug.c:23!
[һ 10��  9 10:54:24 2023] invalid opcode: 0000 [#1] SMP NOPTI
[һ 10��  9 10:54:24 2023] CPU: 5 PID: 1558178 Comm: openvpn Tainted: G           OE     5.10.0-14-cloud-amd64 #1 Debian 5.10.113-1
[һ 10��  9 10:54:24 2023] Hardware name: Amazon EC2 c6i.2xlarge/, BIOS 1.0 10/16/2017
[һ 10��  9 10:54:24 2023] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[һ 10��  9 10:54:24 2023] Code: 48 c7 c6 b8 d6 6d 9d 48 89 ef 49 c7 c7 f0 ff ff ff e8 fb d5 00 00 e9 ee 0f c8 ff 4c 89 c1 48 c7 c7 18 d7 6d 9d e8 3a 15 ff ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 c8 d7 6d 9d e8 23 15 ff
[һ 10��  9 10:54:24 2023] RSP: 0018:ffffb9508500f960 EFLAGS: 00010246
[һ 10��  9 10:54:24 2023] RAX: 0000000000000075 RBX: ffff9d2291500b00 RCX: 0000000000000000
[һ 10��  9 10:54:24 2023] RDX: 0000000000000000 RSI: ffff9d259f15c380 RDI: ffff9d259f15c380
[һ 10��  9 10:54:24 2023] RBP: ffff9d22a89c0000 R08: 0000000000000000 R09: ffffb9508500f788
[һ 10��  9 10:54:24 2023] R10: ffffb9508500f780 R11: ffffffff9dcc92c8 R12: ffff9d22a806cc60
[һ 10��  9 10:54:24 2023] R13: ffff9d2291500c60 R14: ffff9d22a89c0058 R15: ffff9d2291500800
[һ 10��  9 10:54:24 2023] FS:  00007f749b5dc7c0(0000) GS:ffff9d259f140000(0000) knlGS:0000000000000000
[һ 10��  9 10:54:24 2023] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[һ 10��  9 10:54:24 2023] CR2: 00005600e87da970 CR3: 00000001f5644005 CR4: 00000000007706a0
[һ 10��  9 10:54:24 2023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[һ 10��  9 10:54:24 2023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[һ 10��  9 10:54:24 2023] PKRU: 55555554
[һ 10��  9 10:54:24 2023] Call Trace:
[һ 10��  9 10:54:24 2023]  netif_napi_add+0xf8/0x1f0
[һ 10��  9 10:54:24 2023]  ovpn_peer_new+0x258/0x370 [ovpn_dco_v2]
[һ 10��  9 10:54:24 2023]  ovpn_netlink_new_peer+0x1ac/0x420 [ovpn_dco_v2]
[һ 10��  9 10:54:24 2023]  ? genl_family_rcv_msg_attrs_parse.constprop.0+0x54/0xd0
[һ 10��  9 10:54:24 2023]  ? genl_family_rcv_msg_doit+0xea/0x150
[һ 10��  9 10:54:24 2023]  genl_family_rcv_msg_doit+0xea/0x150
[һ 10��  9 10:54:24 2023]  genl_rcv_msg+0xde/0x1d0
[һ 10��  9 10:54:24 2023]  ? __check_block_validity.constprop.0+0x41/0x80
[һ 10��  9 10:54:24 2023]  ? ovpn_netlink_notify+0xf0/0xf0 [ovpn_dco_v2]
[һ 10��  9 10:54:24 2023]  ? genl_get_cmd+0xd0/0xd0
[һ 10��  9 10:54:24 2023]  netlink_rcv_skb+0x50/0xf0
[һ 10��  9 10:54:24 2023]  genl_rcv+0x24/0x40
[һ 10��  9 10:54:24 2023]  netlink_unicast+0x201/0x2c0
[һ 10��  9 10:54:24 2023]  netlink_sendmsg+0x250/0x4a0
[һ 10��  9 10:54:24 2023]  sock_sendmsg+0x5e/0x60
[һ 10��  9 10:54:24 2023]  ____sys_sendmsg+0x22e/0x270
[һ 10��  9 10:54:24 2023]  ? import_iovec+0x17/0x20
[һ 10��  9 10:54:24 2023]  ? sendmsg_copy_msghdr+0x7c/0xa0
[һ 10��  9 10:54:24 2023]  ___sys_sendmsg+0x75/0xb0
[һ 10��  9 10:54:24 2023]  ? pipe_write+0x3f0/0x680
[һ 10��  9 10:54:24 2023]  ? ptep_set_access_flags+0x2f/0x40
[һ 10��  9 10:54:24 2023]  ? do_wp_page+0x227/0x460
[һ 10��  9 10:54:24 2023]  ? handle_mm_fault+0x1122/0x1bf0
[һ 10��  9 10:54:24 2023]  __sys_sendmsg+0x59/0xa0
[һ 10��  9 10:54:24 2023]  do_syscall_64+0x33/0x40
[һ 10��  9 10:54:24 2023]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[һ 10��  9 10:54:24 2023] RIP: 0033:0x7f749beb3e13
[һ 10��  9 10:54:24 2023] Code: 8b 15 b9 a1 00 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 89 54 24 1c 48
[һ 10��  9 10:54:24 2023] RSP: 002b:00007ffd5e6dda48 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[һ 10��  9 10:54:24 2023] RAX: ffffffffffffffda RBX: 00005600e87d9b90 RCX: 00007f749beb3e13
[һ 10��  9 10:54:24 2023] RDX: 0000000000000000 RSI: 00007ffd5e6dda80 RDI: 0000000000000004
[һ 10��  9 10:54:24 2023] RBP: 00005600e87d9520 R08: 0000000000000000 R09: 00005600e88520a0
[һ 10��  9 10:54:24 2023] R10: 0000000000000008 R11: 0000000000000246 R12: 00005600e87da970
[һ 10��  9 10:54:24 2023] R13: 00007ffd5e6dda80 R14: 00007ffd5e6ddb9c R15: 0000000000000000
[һ 10��  9 10:54:24 2023] Modules linked in: msr udp_diag xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_set iptable_security iptable_nat nf_nat iptable_raw iptable_mangle ip_set_hash_net ip_set tcp_diag inet_diag binfmt_misc ovpn_dco_v2(OE) ip6_udp_tunnel udp_tunnel nf_conntrack_netlink nfnetlink xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter intel_rapl_msr intel_rapl_common iosf_mbi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel nls_ascii nls_cp437 vfat crypto_simd fat cryptd glue_helper rapl evdev serio_raw button tcp_bbr uslk(OE) fuse configfs ip_tables x_tables autofs4 ena(OE) crc32c_intel
[һ 10��  9 10:54:24 2023] ---[ end trace 82f742cda9fea2d2 ]---
[һ 10��  9 10:54:24 2023] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[һ 10��  9 10:54:24 2023] Code: 48 c7 c6 b8 d6 6d 9d 48 89 ef 49 c7 c7 f0 ff ff ff e8 fb d5 00 00 e9 ee 0f c8 ff 4c 89 c1 48 c7 c7 18 d7 6d 9d e8 3a 15 ff ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 c8 d7 6d 9d e8 23 15 ff
[һ 10��  9 10:54:24 2023] RSP: 0018:ffffb9508500f960 EFLAGS: 00010246
[һ 10��  9 10:54:24 2023] RAX: 0000000000000075 RBX: ffff9d2291500b00 RCX: 0000000000000000
[һ 10��  9 10:54:24 2023] RDX: 0000000000000000 RSI: ffff9d259f15c380 RDI: ffff9d259f15c380
[һ 10��  9 10:54:24 2023] RBP: ffff9d22a89c0000 R08: 0000000000000000 R09: ffffb9508500f788
[һ 10��  9 10:54:24 2023] R10: ffffb9508500f780 R11: ffffffff9dcc92c8 R12: ffff9d22a806cc60
[һ 10��  9 10:54:24 2023] R13: ffff9d2291500c60 R14: ffff9d22a89c0058 R15: ffff9d2291500800
[һ 10��  9 10:54:24 2023] FS:  00007f749b5dc7c0(0000) GS:ffff9d259f140000(0000) knlGS:0000000000000000
[һ 10��  9 10:54:24 2023] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[һ 10��  9 10:54:24 2023] CR2: 00005600e87da970 CR3: 00000001f5644005 CR4: 00000000007706a0
[һ 10��  9 10:54:24 2023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[һ 10��  9 10:54:24 2023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[һ 10��  9 10:54:24 2023] PKRU: 55555554
[һ 10��  9 10:57:43 2023] tun5: deleting peer with id 0, reason 2