Long-standing connections from container are aborted (rootless, pasta) #24572

maleadt · 2024-11-14T18:43:05Z

maleadt
Nov 14, 2024

I'm in the process of porting my stack of homelab containers to podman + quadlets, on Debian 13/trixie/testing (shipping podman 5.2.5 and passt ee7d0b6 from 20241030), and I'm having issues with some containers. My zigbee2mqtt container, which connects to a zigbee adapter on my local network, successfully manages to connect to the adapter but the connections always get closed after a couple of minutes:

[2024-11-14 18:37:51] info:         zh:zstack:znp: Opening TCP socket with 192.168.0.22:6638
[2024-11-14 18:37:51] info:         zh:zstack:znp: Socket connected
[2024-11-14 18:37:51] info:         zh:zstack:znp: Socket ready
[2024-11-14 18:37:51] info:         zh:zstack:znp: Writing CC2530/CC2531 skip bootloader payload
[2024-11-14 18:37:52] info:         zh:zstack:znp: Skip bootloader for CC2652/CC1352
[2024-11-14 18:39:36] error:         zh:zstack:znp: Socket error Error: read ETIMEDOUT
[2024-11-14 18:39:36] info:         zh:zstack:znp: Port closed
[2024-11-14 18:39:36] info:         zh:zstack:znp: closing

[2024-11-14 18:39:38] info:         zh:zstack:znp: Opening TCP socket with 192.168.0.22:6638
[2024-11-14 18:39:38] info:         zh:zstack:znp: Socket connected
[2024-11-14 18:39:38] info:         zh:zstack:znp: Socket ready
[2024-11-14 18:39:38] info:         zh:zstack:znp: Writing CC2530/CC2531 skip bootloader payload
[2024-11-14 18:39:39] info:         zh:zstack:znp: Skip bootloader for CC2652/CC1352
[2024-11-14 18:40:07] error:         zh:zstack:znp: Socket error Error: read ETIMEDOUT
[2024-11-14 18:40:07] info:         zh:zstack:znp: Port closed
[2024-11-14 18:40:07] info:         zh:zstack:znp: closing

[2024-11-14 18:40:09] info:         zh:zstack:znp: Opening TCP socket with 192.168.0.22:6638
[2024-11-14 18:40:09] info:         zh:zstack:znp: Socket connected
[2024-11-14 18:40:09] info:         zh:zstack:znp: Socket ready
[2024-11-14 18:40:09] info:         zh:zstack:znp: Writing CC2530/CC2531 skip bootloader payload
[2024-11-14 18:40:10] info:         zh:zstack:znp: Skip bootloader for CC2652/CC1352
[2024-11-14 18:42:27] error:         zh:zstack:znp: Socket error Error: read ETIMEDOUT
[2024-11-14 18:42:27] info:         zh:zstack:znp: Port closed
[2024-11-14 18:42:27] info:         zh:zstack:znp: closing

...

This did not happen when I was still using Docker, and also not when using podman compose. This is the quadlet I'm using, which is very trivial:

[Unit]
Description=Zigbee2MQTT Bridge
After=local-fs.target mosquitto.service
Requires=mosquitto.service

[Container]
ContainerName=zigbee2mqtt
Image=docker.io/koenkk/zigbee2mqtt
Network=web.network
Volume=/srv/zigbee2mqtt:/app/data:Z
PublishPort=8080
Environment=TZ=Europe/Brussels

Label="traefik.enable=true"

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

Traefik is configured using socket activation, but that shouldn't matter here. The adapter zigbee2mqtt is connecting to sits on my actual network (192.168.0.*). I also do not have anything noteworthy in containers.conf.

The issue is particularly frustrating because zigbee2mqtt doesn't handle the disconnect very well. Other users have also reported this, e.g. Koenkk/zigbee2mqtt#22590 (comment) which also points to pasta. I personally cannot switch back to slirp4netns (because I need pasta for Traefik to see the actual source IP).

I'd be happy to test out the newly released 5.3, but it's not easy to do so on Debian.

maleadt · 2024-11-15T09:32:03Z

maleadt
Nov 15, 2024
Author

Also happens on 5.3 (which was expected, as this is probably passt-related).

0 replies

Luap99 · 2024-11-15T16:27:31Z

Luap99
Nov 15, 2024
Maintainer

From your config it seems like you are using a custom network (Network=web.network) so the network flow is a bit complicated in that case (#22943 (comment))

If it connects to an outgoing server it should not be handled via our rootless port forwarder so that means it goes through the NAt in the rootless-netns into pasta and then out to the LAN. So it is possible something is wrong with pasta.

In order to debug any pasta issues in that case you can use the following options in containers.conf:

[network]
pasta_options = ["--pcap", "/tmp/pasta.pcap","--trace","--log-file","/tmp/pasta.log"]

They will be added to the pasta cli. In order for them to take effect you must stop all containers that use a custom network and then start them again. Then you should see pasta running with these arguments.

Then when this issue happens again copy both pasta files and upload them here so maybe the pasta maintainers then can have a look at them. Note it is best to do this while you have no other containers running that generate a lot of data to not add to much irrelevant data into the logs.

2 replies

maleadt Nov 15, 2024
Author

Thanks for the reply. I've collected the traces you requested, albeit with 2 containers active (zigbee2mqtt relies on mqtt, so there's some traffic from that). I've included the journalctl log too so that it's possible to match up with where the ETIMEDOUT happens.

pasta.log.zip
pasta.pcap.zip
zigbee2mqtt.log

From your config it seems like you are using a custom network (Network=web.network) so the network flow is a bit complicated in that case (#22943 (comment))

I see; I'm pretty new to podman, so I simply started with a custom network because of thinking it was a necessity. And I followed https://github.com/eriksjolund/podman-traefik-socket-activation since I'm using a reverse proxy which I want to be able to see the original request IP (which I use for filtering purposes). Or doesn't that require a custom network?

Luap99 Nov 15, 2024
Maintainer

customs networks fine and when you need several containers talking directly to each then you want this over sending everything via ports on the host. Also --network pasta also keeps the source ip for ports but then the namespace is isloated and you then would need to forward the ports via the host to other containers.
I just mention this because it makes debugging harder compared to the default --network pasta where you could have added the arguments via the cli.

Luap99 · 2024-11-15T18:14:44Z

Luap99
Nov 15, 2024
Maintainer

126.1137:          pasta: epoll event on connected TCP socket 211 (events: 0x00000001)
126.1137:          Flow 0 (TCP connection): flag at tcp_buf_data_from_sock:393
126.1137:          Flow 0 (TCP connection): flag at tcp_buf_data_from_sock:414
126.1137:          Flow 0 (TCP connection): ACK_FROM_TAP_DUE
126.1137:          Flow 0 (TCP connection): timer expires in 0.010s
126.1138:          pasta: epoll event on connected TCP socket 211 (events: 0x00000001)
126.1138:          Flow 0 (TCP connection): flag at tcp_buf_data_from_sock:389
126.1138:          Flow 0 (TCP connection): STALLED
126.1138:          pasta: epoll event on connected TCP socket 211 (events: 0x00000005)
126.1138:          Flow 0 (TCP connection): flag at tcp_buf_data_from_sock:389
126.1237:          pasta: epoll event on TCP timer 211 (events: 0x00000001)
126.1237:          Flow 0 (TCP connection): timer expires in 0.010s
126.1338:          pasta: epoll event on TCP timer 211 (events: 0x00000001)
126.1338:          Flow 0 (TCP connection): timer expires in 0.010s
126.1439:          pasta: epoll event on TCP timer 211 (events: 0x00000001)
126.1439:          Flow 0 (TCP connection): timer expires in 0.010s
126.1453:          pasta: epoll event on /dev/net/tun device 9 (events: 0x00000001)
126.1453:          tap: protocol 6, 192.168.0.10:49958 -> 192.168.0.22:6638 (1 packet)
126.1453:          Flow 0 (TCP connection): packet length 20 from tap
126.1453:          Flow 0 (TCP connection): event at tcp_tap_handler:2010
126.1453:          Flow 0 (TCP connection): CLOSED: ESTABLISHED -> CLOSED
126.1453:          Flow 0 (TCP connection): Side 0 hash table remove: bucket: 179630
126.1453:          Flow 0 (FREE): ACTIVE -> FREE
126.1453:          Flow 0 (FREE): TAP [192.168.0.10]:49958 -> [192.168.0.22]:6638 => HOST [0.0.0.0]:0 -> [192.168.0.22]:6638

@sbrivio-rh @dgibson PTAL

5 replies

sbrivio-rh Nov 15, 2024
Collaborator

Heh, we get a RST from the client (frame #179 in the packet capture) and close the connection. I'm looking into the ZigBee thing now...

sbrivio-rh Nov 15, 2024
Collaborator

I think frames #114 and #115 in the packet capture are confusing pasta. According to Wireshark those should be interpreted as keep-alive segments, but if that's the case, why is the sequence rewinding by one (relative sequence: 268, then 267)?

pasta takes it as a request for fast retransmit, but it should probably just discard the whole thing because it's invalid. Still, the "znp" stack doesn't look like low-level enough as to be able to rewind a TCP sequence like that...

sbrivio-rh Nov 15, 2024
Collaborator

@maleadt could you please try this patch on top of current passt's HEAD:

diff --git a/tcp.c b/tcp.c
index c95dcaf..8b0c61b 100644
--- a/tcp.c
+++ b/tcp.c
@@ -1195,8 +1195,10 @@ int tcp_prepare_flags(const struct ctx *c, struct tcp_tap_conn *conn,
 	int s = conn->sock;
 
 	if (SEQ_GE(conn->seq_ack_to_tap, conn->seq_from_tap) &&
-	    !flags && conn->wnd_to_tap)
+	    !flags && conn->wnd_to_tap) {
+		conn_flag(c, conn, ~ACK_TO_TAP_DUE);
 		return 0;
+	}
 
 	if (getsockopt(s, SOL_TCP, TCP_INFO, &tinfo, &sl)) {
 		conn_event(c, conn, CLOSED);

...and see if it solves the problem for you? I'm not quite sure but it's worth a try. Building passt is really easy, git clone git://passt.top/passt; cd passt; make. You can install it to /usr/local/bin with make install, and make uninstall will wipe it from there, so it won't conflict with distribution packages.

Regardless of whether it solves the problem, a new pasta.log taken with --trace and this patch would be appreciated.

maleadt Nov 16, 2024
Author

Thanks for the patch. Didn't seem like that helped; here's a new trace: trace.zip

You can install it to /usr/local/bin with make install, and make uninstall will wipe it from there, so it won't conflict with distribution packages.

How do you set a custom path to passt in containers.conf? default_rootless_network_cmd doesn't work, and making sure /usr/local/bin comes first in PATH when starting the quadlet doesn't make podman use /usr/local/bin/pasta.

I ended up rebuilding the Debian package (based on git20241030.ee7d0b6) to be sure the patch was picked up.

sbrivio-rh Nov 16, 2024
Collaborator

Thanks for the patch. Didn't seem like that helped; here's a new trace: trace.zip

Thanks a lot, that's really helpful. The patch removed quite some cruft (useless rescheduling of a timer) and now things are a bit clearer.

I start thinking that the issue we hit might be related to the nature of the protocol which implies many small messages (1-20 bytes). I'm not sure in which way: we seem to transmit right away what the client sends, because we acknowledge it without delay (perhaps this is the issue?), and we don't acknowledge things until the kernel confirms to us that the receiver also acknowledged those bytes.

Looking at frames #61-76 of the latest capture you shared: the client sends 10 bytes (frame #61), pasta acknowledges it in 8ms (frame #62) together with 14 bytes of response (?). The client acknowledges that but it looks like it was waiting for some further response. It starts nervously sending keep-alive segments (no effect on pasta, frames #64-73), but nothing comes back from the server (I have no idea what it should be), until it decides to send a RST.

I'm wondering if some socket "corking" (further aggregation on top of what's dictated by Nagle's algorithm), as I would expect from slirp4netns, might help hiding some issue. To check if that's the case, could you try this patch (on top):

diff --git a/tcp.c b/tcp.c
index 6b31715..57e1543 100644
--- a/tcp.c
+++ b/tcp.c
@@ -1620,6 +1620,8 @@ static void tcp_conn_from_tap(const struct ctx *c, sa_family_t af,
 		flow_trace(conn, "failed to set TCP_MAXSEG on socket %i", s);
 	MSS_SET(conn, mss);
 
+	setsockopt(s, SOL_TCP, TCP_CORK, &(int){ 1 }, sizeof(int));
+
 	tcp_get_tap_ws(conn, opts, optlen);
 
 	/* RFC 7323, 2.2: first value is not scaled. Also, don't clamp yet, to

?

This is just an experiment (the kernel will try to aggregate all the data we send out of sockets for up to 200ms). If it doesn't help, by the way, the opposite experiment (just replace TCP_CORK with TCP_NODELAY) might also tell us something.

You can install it to /usr/local/bin with make install, and make uninstall will wipe it from there, so it won't conflict with distribution packages.

How do you set a custom path to passt in containers.conf? default_rootless_network_cmd doesn't work, and making sure /usr/local/bin comes first in PATH when starting the quadlet doesn't make podman use /usr/local/bin/pasta.

Ouch, sorry, I forgot. With podman-run it's the CONTAINERS_HELPER_BINARY_DIR environmental variable (say, CONTAINERS_HELPER_BINARY_DIR=/usr/local/bin/ or CONTAINERS_HELPER_BINARY_DIR=~/passt/. For containers.conf, that's the helper_binaries_dir array (see containers.conf(5)).

Something like helper_binaries_dir=["/usr/local/bin", "/usr/bin"] should work.

I ended up rebuilding the Debian package (based on git20241030.ee7d0b6) to be sure the patch was picked up.

Well, in the end, it probably takes about the same time and effort.

maleadt · 2024-11-16T20:03:04Z

maleadt
Nov 16, 2024
Author

Here's new traces with TCP_CORK and TCP_NODELAY; both didn't seem to have helped. In the case of TCP_CORK the timeout happened very quickly, while with TCP_NODELAY it took a little longer (so the pasta.log is significantly larger), but both may have been purely coincidental.

trace2.zip

1 reply

sbrivio-rh Nov 17, 2024
Collaborator

Here's new traces with TCP_CORK and TCP_NODELAY; both didn't seem to have helped. In the case of TCP_CORK the timeout happened very quickly, while with TCP_NODELAY it took a little longer (so the pasta.log is significantly larger), but both may have been purely coincidental.

Right, yeah, that didn't really affect the peculiar behaviour here: a bunch of keep-alives and a connection reset from the client.

There's something else I noticed though (I should have read that part of RFC 9293 more carefully, and it was actually already there in RFC 1122).

The first keep-alive segment (frame #77 in the "TCP_CORK" capture) is sent 15 seconds after the last data packet. This is definitely not the default Linux value that a process can get with SO_KEEPALIVE, it needs a sysctl change, so I wonder how your container actually does that (does it run with --privileged or something?).

Anyway, it fits with the "KeepAliveTime" attribute in Table 14-4 of the Zigbee Cluster Library Specification, so I would assume that some component from zigbee-herdsman or its dependencies is setting that.

And RFC 9293 3.8.4 says:

       Implementers MAY include "keep-alives" in their TCP implementations
       (MAY-5), although this practice is not universally accepted.  Some
       TCP implementations, however, have included a keep-alive mechanism.
       To confirm that an idle connection is still active, these
       implementations send a probe segment designed to elicit a response
       from the TCP peer.  Such a segment generally contains SEG.SEQ =
       SND.NXT-1 and may or may not contain one garbage octet of data.  If
       keep-alives are included, the application MUST be able to turn them
       on or off for each TCP connection (MUST-24), and they MUST default to
       off (MUST-25).

...it's the first time I see this, but it seems to fit: we have keep-alive segments with no data, and the sequence rewinds by 1. We should ignore those, though, not consider those as fast re-transmit requests as we do.

I prepared a branch with a tentative fix for this at https://passt.top/passt/log/?h=podman24572. You can also just cherry-pick commit 5dc962affc07 ("tcp: Properly ignore keep-alive segments") on top of the 2024_10_30 tag. Could you try that? Thanks a lot.

maleadt · 2024-11-17T14:40:42Z

maleadt
Nov 17, 2024
Author

I just realized that my previous trace might not have been with the setsockopts active, because I only installed the passt binary, and not passt.avx2 which it seems to dispatch to.

Anyhow, testing your branch (after fixing the if test, which needs a curly brace), everything seems to be working perfectly 🎉 Thanks for looking into this! I'll leave it running for a while to make sure; let me know if you need anything else.

1 reply

maleadt Nov 17, 2024
Author

Actually, later today I started seeing ETIMEDOUT happen again 😞 I think it must have something to do with activity on the zigbee network, i.e., when there's lots of activity the timeout doesn't happen. I'll try to reproduce again tomorrow on 5dc962affc07 when the network is idle.

maleadt · 2024-11-17T20:19:59Z

maleadt
Nov 17, 2024
Author

Here's a new trace, taken on https://passt.top/passt/commit/?id=5dc962affc07e6f8f449d56836476cb2350570ce:

❯ /usr/bin/pasta --version
passt 2024_10_30.ee7d0b6-45-g5dc962a

0 replies

sbrivio-rh · 2024-11-17T21:30:19Z

sbrivio-rh
Nov 17, 2024
Collaborator

(after fixing the if test, which needs a curly brace)

Oops, sorry, I build tested first, then slightly changed the condition. :)

Thanks a lot for re-testing and for the new traces. I didn't look into them yet, but still my earlier question remains: do you run the container as privileged (that is, rootless, but with all the capabilities available as the namespace is detached), or with any capability (CAP_NET_ADMIN) inside the namespace?

I'm wondering because it shouldn't be possible for a regular user to affect keep-alive parameters (net.ipv4.tcp_keepalive_time, net.ipv4.tcp_keepalive_intvl, net.ipv4.tcp_keepalive_probes sysctls). And yet we have keep-alive segments after 15 seconds, with one second interval.

Regardless of that, I guess we should forget for a moment about this part of RFC 9293 3.8.4:

   It is extremely important to remember that ACK segments that contain
   no data are not reliably transmitted by TCP.  Consequently, if a
   keep-alive mechanism is implemented it MUST NOT interpret failure to
   respond to any specific probe as a dead connection (MUST-29).

(I haven't had a look at the corresponding kernel implementation yet) and just make the client happy by acknowledging those keep-alive segments, because here it looks like the client/sender is interpreting that as a dead connection.

4 replies

sbrivio-rh Nov 17, 2024
Collaborator

just make the client happy by acknowledging those keep-alive segments

Something like commit 3a93acc7c4c0 in the updated podman24572 branch. Could you give that a shot?

dgibson Nov 18, 2024
Collaborator

I'm wondering because it shouldn't be possible for a regular user to affect keep-alive parameters (net.ipv4.tcp_keepalive_time, net.ipv4.tcp_keepalive_intvl, net.ipv4.tcp_keepalive_probes sysctls). And yet we have keep-alive segments after 15 seconds, with one second interval.

Sorry, I've only started looking at this ticket just now. But I did wonder about this: could the application be overriding the keepalive behaviour without privilege using the TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL socket options?

maleadt Nov 18, 2024
Author

my earlier question remains: do you run the container as privileged (that is, rootless, but with all the capabilities available as the namespace is detached), or with any capability (CAP_NET_ADMIN) inside the namespace?

Sorry, missed that question. I'm running rootless, without privileged, without adding any additional capabilities (to this container at least):

On the host I have:

❯ rg ipv4 /etc/sysctl.*
/etc/sysctl.d/99-rootless.conf
1:net.ipv4.ip_unprivileged_port_start=80
2:net.ipv4.ping_group_range=0 2000000

/etc/sysctl.d/syn_flood.conf
8:net.ipv4.tcp_max_syn_backlog=4096


❯ sudo sysctl -a | grep ipv4.tcp_keepalive
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200

just make the client happy by acknowledging those keep-alive segments

Something like commit 3a93acc7c4c0 in the updated podman24572 branch. Could you give that a shot?

Cautiously optimistic about this; it seems to be working fine, but since I misreported success before I'll keep it running for a while first. I've recorded a trace of a couple of minutes of everything working fine on this commit, let me know if it's useful to share those.

could the application be overriding the keepalive behaviour without privilege using the TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL socket options?

From a quick look I couldn't find those in the relevant sources:

adaptor: https://github.com/mercenaruss/uzg-firmware/
client: https://github.com/Koenkk/zigbee2mqtt

sbrivio-rh Nov 18, 2024
Collaborator

But I did wonder about this: could the application be overriding the keepalive behaviour without privilege using the TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL socket options?

Oops, right, I totally missed those.

By the way, we could even think of replicating those (approximately) on our sockets: we can infer at least TCP_KEEPIDLE and TCP_KEEPINTVL. Maybe in some cases, including here, those are actually needed by the receiver. But dropping the connection like Linux does probably clashes with MUST-29 of RFC 9293.

From a quick look I couldn't find those in the relevant sources:
* adaptor: https://github.com/mercenaruss/uzg-firmware/

* client: https://github.com/Koenkk/zigbee2mqtt

I think it comes from https://nodejs.org/api/net.html#socketsetkeepaliveenable-initialdelay and this in openSocketPort() in src/adapter/z-stack/znp/znp.ts (zigbee-herdsman):

        this.socketPort.setKeepAlive(true, 15000);

the other parameters seem to match: 10 keep-alive segments at one-second intervals before terminating the connection.

maleadt · 2024-11-19T08:06:01Z

maleadt
Nov 19, 2024
Author

https://passt.top/passt/commit/?id=3a93acc7c4c0eb26eab4f6a2362ce89fb4b70509 has been working fine for over 24h, so I think we can declare victory here. Thanks for debugging this!

1 reply

sbrivio-rh Nov 19, 2024
Collaborator

And thanks for testing over and over! I'll submit the essential fix soon.

Longer term, there might be a quick, approximate way to send outbound keep-alives if we detect them guest/container-side, for example setting TCP_KEEPIDLE to 1 as soon as we receive one, with a very high (INT_MAX) TCP_KEEPINTVL (we don't want to reset the connection, unlike Linux does, because of MUST-29 in RFC 9293), so that we send out one (and just one) keep-alive segment shortly after we get one. I might have a look into this at some point. Patches of course welcome! I expect it to be a relatively small and self-contained implementation.

sbrivio-rh · 2024-11-21T15:50:08Z

sbrivio-rh
Nov 21, 2024
Collaborator

This is now fixed in passt 2024_11_21.238c69f, and matching Debian package passt-0.0~git20241121.238c69f-1.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Long-standing connections from container are aborted (rootless, pasta) #24572

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 14 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Long-standing connections from container are aborted (rootless, pasta) #24572

maleadt Nov 14, 2024

Replies: 9 comments · 14 replies

maleadt Nov 15, 2024 Author

Luap99 Nov 15, 2024 Maintainer

maleadt Nov 15, 2024 Author

Luap99 Nov 15, 2024 Maintainer

Luap99 Nov 15, 2024 Maintainer

sbrivio-rh Nov 15, 2024 Collaborator

sbrivio-rh Nov 15, 2024 Collaborator

sbrivio-rh Nov 15, 2024 Collaborator

maleadt Nov 16, 2024 Author

sbrivio-rh Nov 16, 2024 Collaborator

maleadt Nov 16, 2024 Author

sbrivio-rh Nov 17, 2024 Collaborator

maleadt Nov 17, 2024 Author

maleadt Nov 17, 2024 Author

maleadt Nov 17, 2024 Author

sbrivio-rh Nov 17, 2024 Collaborator

sbrivio-rh Nov 17, 2024 Collaborator

dgibson Nov 18, 2024 Collaborator

maleadt Nov 18, 2024 Author

sbrivio-rh Nov 18, 2024 Collaborator

maleadt Nov 19, 2024 Author

sbrivio-rh Nov 19, 2024 Collaborator

sbrivio-rh Nov 21, 2024 Collaborator

maleadt
Nov 14, 2024

Replies: 9 comments 14 replies

maleadt
Nov 15, 2024
Author

Luap99
Nov 15, 2024
Maintainer

maleadt Nov 15, 2024
Author

Luap99 Nov 15, 2024
Maintainer

Luap99
Nov 15, 2024
Maintainer

sbrivio-rh Nov 15, 2024
Collaborator

sbrivio-rh Nov 15, 2024
Collaborator

sbrivio-rh Nov 15, 2024
Collaborator

maleadt Nov 16, 2024
Author

sbrivio-rh Nov 16, 2024
Collaborator

maleadt
Nov 16, 2024
Author

sbrivio-rh Nov 17, 2024
Collaborator

maleadt
Nov 17, 2024
Author

maleadt Nov 17, 2024
Author

maleadt
Nov 17, 2024
Author

sbrivio-rh
Nov 17, 2024
Collaborator

sbrivio-rh Nov 17, 2024
Collaborator

dgibson Nov 18, 2024
Collaborator

maleadt Nov 18, 2024
Author

sbrivio-rh Nov 18, 2024
Collaborator

maleadt
Nov 19, 2024
Author

sbrivio-rh Nov 19, 2024
Collaborator

sbrivio-rh
Nov 21, 2024
Collaborator