mcap record ros1 messages with rosbag-like topic regex include/exclude and size limit rollover

[lucasw]

I think I have a decent threaded architecture where there is a tokio thread per ros1 subscriber, but all they do is shove messages into a mpsc which a thread that owns the mcap file writer object receives,. There's also schema handling in another thread which is where message definitions go. But as before any suggestions are welcome (especially improving the handling of the ctrl-c).

https://github.com/lucasw/mcap_tools/blob/main/mcap_tools/src/mcap_record.rs

Later on I'll see if anyone in the foxglove mcap github discussion area (or discord) can say anything about how I'm handling the mcap files.

It looks to perform on par with rosbag record -a --bz2 with some tests bags I've thrown at it (e.g. 3500 Hz total message throughput on around 100 topics, 6MB/s total).

It depends on https://github.com/lucasw/roslibrust/commits/subscribe_any/ which has some modifications (some of which are getting merged in here).

I've noticed a lot (e.g. 2 or 3%) of messages are missing from the recorded mcap (and rosbag record bag is no better) even when starting the recorder, then the playback, and killing the recorder well after the playback is done. I'm guessing they are from the beginning of the playback, and maybe it's the fault of rosbag play (even --delay 1.0 doesn't help, though not sure how properly that is implemented). If I develop the mcap playback further it'll be a valuable tool for debugging this, and it wouldn't be hard to write a script to determine exactly which messages didn't make it and where they are distributed in time.

This integration test publishes thousands of messages to itself over a few seconds and they all arrive unlike in the messier rosbag playback environment: lucasw@019d47d - any interest in that going into to roslibrust here?

[Carter12s]

This is awesome!

Didn't have a ton of time at the moment, to take a look, but would like to dig in later.

Glad to hear the performance is okay, we really didn't pay much attention to it and I know there are some big savings we can still do.

Would love to setup integration tests with rosbag, rospy, roscpp etc. to start catching more of the interopability bugs. I'd defo want to dig in to any dropped messages here. It should be pretty easy to setup wireshark to capture the packets between rust and rosbag and see where the packets are being dropped.

I'll try to dig into this a little more next time I get a chance.