loading different yolov8 model other then the standard downloadable lead to crash of ros2 composable_node

[TeamRoboTo]

hello everyone,
i did train a yolov8s model by my self but when i send it as launch file parameter, everithing goes well unitll i start the image_publisher node, that lead to the composable_node to crash without any infos about the crash in itself.
Can you help me?
I'll provide further infos if necessary.
Thanks in advance for you time,
Franzhd, member of Team Roboto.

[kajananchinniahNV]

Hi Franzhd,

Just to confirm, have you tried running the quickstart and did it work as expected? Also, what platform are you running this on?

If you could provide us with the entire log that would be useful. Even if it doesn't provide any info about the crash, we might be able to see if anything went wrong during launch. Also, having steps to reproduce the issue would be nice as well.

[TeamRoboTo]

Hello!
yep, i followed strictly the quickstart guide, only changed the parameter for model and plan.
Also i used the image_publisher to access the stream from the built in webcam.
i'm on UBUNTU LINUX 22.04 x86, inside the official docker container.
This is the standard version from the quick start, runned with this command:
cd /workspaces/isaac_ros-dev && ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/tmp/yolov8s.onnx engine_file_path:=/tmp/yolov8s.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45

This is the command that i run myself, runned with this command:
cd /workspaces/isaac_ros-dev && ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=./src/isaac_ros_object_detection/resources/s_best.onnx engine_file_path:=./src/isaac_ros_object_detection/resources/s_best.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45

changing the --log-level inside the yolov8_tensor_rt.launch.py to DEBUG, it gave me this:

Thanks, in Advance for your help,
Franzhd, Team Roboto.

[TeamRoboTo]

This is the full log file:

[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-11-10-13-08-55-189247-Egidio-24912
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [24923]
[INFO] [isaac_ros_yolov8_visualizer.py-2]: process started with pid [24925]
[component_container_mt-1] [INFO] [1699618135.582171538] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/lib/libdnn_image_encoder_node.so
[component_container_mt-1] [INFO] [1699618135.592713725] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::DnnImageEncoderNode
[component_container_mt-1] [INFO] [1699618135.592764770] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::DnnImageEncoderNode
[component_container_mt-1] [INFO] [1699618135.595075367] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1699618135.595159835] [dnn_image_encoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1699618135.595515149] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1699618135.599350873] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1699618135.601355883] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1699618135.603490936] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1699618135.606247742] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1699618135.606723383] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1699618135.609863533] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] �[33m2023-11-10 13:08:55.609 WARN gxf/std/program.cpp@514: No system specified. Nothing to do�[0m
[component_container_mt-1] [INFO] [1699618135.610421997] [dnn_image_encoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1699618135.610436181] [dnn_image_encoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1699618135.611494960] [dnn_image_encoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1699618135.611504906] [dnn_image_encoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1699618135.612646248] [dnn_image_encoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1699618135.612655400] [dnn_image_encoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1699618135.612828763] [dnn_image_encoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1699618135.612941141] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1699618135.613844858] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1699618135.614780564] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1699618135.617354916] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1699618135.620207594] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1699618135.620991880] [dnn_image_encoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1699618135.623821309] [dnn_image_encoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1699618135.706303928] [dnn_image_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1699618135.708478148] [dnn_image_encoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1699618135.715748675] [dnn_image_encoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dnn_image_encoder' in container '/tensor_rt_container'
[component_container_mt-1] [INFO] [1699618135.717476298] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_tensor_rt/lib/libtensor_rt_node.so
[component_container_mt-1] [INFO] [1699618135.745946262] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::TensorRTNode
[component_container_mt-1] [INFO] [1699618135.745991929] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::TensorRTNode
[component_container_mt-1] [INFO] [1699618135.749109613] [tensor_rt]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1699618135.749684961] [tensor_rt]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1699618135.749695549] [tensor_rt]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1699618135.750855132] [tensor_rt]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1699618135.750888456] [tensor_rt]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1699618135.751154350] [tensor_rt]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1699618135.751158700] [tensor_rt]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1699618135.751240442] [tensor_rt]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1699618135.751604284] [tensor_rt]: [NitrosContext] Loading extension: gxf/tensor_rt/libgxf_tensor_rt.so
[component_container_mt-1] [INFO] [1699618135.752749844] [tensor_rt]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1699618135.753641743] [tensor_rt]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1699618135.760960677] [tensor_rt]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1699618135.761525421] [tensor_rt]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1699618135.761543156] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "inference/rx" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1699618135.761556515] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1699618135.762876603] [tensor_rt]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/tensor_rt' in container '/tensor_rt_container'
[component_container_mt-1] [INFO] [1699618135.764041196] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_yolov8/lib/libyolov8_decoder_node.so
[component_container_mt-1] [INFO] [1699618135.767948614] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::yolov8::YoloV8DecoderNode
[component_container_mt-1] [INFO] [1699618135.767974721] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::yolov8::YoloV8DecoderNode
[component_container_mt-1] [INFO] [1699618135.771607201] [yolov8_decoder_node.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1699618135.771708514] [tensor_rt]: Negotiating
[component_container_mt-1] [INFO] [1699618135.771750225] [tensor_rt]: Could not negotiate
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/yolov8_decoder_node' in container '/tensor_rt_container'
[component_container_mt-1] [INFO] [1699618135.823895525] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618135.825065956] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618135.825374837] [dnn_image_encoder]: Negotiating
[component_container_mt-1] [INFO] [1699618135.847778976] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618135.884266697] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618135.916656467] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618135.950436134] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618135.982372779] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.014868485] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.048557047] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.080010593] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.115595566] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.149796634] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.181579447] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.214816025] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.246244541] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.277365303] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.311849253] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.344481576] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.379480852] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.412692778] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.443042909] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.477456313] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.510254927] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.542612481] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.575816677] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.610059008] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.643105421] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.676030917] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.708685238] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1699618136.716161307] [dnn_image_encoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1699618136.716263865] [dnn_image_encoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1699618136.716315190] [dnn_image_encoder]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1699618136.716352848] [dnn_image_encoder]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1699618136.716382651] [dnn_image_encoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1699618136.716503863] [dnn_image_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1699618136.761597338] [dnn_image_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/share/isaac_ros_dnn_image_encoder/LAIZSKTIJZ.yaml"
[component_container_mt-1] [INFO] [1699618136.761649343] [dnn_image_encoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1699618136.761657420] [dnn_image_encoder]: In DNN Image Encoder Node preLoadGraphCallback().
[component_container_mt-1] [INFO] [1699618136.761677990] [dnn_image_encoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1699618136.761685110] [dnn_image_encoder]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/share/isaac_ros_dnn_image_encoder/LAIZSKTIJZ.yaml'
[component_container_mt-1] [INFO] [1699618136.763108944] [tensor_rt]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1699618136.763137479] [tensor_rt]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1699618136.763142890] [tensor_rt]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1699618136.763147230] [tensor_rt]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1699618136.763152841] [tensor_rt]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1699618136.763165206] [tensor_rt]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1699618136.764401775] [dnn_image_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1699618136.764455474] [dnn_image_encoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1699618136.764460883] [dnn_image_encoder]: In DNN Image Encoder Node postLoadGraphCallback().
[component_container_mt-1] [INFO] [1699618136.764546636] [dnn_image_encoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1699618136.765181201] [tensor_rt]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_tensor_rt/share/isaac_ros_tensor_rt/QVKVPJAXEA.yaml"
[component_container_mt-1] [INFO] [1699618136.765201638] [tensor_rt]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1699618136.765205509] [tensor_rt]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1699618136.772931135] [dnn_image_encoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1699618136.773048088] [dnn_image_encoder]: [NitrosNode] Starting a heartbeat timer (eid=141)
[component_container_mt-1] [INFO] [1699618136.773058831] [tensor_rt]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_tensor_rt/share/isaac_ros_tensor_rt/QVKVPJAXEA.yaml'
[component_container_mt-1] �[33m2023-11-10 13:08:56.774 WARN gxf/std/yaml_file_loader.cpp@1058: Using unregistered parameter 'dummy_rx' in component ''.�[0m
[component_container_mt-1] �[33m2023-11-10 13:08:56.775 WARN gxf/std/yaml_file_loader.cpp@1058: Using unregistered parameter 'dev_id' in component 'stream'.�[0m
[component_container_mt-1] [INFO] [1699618136.775283784] [tensor_rt]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1699618136.775325973] [tensor_rt]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1699618136.775332028] [tensor_rt]: In TensorRTNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1699618136.919363487] [tensor_rt]: Read tensor shape information from TRT Model Engine: ./src/isaac_ros_object_detection/resources/s_best.plan
[component_container_mt-1] [INFO] [1699618136.919443705] [tensor_rt]: Tensors 168000 bytes, num outputs 40 x tensors per output 1 = 40 blocks
[component_container_mt-1] [INFO] [1699618136.919534664] [tensor_rt]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1699618136.923644628] [tensor_rt]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1699618136.923764293] [tensor_rt]: [NitrosNode] Starting a heartbeat timer (eid=228)
[ERROR] [component_container_mt-1]: process has died [pid 24923, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args --log-level INFO --ros-args -r __node:=tensor_rt_container -r __ns:=/'].

[kajananchinniahNV]

Unfortunately, you're right that the log didn't provide us with additional info.

I was wondering if you could provide me with additional information about the model you're running. What are the input and output sizes of it? If possible, could you also try launching the graph without all the components? For example, running the yolov8 model without the yolov8_decoder in yolov8_tensor_rt.launch.py and seeing if it crashes. If it doesn't, then we know which node is problematic, otherwise, try removing the tensor_rt_node and seeing if it crashes, etc.

Adding on, having the steps you took to prepare your model (or access to the model in onnx format if that is acceptable) would also be useful.

[TeamRoboTo]

i had access to the testing setup only today, i'm going to proceed with testing asap.

[kajananchinniahNV]

Adding on, we recently had a similar report from another user (#32) using a custom model. The issue stems from parameters that we hard coded in the decoder node. We'll be working out a solution soon.

In the mean time, could you give the fix recommended here a try? If your custom model doesn't have the same dimensions as the model we have in our example, that could cause problems.