[LINUX][NVIDIA] Doesn't work properly with NVIDIA GPU

[mercifulboss]

When using the "CPU" option, the background remover updates as you move around and continues removing the background. When you switch to "GPU - TensorRT" option, it updates the silhouette once and then doesn't update or do anything after the first time.

To Reproduce

Steps to reproduce the behavior:

1. Go to Background Removal Filter on OBS
2. Click on "GPU - TensorRT"
3. See it doesn't update the background

Expected behavior

Supposed to update the background using the "GPU" option

Desktop (please complete the following information):

OS: Arch Linux x86_64
Kernel: 6.9.9-arch1-1
DE: Plasma 6.1.2
CPU: Intel i7-10750H (12) @ 5.000GHz
GPU: NVIDIA GeForce RTX 2060 Mobile
Browser: Firefox
OBS Version: 30.2.0

[royshil]

can you please share the OBS logs so we can investigate?

[nessvan]

Thank you Roy for the great work!
I have the same issue:

This part of the log is for cpu where it works:

info: [obs-backgroundremoval] get_latest_version:
info: [obs-backgroundremoval] Background filter updated
info: [obs-backgroundremoval] Model models/mediapipe.onnx input 0: name input_1:0 shape (4 dim) 1 x 144 x 256 x 3
info: [obs-backgroundremoval] Model models/mediapipe.onnx output 0: name segment:0 shape (4 dim) 1 x 144 x 256 x 2
info: [obs-backgroundremoval] Allocated 110592 sized float-array for input 0
info: [obs-backgroundremoval] Allocated 73728 sized float-array for output 0
info: [obs-backgroundremoval] Background Removal Filter Options:
info: [obs-backgroundremoval] Source: Background Removal
info: [obs-backgroundremoval] Model: models/mediapipe.onnx
info: [obs-backgroundremoval] Inference Device: cpu
info: [obs-backgroundremoval] Num Threads: 1
info: [obs-backgroundremoval] Enable Threshold: true
info: [obs-backgroundremoval] Threshold: 0.500000
info: [obs-backgroundremoval] Contour Filter: 0.050000
info: [obs-backgroundremoval] Smooth Contour: 0.500000
info: [obs-backgroundremoval] Feather: 0.000000
info: [obs-backgroundremoval] Mask Every X Frames: 1
info: [obs-backgroundremoval] Enable Image Similarity: true
info: [obs-backgroundremoval] Image Similarity Threshold: 35.000000
info: [obs-backgroundremoval] Blur Background: 0
info: [obs-backgroundremoval] Enable Focal Blur: false
info: [obs-backgroundremoval] Blur Focus Point: 0.100000
info: [obs-backgroundremoval] Blur Focus Depth: 0.000000
info: [obs-backgroundremoval] Disabled: true
info: [obs-backgroundremoval] Model file path: /usr/share/obs/obs-plugins/obs-backgroundremoval/models/mediapipe.onnx

This part is for GPU-TensorRT enabled:

info: [obs-backgroundremoval] get_latest_version:
info: [obs-backgroundremoval] get_latest_version:
info: [obs-backgroundremoval] Background filter updated
error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libnvinfer.so.8: cannot open shared object file: No such file or directory

error: [obs-backgroundremoval] Failed to create ONNXRuntime session. Error code: 5
info: [obs-backgroundremoval] Background filter updated
info: [obs-backgroundremoval] Background Removal Filter Options:
info: [obs-backgroundremoval] Source: Background Removal
info: [obs-backgroundremoval] Model: models/mediapipe.onnx
info: [obs-backgroundremoval] Inference Device: tensorrt
info: [obs-backgroundremoval] Num Threads: 1
info: [obs-backgroundremoval] Enable Threshold: true
info: [obs-backgroundremoval] Threshold: 0.500000
info: [obs-backgroundremoval] Contour Filter: 0.050000
info: [obs-backgroundremoval] Smooth Contour: 0.500000
info: [obs-backgroundremoval] Feather: 0.000000
info: [obs-backgroundremoval] Mask Every X Frames: 1
info: [obs-backgroundremoval] Enable Image Similarity: true
info: [obs-backgroundremoval] Image Similarity Threshold: 35.000000
info: [obs-backgroundremoval] Blur Background: 0
info: [obs-backgroundremoval] Enable Focal Blur: false
info: [obs-backgroundremoval] Blur Focus Point: 0.100000
info: [obs-backgroundremoval] Blur Focus Depth: 0.000000
info: [obs-backgroundremoval] Disabled: true
info: [obs-backgroundremoval] Model file path: /usr/share/obs/obs-plugins/obs-backgroundremoval/models/mediapipe.onnx
error: [obs-backgroundremoval] Model is not initialized
error: [obs-backgroundremoval] Model is not initialized
error: [obs-backgroundremoval] Model is not initialized

It seems it can not find the right version of libnvinfer (I do have /usr/lib/x86_64-linux-gnu/libnvinfer.so.10)

[razllivan]

Ubuntu 24.04
I have same issue. Part of the log:

error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libnvinfer.so.8: cannot open shared object file: No such file or directory

error: [obs-backgroundremoval] Failed to create ONNXRuntime session. Error code: 5
error: [obs-backgroundremoval] Model is not initialized

[umireon]

What version of CUDA do you have?

[umireon]

@royshil I suppose many want to use our plugin with CUDA 12. Are you okay with our plugin supporting CUDA 12 and dropping support for CUDA 11? Users who wish to use our plugin with CUDA 11 can still use the older version of our plugin.

[razllivan]

What version of CUDA do you have?

12.2

[umireon]

CUDA 12 is not supported. Can you try CUDA 11?

[razllivan]

CUDA 12 is not supported. Can you try CUDA 11?

I would try it if I could find a way to downgrade the version

[ryzendew]

CUDA 12 is not supported. Can you try CUDA 11?

11 is not supported anymore and no way to downgrade either on Fedora or even Arch it will always give an error

[HughPH]

Same issue here. Even if I were to hack around the issue, it would be onerous to have to switch CUDA versions every time I want to use OBS - and would make it impossible to use both at once.

[fritz-fritz]

Also would love to see support for CUDA 12 for this amazing plugin. Without it, I am forced to use CPU. In order to have acceptable results with CPU it introduces significant CPU consumption which leads to encoder overload and dropped frames.

Took a cursory glance at the codebase but didn't see where the tensorrt relevant code was. Could you point to what makes this break on CUDA 12?

[umireon]

I don't think nothing would break with CUDA 12. We are just searching for people who can make changes to support CUDA 12 for our code.

[fritz-fritz]

Does tensorrt need to be installed for this to function properly? Or is CUDA theoretically enough (assuming on the supported version). Wonder if that could be the root of the issue as I have the same exact symptoms and only have CUDA 12 installed.

[HughPH]

Does tensorrt need to be installed for this to function properly? Or is CUDA theoretically enough (assuming on the supported version). Wonder if that could be the root of the issue as I have the same exact symptoms and only have CUDA 12 installed.

The problem right now is that it targets the wrong version of the libraries - so it's simply a File Not Found. Since there's a major version change in the library versions, there could be breaking changes. The way I would approach it would be to target the later library version(s) and then test. Others might list all the API calls and form a gap analysis between the library versions, then make targeted fixes if there are any.

[darkhog]

Does disabling "Skip image based on similarity" in the advanced settings fix the issue?

[oasido]

Does disabling "Skip image based on similarity" in the advanced settings fix the issue?

Doesn't change anything for me.

---

Running on openSUSE Tumbleweed:

info: [obs-backgroundremoval] Background filter created
info: [obs-backgroundremoval] Background filter updated
error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libcudnn.so.8: cannot open shared object file: No such file or directory

error: [obs-backgroundremoval] Failed to create ONNXRuntime session. Error code: 5

Cuda 12.6, TensorRT 10.3.0.26
Even tried symlinking version 9 files to 8 just for the lols, to no avail.. (might have done that incorrectly)

Is there any documentation on what version to use/what to specifically install?
Also, should I downgrade either TensorRT/Cuda (or both) or wait for the fix? 😜

---

Some more info if you find it useful:

info: Kernel Version: Linux 6.10.3-1-default
info: Flatpak Branch: stable
info: Flatpak Arch: x86_64
info: Flatpak Runtime: runtime/org.kde.Platform/x86_64/6.6
info: App Extensions:
info:  - com.obsproject.Studio.Plugin.BackgroundRemoval
info:  - com.obsproject.Studio.Locale
info: Runtime Extensions:
info:  - org.freedesktop.Platform.GL.default
info:  - org.freedesktop.Platform.GL.nvidia-550-107-02
info:  - org.freedesktop.Platform.openh264
info:  - org.gtk.Gtk3theme.Breeze
info:  - org.kde.Platform.Locale
info:  - org.freedesktop.Platform.GL.default
info: Flatpak Framework Version: 1.15.8
info: Desktop Environment: KDE (KDE)
info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.21.1
info: Qt Version: 6.6.3 (runtime), 6.6.3 (compiled)
info: Portable mode: false
info: OBS 30.2.2 (linux)

[Gelmo]

Looking forward to a new build with support for CUDA 12

[j2l]

Same error here using CUDA 12.6, can't downgrade.
Would it be possible to make a docker container that use CUDA 11 while accessing the webcam on host?
Since it's a GUI App I wonder.

[madalinignisca]

Why not support current CUDA, and people who stay on old versions use the CPU?

In general, we use "enterprise" distributions such as Debian for servers and opt for distributions that provide a better desktop and laptop experience.

[leandroribeiro]

Same error here.

OS: Linux Mint 21.3 x86_64
DE: Cinnamon 6.0.4
Kernel: 6.8.0-47-generic
CPU: AMD Ryzen 5 3600 (12) @ 3.600GHz
GPU: NVIDIA GeForce RTX 4060 Ti
Driver Version: 560.35.03
CUDA Version: 12.6
OBS: 30.2.3

[darkhog]

Confirmed here. KDE Plasma X11, RTX 3070, drivers 550.127.05, distro openSuSE Tumbleweed. I don't think updating drivers will help, and won't do that anyway until they're in opensuse repo to avoid problems.