Flickering of the disparity image

[TessaLin]

Using 20 frames image at same position, and disparity image will flicker

photo and video:
https://drive.google.com/file/d/1P0hc1h0rokUyQbUM063c_SlsCZFn48P8/view?usp=sharing

[TessaLin]

Parameter:
kitti.tar
testres=0.5
max_disp=1280
level=1

[gengshan-y]

The pixel intensity changing over time, and the KITTI model being not very robust may cause the flickering.
Switching to Middlebury model with testres=3 (based on your image size) should give better results.

[TessaLin]

Change to Middlebury model,still have filckering issue
Parameter:
final-768px.tar
testres=3
max_disp=480
level=1

https://drive.google.com/file/d/1jdjg526eSbH55_GX7IKB59mEiwc_fdVS/view?usp=sharing

[gengshan-y]

Max disparity to search over seems small. Can you try increasing max_disp to 1280? If memory turns out to be an issue, you could set testres=2.

The flicking could be removed by passing --clean 1.0, which filters uncertain disparity predictions.

[daalberti1]

Hi, I think I'm dealing with the same problem. @TessaLin did you eventually find a solution?

I'm performing a 3D reconstruction based on the images (with resolution 1920*1080) that I receive from the binocular endoscope of the da Vinci Research Kit and actually the disparity map that I'm obtaining with this method seems to be quite good. However I noticed that the disparity map changes even if the endoscope stays still and as a consequence the corresponding point cloud that I compute through reprojectImageTo3D (by OpenCV) is not stable.

I followed @gengshan-y suggestions and then I supposed that the problem may have been related to the presence of gaussian noise in the images, so I tried to apply some filters but without any significant improvement.

[gengshan-y]

@elDvd96 If the problem is still there, can you share some example outputs? Ideally the model should be robust to small input noise because of training time color augmentations. Depending on how much noise are there in your inputs, one solution is to add gaussian noise to input images here and finetune the model.

[daalberti1]

@gengshan-y Thank you for the reply and forgive me for the delay. Here you can find some examples of the rectified input images taken from the endoscope and the computed disparity maps and point clouds.
I'm also adding a video which show more clearly the problem.
I tried to tune the model following your suggestions but I didn't get better results.

[gengshan-y]

Hi @elDvd96, I looked at your outputs. The flickering in frame 0-5 seems expected, which is due to sensor noise (ISO too large?). To reduce sensor noise, you may want to use a lower ISO and longer exposure, but I'm not familiar with the one you are using.

The sudden change from frame 5 to frame 6 is abnormal, and seem to be a bug in saving disparity values, which can be fixed.

[daalberti1]

Hi @gengshan-y, unfortunately I think it's not possible to adjust ISO and exposure of the endoscope, anyway thank you for your suggestions, I'll look at least for the bug in saving disparity values.