Inquiry about ARAP Loss Function

[SHIBOYA]

During my debugging sessions, I noticed that the program does not seem to enter the arap_loss function. Consequently, it appears that the ARAP loss is not utilized during the training process. And the arap_loss remains 0 during the entire training.

Could you please provide some clarification or insights into why this might be happening?

Thank you very much for your assistance. Looking forward to your response.

[yihua7]

Hi,
The code does use the ARAP loss and I'm not sure why your debugging didn't go through it. You can check the function here:

SC-GS/utils/time_utils.py

Line 1080 in 26cd57d

def arap_loss(self, t=None, delta_t=0.05, t_samp_num=2):

.

The ARAP loss is added to the training objective after the warm up stage. Please view the code here:

SC-GS/train_gui.py

Line 1288 in 26cd57d

if self.iteration_node_rendering > self.opt.node_warm_up:

[SHIBOYA]

Thank you for your helpful response! Your guidance has resolved my problem.

I have another question regarding the calculation of FPS (Frames Per Second). In my tests, the maximum FPS I am able to achieve is around 200, which is below the target of 300 FPS. How to reach the desired 300 FPS?

[SHIBOYA]

I use this code to test the FPS:
https://github.com/ingra14m/Deformable-3D-Gaussians/blob/ac423728df780ee57a82025a6d7c4c7312fdf9bb/render.py#L74

[SHIBOYA]

And, I have encountered another question.

I am trying to understand the feature attribute of self.nodes. What the feature specifically refers to, and where in the code it is assigned its values? I noticed the dimension "3 + self.hyper_dim", where I understand that 3 refers to the xyz coordinates. Which feature dimensions does the self.hyper_dim represent?

[yihua7]

Thank you for your helpful response! Your guidance has resolved my problem.

I have another question regarding the calculation of FPS (Frames Per Second). In my tests, the maximum FPS I am able to achieve is around 200, which is below the target of 300 FPS. How to reach the desired 300 FPS?

Hi, please firstly press the cache_nn button on the GUI or call the callback function here:

SC-GS/train_gui.py

Line 518 in 26cd57d

def callback_cache_nn(sender, app_data):

to store the pre-computed nearest neighbourhood after training, which avoids the re-searching for NN during the inference time.

I recommend you to call the function here:

SC-GS/train_gui.py

Line 1641 in 26cd57d

def test_speed(self, round=500):

to test the FPS.

[yihua7]

And, I have encountered another question.

I am trying to understand the feature attribute of self.nodes. What the feature specifically refers to, and where in the code it is assigned its values? I noticed the dimension "3 + self.hyper_dim", where I understand that 3 refers to the xyz coordinates. Which feature dimensions does the self.hyper_dim represent?

Hi, this is a good question! And that's our special design for the spatially sparse control.
If we use the first 3 coordinates only, the NN control points are searched in the 3D Euclidean space via a P2 norm distance. However, it may results in some artifacts for those regions close to each other but not connected, because their motions are independent and shouldn't rely on some shared control points, even they are quite close. (e.g the base and the arm in Lego scene).
We assign learnable hypercoordinates to both control points and Gaussians to lift them up from 3D space to ND hyperspace, making them able to learn a more flexible topology and spatial relationship there without degrading their distribution on Euclidean space. This is our insight on the design.

Hope this info helps!

[SHIBOYA]

Thank you very much for your detailed response to my questions. It is indeed an impressive work!

[SHIBOYA]

Hi, I am currently trying to use your repository with the DyNeRF dataset, eg. the cut_roasted_beef. However, I am unsure of the correct way to input this dataset into your code. Could you please let me know if your repository supports the DyNeRF dataset? If it does, could you provide some guidance on how to correctly set up and input the cut_roasted_beef, or any other DyNeRF dataset?

[SHIBOYA]

When I try this command:

python train_gui.py --source_path /data/dynerf/cut_roasted_beef  --model_path outputs/cut_roasted_beef --deform_type node --node_num 512 --hyper_dim 8 --init_isotropic_gs_with_all_colmap_pcl --eval --gt_alpha_mask_as_scene_mask --local_frame --resolution 2 --W 800 --H 800

where I refer to issue#12, it occurs errors:

Traceback (most recent call last):
  File "train_gui.py", line 2086, in <module>
    gui.train(args.iterations)
  File "train_gui.py", line 1073, in train
    self.train_node_rendering_step()
  File "train_gui.py", line 1502, in train_node_rendering_step
    hyper_pcl = (trans_samp + original_gaussians.get_xyz[:, None]).reshape(
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous
Training progress:   0%|                                                                                                           | 0/80000 [09:06<?, ?it/s]