LEN, ODN, and Joint training result looks odd

[chengzhag]

Hi Yinyu:

I tried LEN, ODN training code with batch size 32, but get the following loss curve:

It looks like the test loss stops converging soon after a few epochs.
All the test loss shows below:

I then tried joint training with the best LEN and ODN I have and the downloaded pretrained MGNet (Total3D_downloaded_mgnet):

The test results of IoU compared to Total3D without joint training (Total3D_mgnet_beforejoint) and the downloaded pretrained Total3D (Total3D downloaded):

The results shows that joint training with my best LEN and ODN makes 3D Layout IoU worsen. However it's slightly better than the downloaded Total3D which is close to the results of the paper.

Also, It seems that only IoU is provided in the testing results. How can I get the mean absolute error of cam pose, mAP of 3D detection, object translation, rotation and scale errors same as the paper?

[yinyunie]

Hi,

For the ODN and LEN training, SUN-RGBD is of a small-scale dataset (~5000 images for training), it indeed overfits with training too many epochs, but we did not see such a serious overfitting problem. We adopt similar learning targets and evaluation metrics as https://github.com/thusiyuan/cooperative_scene_parsing. There would not be too much difference in the test curves, I think you could also have a check.

[chengzhag]

I'll check it out. Thanks a lot!

[chengzhag]

I tried the layout_evaluate code from cooperative_scene_parsing on the test results of the downloaded checkpoint with essential changes to path settings. However, I got 0 IoU from every sample. Are there any details that I missed? Is it possible to opensource the full evaluation code of the paper?

[Wi-sc]

Hi,

I also have some problems about the training. I tried to use downloaded pretrained models and my pretrained models to initialize the framework before joint training, but the results decrease both. Can you help me check the config file?

Downloaded pretrained models to initialize joint training:
layout_iou: 0.440098
iou_3d: 0.109684
iou_2d: 0.594273
pitch_mae: 6.258617
roll_mae: 6.970254

My pretrained models to initialize joint training:
layout_iou: 0.456693
iou_3d: 0.118330
iou_2d: 0.599362

This is my config setting:

method: TOTAL3D
resume: False
finetune: True
weight: ['out/pretrained_models/pretrained_model.pth', 'out/pretrained_models/meshnet_model.pth']
seed: 123
device:
use_gpu: True
num_workers: 2
data:
dataset: sunrgbd
split: data/sunrgbd/splits
tmn_subnetworks: 2
face_samples: 1
with_edge_classifier: True
model:
layout_estimation:
method: PoseNet
loss: PoseLoss
object_detection:
method: Bdb3DNet
loss: DetLoss
mesh_reconstruction:
method: DensTMNet
loss: ReconLoss
optimizer:
method: Adam
lr: 1e-4
betas: [0.9, 0.999]
eps: 1e-08
weight_decay: 1e-04
scheduler:
patience: 5
factor: 0.5
threshold: 0.01
train:
epochs: 200
phase: 'joint' # 'layout_estimation' or 'object_detection' or 'joint'.
freeze: ['mesh_reconstruction']
batch_size: 3
test:
phase: 'joint'
batch_size: 3
demo:
phase: 'joint'
log:
vis_path: visualization
save_results: True
vis_step: 100
print_step: 50
path: out/total3d

[longbowzhang]

Hi @chengzhag,

Sorry to bother you. I am also trying to reproduce the results presented in the paper.
Now I am working on the first stage, which is training LEN and ODN individually. But I fail to reproduce the results. For example, '3D Layout' in table 1 says 57.6 w/o joint training. But the highest value I can get is 48.9. Therefore, I am wondering whether you could kindly share your config. file?

Best,
Longbow