How to tune hyperparameters for custom datasets?

[krumo]

Thanks for your excellent work! I am trying to run SIGMA and SIGMA++ on a new UDA benchmark for adapting Cityscapes to ACDC dataset. I find for different UDA tasks, you adopt different GA_DIS_LAMBDA, GRL_WEIGHT_{P3-P7}, MATCHING_LOSS_WEIGHT and BG_RATIO in configs. Could you share some insights about how to tune these hyperparameters for a new UDA task? Looking forward to your response!

[wymanCV]

I am so glad that you can pay attention to our work! Honestly, compared with your excellent work DAFRCNN, this FCOS-based (or EPM-based) framework has so many imperfections.

We find that GRL_WEIGHT_{P3-P7} are sensitive to the dataset. Except for City-to-Foggy, it seems that a relatively larger value is required for all the other settings. Otherwise, the discriminator may beat the backbone and give a small adv loss, as shown here. For example. in KITTI and Pascal settings, I tried GRL_WEIGHT_ in the range of 0.01 to 1.0 and found the results would make a huge difference, which also gave me a headache (adv almost does not work in KITTI with 0.01). The larger domain gap (especially for the style-based gap) and the imperfect discriminator design for dense detectors may be the reason.

For GA_DIS_LAMBDA , if the adv loss is normal but the performance is significantly lower than other detector baselines, it may need to enlarge GA_DIS_LAMBDA to improve the adv intensity.

For BG_RATIO, we set it as (num_cls+1), which will sample 1/(num_cls+1) background points as the negative node. The MATCHING_LOSS_WEIGHT is robust for different datasets, which can be set as default 0.1 for other datasets.

I am also looking forward to seeing new works that can address these imperfections since I like the FCOS simple architecture. However, it seems that the current trend in this field is turning to Mean-Teacher and DETR-based architecture. Haha~