Cannot achieve same result in paper

[thanhhvnqb]

Dear Mr. Adrian Bulat,
I run your code based on the Nvidia-docker file you provided. But unfortunately, I cannot achieve the same result as you reported in the paper. I downloaded the model from your provided link.

So could you review and check are there any problems with the code and models?
Thank you so much and sorry for any inconvenience.

Here is my evaluation result
===300W-Testset-3D
600
('AUC: ', 0.6637857142857142)
===Menpo-3D
8955
('AUC: ', 0.42285235702321128)
===300VW-CatA
62643
('AUC: ', 0.52723026857407407)
===300VW-CatB
32872
('AUC: ', 0.55551620136981539)
===300VW-CatC
27245
('AUC: ', 0.26830873292609392)

Here is the link to download the result of 3D-FAN that I generate
3dFAN_result.zip
Here is my modified code.
3dFAN.zip

[1adrianb]

While unfortunately I don't have the time to look thru your code line-by-line at the moment, my comments are as follow (will have a look at a later point):

• The demo is setup to mainly reproduce the results on 300-W and Menpo, for 300VW we used a separate model which was also finetuned on 300VW-testset for the 2D case. The main reason for this is that the videos contain a different type of noise (ex: motion blur), one not found in static images. Unfortunately the fine-tuned models are not available online at the moment.
• Make sure you are not using dlib to detect faces. Dlib was included only for simplicity. One will not be able to achieve the same results with it.
• In the original code there is a particular shift added to the predictions, due to the ground truth been 0 or 1 based, this should not be the case for the case for the 3D points, which should be consistent (no shift needed for them).
• The model provided should be able to reproduce the exact numerical results for Menpo-3D and 300W-3D, however since the model that produced the results on 300VW is slightly different you will most likely get worse performance on them using the available model.

All in all, to conclude, the provided model should be able to reproduce the exact numerical results on 300W and Menpo (both 2D&3D), however you will need to retrain a bit on video frames the models to reproduce the 300VW results.

[thanhhvnqb]

Thank you for your quick response.
So could you send me the .t7 files, which are the predicted results of 3DFAN on LS3D-W dataset? I want to analyze them.
Additionally, If it is OK, could you send me the predicted result of 3DDFA also? I can not run their code correctly.
Or at least, send me the 3D numerical results on 300W, Menpo, 300VW-CatA/B/C. of 3DFAN and 3DDFA? I cannot find the number in the paper.
Thank you so much again for your help.

[1adrianb]

@thanhhvnqb I have run the script available online and I got 300W-3D: 81.1% Menpo-3D: 72.5% I am now away but I can send you this particular predictions obtained using this code on the begging on the next week.
Regarding the rest, this may take a while, all of that data is on my university machine, however I am not at the university anymore and such it will take some time to get a hand on it and search thru all of that mess that accumulated in time :D

[thanhhvnqb]

Oh. That's quite strang. I also run the online code but cannot achieve the same result. I run by using the nvidia-docker of this repository. So could you send me the code you use to run? Maybe I can check and find whether my configuration is right or not.
Additionally, could you send me the result of 3DDFA also, at least the numerical result?
Thank you so much for your help.