Missing key "subject_encoder" in the loaded_state_dicts from pretrained model "epoch_8.pth"

[MW-ZHOU]

Hi,

I am running the file rel_test_full.py to do some testing, but I got a KeyError: "subject_encoder" when accessing the loaded_state_dicts which is loaded from the pre-trained model epoch_8.pth, the variable save_work_dir is set to be
work_dirs/relation/rel_ips_transformer, which contains the model epoch_8.pth

as far as I can tell, we are supposed to load the model from epoch_8.pth which is under the folder "mask2former_r50_ips"

Can you please help me with this? Please point out any mistakes that I have made. Thanks

[Jingkang50]

No, the model for rel_test are the second stage models, you can find them in the ckpt table in readme, such as the link below vanilla, filter, conv, and transformer

[MW-ZHOU]

So I need to load the model from one of these latest.pth file under the relation folder, if I understand you correctly?

[MW-ZHOU]

The links below vanilla, filter, conv, and transformer all go to the same webpage.

[Jingkang50]

Please follow the entire process here: https://github.com/LilyDaytoy/OpenPVSG?tab=readme-ov-file#training-and-testing

Basically if you need test, you should save feature tube from the first stage with epoch_8.ckpt and run test with any provided models in the directory you mentioned.

[MW-ZHOU]

Can I skip the "Train IPS" and start with "Tracking and save query features"?

Because I am currently also running prepare_query_tube_ips.py, which is used to save the feature tubes as you said, right? And I think it uses the epoch_8.pth, which is generated in the first stage.

[Jingkang50]

Yes, make sense to me. If you want to try quickly to know how it works, you could shrink the video list here:

OpenPVSG/tools/prepare_query_tube_ips.py

Line 238 in 4aee63b

for video_name in video_names:

[MW-ZHOU]

Thanks a lot for your time and help.

[Jingkang50]

No problem! Thank you for your interest in the project!

[MW-ZHOU]

One more question, I want to use the trained model from stage two to generate scene graph and extract relation features for another dataset, but this dataset does not have an annotation file like pvsg.json. Is it possible to do this? Because as what I can see, pvsg.json is used to generate feature tube and is used in every step for both training and testing.

[Jingkang50]

I think you might need to rewrite some dataset class (ref. here). For inference, the pvsg.json only provides the video name, so I think you can modify some code for your task.

[MW-ZHOU]

Hi,
I wonder how to generate relation predictions for an entire video like you did in the image on the top of the page. Because when I run rel_test_full.py, it only generates one prediction for the video "0004_11566980553". What mistake did I make?
What am I missing?

[MW-ZHOU]

No worries, I have found the python file to do this.

[MW-ZHOU]

Hi,

I have a question about using the code from PVSGAnnotation. What does xxxxxx mean in the link? How do I use this link? Can you please help me with this?

[Jingkang50]

Oh yes, xxxx means a job id in superannotate.
Here is an example you can play with.

We can arrange a zoom chat after eccv, like this weekend if you are interested in.

[MW-ZHOU]

Sure, that would be great. How about I connect with you via email for later communication?