Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement modnet + matting on stage 1 training #25

Open
johndpope opened this issue Nov 5, 2024 · 1 comment
Open

implement modnet + matting on stage 1 training #25

johndpope opened this issue Nov 5, 2024 · 1 comment

Comments

@johndpope
Copy link
Owner

johndpope commented Nov 5, 2024

known issue

1x single video
https://wandb.ai/snoozie/megaportraits/runs/khl4e01g?nw=nwusersnoozie

4x videos
https://wandb.ai/snoozie/megaportraits/runs/zhos6yxx?nw=nwusersnoozie

looking at whether or not to follow megaportraits paper - to remove background - or provide model some extra weights for face masks like emo.....

or find code that's causing this - usually it's in a batch operation that corrupts this

https://github.com/johndpope/VASA-1-hack/blob/main/data/driving_video.json

UPDATE

so this paper specifically addresses id leakage - https://arxiv.org/abs/2404.19110

I look at the EMOPortraits code - its so full of args - I attempt to untangle it

https://github.com/johndpope/EMOPortraits/tree/feat/take2

reading this - it seems like they may have upgraded expression embedding with emo
https://byteaigc.github.io/X-Portrait2/

We ensure strong disentanglement of appearance and motion in training our expression encoder such that the encoder only pay attention to the expression-related information in the driving video. Our model can achieve strong cross-style and cross-domain expression transfer, covering both realistic portraits and cartoon images. This makes our model highly adaptive to a wide range of use cases including real-world storytelling, character animation, virtual agents and visual effects.

@johndpope
Copy link
Owner Author

i think i missed the modnet + matting which is causing this >1 (4x videos) generation bug.
its not the id leakage.
https://wandb.ai/snoozie/megaportraits/runs/zhos6yxx?nw=nwusersnoozie
johndpope/MegaPortrait-hack#59

@johndpope johndpope changed the title ID Leakage on stage 1 training implement modnet + matting on stage 1 training Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant