Support for Video with Qwen2-VL

[tmoroney]

It would be really great if support for Video could be added to Qwen2-VL as it seems to only support images at the moment. I am working on a project that would seriously benefit from it.

[Blaizzy]

Hey @tmoroney

Indeed, it's a cool feature of Qwen2-VL. I will work on adding it after #41 and Molmo port.

Could you share a little bit more about your project?

[Blaizzy]

I got multi-image support working which is pretty close to video.

From here to video it’s pretty close but I want to make the API seamless for all models, that’s why it will take a bit longer.

https://x.com/Prince_Canuma/status/1841634911825858978

[tmoroney]

Hey @tmoroney

Indeed, it's a cool feature of Qwen2-VL. I will work on adding it after #41 and Molmo port.

Could you share a little bit more about your project?

Essentially the project is Copilot for video editors. As part of my masters in computer science I am researching how to create an AI video editing assistant that will suggest the next shot as you edit based on the context of the story so far and the emotional tone etc, along with providing inspiration to remove creative blockers. I want to analyse all of the project's footage on device using small and efficient ML models, along with algorithms (Computer vision, sentiment analyses etc) whenever possible in order to reduce compute.

[tmoroney]

I got multi-image support working which is pretty close to video.

From here to video it’s pretty close but I want to make the API seamless for all models, that’s why it will take a bit longer.

https://x.com/Prince_Canuma/status/1841634911825858978

Amazing! Thanks for all your hard work :)

[anishjain123]

@Blaizzy any update on the video support? Love the work you've been doing!

[Blaizzy]

Thanks guys!

There is a PR for video support #97 it works but needs a bit of polishing

I will do that and merge it over the weekend.

[anishjain123]

you're a g @Blaizzy , its insane how much the memory usage gets on this since it parses it as an array of images. Im wondering if this is the right architecture for on device video processing

[Blaizzy]

Could you elaborate?

[andimarafioti]

SmolVLM should really help with that!