I made a WKWebView-based renderer on macOS

[heshuimu]

https://github.com/heshuimu/RemotionWebKitRenderer

This is a hack project I put together. I found that using Remotion's own built-in rendering with Chromium instances, the rendering speed was not up to frame rate, and due to memory usage, I can only enable --concurrency 2.

With this tool displaying a WKWebView on screen, using 1 view alone significantly reduced rendering time. For example, on my MacBook Pro (Late 2019, Intel i9, Radeon 5500M), rendering using provided Chromium takes 83 seconds plus 4 seconds of encoding time. My tool took 20 seconds, doing both rendering and encoding at the same time.

https://www.youtube.com/watch?v=luR654eyqJo

One bonus is that my tool skips writing rendered image to disk by encoding the capture directly with AVFoundation features, which should perform better for longer videos and have better capture quality before encoding (Remotion writes the capture as JPEG).

The ugly part is that the view has to be on screen or JavaScript won't run, so there has to be a window on screen displaying the content.

[JonnyBurger]

@heshuimu Wow! This is really special! I also wanted to get into this at some point, but never knew how to tackle it!

I'd love to merge and integrate this officially into Remotion. Since we also support audio composition, I would still use FFMPEG for that, and try to bring back the concurrency option!

This would be the first foray into custom renderers, and a good start to later support more complex stuff like custom Blink builds that can run in headless mode on all operating systems. This would enable many opportunities in the long run!

[heshuimu]

Glad that my tool could be useful to the main project!

I have some improvements to do (here's one linked below). Feel free to open issues in my repo for more things to improve.
heshuimu/RemotionWebKitRenderer#2

It would not be hard for me to add simple audio support into the tool during the AVAssetWriter routine, so long as the supported use case is simple, i.e. only 1 audio track for the entire duration of the composition. But I agree that using FFmpeg is much easier for that.

I think the issue with Concurrency option will be the memory usage. I have been using the Audiogram composition for testing, and while the app process have very low memory usage, the WebKit process peaked at nearly 2GB. I haven't checked what would happen if WebKit will still try to run if memory pressure is on. I'd like to find some ways to detect it so that It can have the maximum amount of concurrency enabled. But with the current performance, I'm think the tool is still much more efficient without the concurrency.

[JonnyBurger]

Definitely! We see some problems with the default Chrome renderer and want to have some options.

Since the audio pipeline is very complex (mixing multiple audio tracks, setting volume frame-by-frame), I'd rather keep the FFMPEG codebase for it, since that works on all 3 OS's we support.

Don't worry about the memory usage - I think it's because of the audio template. I cannot imagine WKWebView taking more memory than a full Chrome instance.

I'll work on some other things on our roadmap first, but will explore adding support for other renderers in the future!
Also, I'll be working on building Chromium from source and seeing if we can provide optimized binaries for Remotion users.