video-audio drift

[hholst80]

I noticed that I get audio/video drift (they do not sync tight) on wf-recorder. I noticed I have the same issue with wl-screenrec.

Is there any remedies I can take to avoid the lag/drift that appears? It usually takes 5min+ for the drift to be noticeable and I think it says <1sec. But even a few hundred ms in drift starts to be noticeable and irritating. At one second I'd say the recording is broken.

I am using pulse audio and the options:

wl-screenrec --low-power=off --encode-resolution 1920x1080 --audio 

Output #0, mp4, to 'screenrecord.mp4':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), vaapi, 1920x1080, q=2-31, 40000 kb/s, 90k tbn
  Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s

ammen99/wf-recorder#241

[russelltg]

Thanks for the report. I can't seem to repro (made a 7:30 video). Is the audio behind or ahead?

Currently wl-screenrec assumes that it's being fed exactly sample_rate samples per second. If this isn't the case, the audio could end up behind the video.

It's also possible that the video and audio come from different clocks, in which case we will need something like -async 1, as noted on the wl-recorder issue.

[hholst80]

Is the audio behind or ahead?

I think it actually has been both ways, but when I noticed this time the audio was ahead of the video. My recording locally has the same drift as the YouTube asset below:

https://youtu.be/vsXXgenUCDY?t=1343

I suspect the drift is even more than one second at this point.

Currently wl-screenrec assumes that it's being fed exactly sample_rate samples per second.

I do not know the internals of how this works, but what happens if there is a dropped chunk of audio or video? Is there any compensation?

[russelltg]

Video is timestamped using the timestamp from the compositor (which compositor are you using btw?) And audio is timestamped using how many samples have been received from the beginning of the stream, so audio isn't robust to dropout. This isn't ideal and I'll post a patch here that you can test so see if that's the issue.

[hholst80]

which compositor are you using btw?

[russelltg]

Both sway and Hyprland (correctly, imo) sample from the monotonic clock, so that shouldn't be why I can't repro.

Can I assume you're using the pipewire pulseaudio implementation?

[hholst80]

No pipewire, only pulse audio.

[russelltg]

What does pacman -Q | grep pulse give

[hholst80]

I saw that the 'neofetch' command was from my desktop, but I have the exact same setup (famous last words, right) on both the laptop and the workstation.

Anyways:

[I] root@trump ~
# pacman -Q | grep pulse
libpulse 16.1-7
pulseaudio 16.1-7
pulseaudio-jack 16.1-7
[I] root@trump ~
# 

[russelltg]

Can you attempt installing pipewire-pulse (which most distros are shipping by default) instead of pulseaudio and see if it still repros?

(note that I still consider this a bug, but might be a ok workaround for now)

[hholst80]

I managed to install pipewire again (changed Window mananger to Hyprland) hoping that

1. it will not break firefox
2. it will fix wl-screenrec.
   ;-)

[hholst80]

I installed hyprland and the pipewire stack again. It seems to work fine with Firefox so far.

As an added bonus, I just recorder 45 minutes of minecraft with my son and we had no audio/video lag in the recording.

[russelltg]

Glad you have a workaround! I'll look into adding a clock filter+resampler. Fixing this will not be trivial so may not be soon

[hholst80]

Closing this not to bulk up on your tidy Issue page. Drift when it happens is very irritating so any fundamental work to protect against this would be great to have.

Thanks for the great work you put into this fine tool!

[russelltg]

Appreciate the sentiment but it is an issue and I'd like to keep it open for anyone else who runs into it and so I don't forget it exists.

[vnuxa]

im experiencing the same issue, though with nixos
but the video is lacking behind the audio as it seems, i use wl-recorder in the background and it starts whenever i boot up my computer on hyprland with these options

wl-screenrec -f ~/Videos/clips/screenrec_$(date +%y-%m-%d_%h-%m-%s).mp4 --codec avc --history 45 --audio --audio-device clipping-tunnel.monitor -b "0.65 MB" --encode-resolution 1280x720 &

and im using pipewire
apparently the drift is really noticable when my system has been on for a while

for example this clip which i took yesterday from playing gmod, the desync is like 4 seconds
https://github.com/user-attachments/assets/f28796ab-a31a-48a6-a0b1-dc08dd9f005d

[russelltg]

Can anyone easily repro this? I can't seem to, but I think I have a fix... @pparaxan @hholst80 @vnuxa

diff --git a/src/audio.rs b/src/audio.rs
index ec6a49a..354a46d 100644
--- a/src/audio.rs
+++ b/src/audio.rs
@@ -387,7 +387,7 @@ fn audio_filter(
         .input("out", 0)
         .unwrap()
         .parse(&format!(
-            "aformat=sample_rates={}:sample_fmts={}:channel_layouts={:#x}",
+            "aresample=async=1,aformat=sample_rates={}:sample_fmts={}:channel_layouts={:#x}",
             codec_sample_rate,
             codec_sample_format.name(),
             codec_channel_layout.bits(),

can anyone who can repro before that diff confirm if they can repro after?

[pparaxan]

Can anyone easily repro this? I can't seem to

Huh... Well that's odd.
I can't repro it myself now...

[pparaxan]

update: can repro, testing the fix now...

[pparaxan]

Just recorded a video playing a game, the audio unfortunately is ahead of the video even with the patch you gave.

My theory is that pipewire/pulseaudio itself is the issue Tested it with OBS and the audio and video both are in synced.

[russelltg]

OBS is probably doing something smart here, like ffmpeg can under the hood (but clearly I setup wrong :) ).

Do you use pipewire or pulseaudio? (you can check with ps aux | grep pulse What is your full commandline? What compositor? How long does it take to be noticeable?

[pparaxan]

Do you use pipewire or pulseaudio?

pipewire[-pulse]

What is your full commandline?

wl-screenrec --audio --audio-device alsa_output.pci-0000_00_1f.3.analog-stereo.monitor --low-power=off --bitrate=2MB --output=eDP-1 --codec=hevc --filename=$(date +%Y-%m-%d_%H-%m-%s).mp4

What compositor?

Sway

How long does it take to be noticeable?

2 minutes

[pparaxan]

(I hope that's what you meant by commandline lol)

[pparaxan]

Still have the patch applied-
Just recorded another video because of boredom in the same game and now the audio is synced?

seems to be an on and off thing, ¯\_(ツ)_/¯