Audio support?

[kevzettler]

Maybe I missed this but is there currently support for audio? The diablo demo has no sound. I'm using the Brave browser.

[giongto35]

Yes, Audio is not supported right now : D I need to research how to capture audio in headless machines but don't have time to work on that.

[Rock-N-Troll]

Possible via FFMPEG with Linux PulseAudio
https://trac.ffmpeg.org/wiki/Capture/PulseAudio

Or without FFMPEG as mentioned here:
https://superuser.com/questions/1339038/how-can-i-can-create-a-virtual-pulseaudio-soundcard-on-linux-with-no-audio-hardw

That StackOverflow article also shows PulseAudio can stream over RTP which could prove useful:
https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/#index7h2

[giongto35]

Hi @Rock-N-Troll ,
Thanks very much for the resource. The hardest challenge is to capture it when there is no audio hardware. Seems like it's not so complicated with Pulse Audio.
Currently, Im prioritizing to run more applications on the platform. It will be very nice if you can contribute on the Audio; or an example about how to capture and stream out the audio at one port is also very nice.

[Rock-N-Troll]

@giongto35
https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/RTP/

"Stream it with vlc as mp3 for low bandwidth" section (demonstrates on-the-fly transcoding to mp3)

load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 description="RTP Multicast Sink"
load-module module-rtp-send source=rtp.monitor destination=127.0.0.1 port=46998 loop=1

Log off and back on or reboot to enable the changes 3. Now you can transcode the incomming RTP/s16be stream with vlc and send the RTP/mp3 stream to a multicast address

cvlc --ipv4 rtp://@127.0.0.1:46998 ":sout=#transcode{acodec=mp3,ab=256,channels=2}:duplicate{dst=rtp{dst=225.0.0.1,mux=ts,port=12345}}"
Playback the stream on any computer

vlc --ipv4 rtp://@225.0.0.1:12345

OR

https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/#index2h3

How can I use PulseAudio to stream music from my main PC to my LAN with multiple PCs with speakers?
On the sender side create an RTP sink:

load-module module-null-sink sink_name=rtp
load-module module-rtp-send source=rtp.monitor
set-default-sink rtp

This will make rtp the default sink, i.e. all applications will write to this virtual RTP device by default.

As you can see, it is very straightforward to send audio via RTP if you use PulseAudio and point it to the correct application as the "sink" or the whole Linux environment as the "sink".

Potentially can look at WebRTC Audio Processing code for Linux. Seems more complicated: https://freedesktop.org/software/pulseaudio/webrtc-audio-processing/

[giongto35]

Just research around today. Seems really promising with PulseAudio. Thanks very much. It helps me a lot indeed! Will try to integrate soon.

[Rock-N-Troll]

Glad to help :).

I am curious how to synchronize audio and video via pulse audio and xvfb in a ffmpeg stream. I see you already handled audio with cloud retro so maybe I should see how you handled it there.

I'm not positive, but I'm guessing you can either combine the audio into the current ffmpeg video stream or you can send the audio and video streams separately to the client, but I think that will require more engineering. I'll see what else I can contribute down the line.

[giongto35]

@Rock-N-Troll #44 Added a simple implementation here, it costs my weekend.
It works but there is an expected issue that video and audio are not in sync. Will think about it later.
Thanks for giving guides and encouraging me to do it.

[Rock-N-Troll]

https://trac.ffmpeg.org/wiki/StreamingGuide

maybe try:
https://stackoverflow.com/questions/16658873/how-to-minimize-the-delay-in-a-live-streaming-with-ffmpeg
-fflags nobuffer
-flags low_delay

(and maybe less important but try a smaller probesize)
-probesize 32

https://ffmpeg.org/ffmpeg-formats.html

can also maybe try alsa instead of pulse https://superuser.com/questions/144648/how-do-alsa-and-pulseaudio-relate
-f alsa

I think using ffplay to test the delay is the best option to see if it's some part of the stream or the actual audio delay of the whole system:
https://stackoverflow.com/questions/64488707/how-to-solve-the-incremental-delay-in-ffplay-streaming

[Rock-N-Troll]

Also, forgot to say how cool that is you were able to get it working!

[sergystepanov]

GJ but one small remark.
If you want to use RTP from FFmpeg make sure:

• it generates proper timestamps (delays) for varying video frame times (use_wallclock_as_timestamps?),
• it captures and passes ~20ms audio frames into the Opus encoder without any additional bufferization.
  That should be enough for some basic AV-sync with RTP/RTCP/NTP.

Sync RTFM.