(RuntimeError) Cannot find default RTP payload type for encoding AAC

[jongirard]

Discussed this on Discord, apparently a commit needs to be reverted in order for the latest version of membrane_rtp_aac_plugin to work.

Unfortunately, even if you fix this problem we have to revert one commit in AAC.Depayloader: https://github.com/membraneframework/membrane_rtp_aac_plugin/pull/6/files

00:21:11.384 [error] GenServer #PID<0.1084.0> terminating
** (RuntimeError) Cannot find default RTP payload type for encoding AAC
    (membrane_rtp_plugin 0.5.0) lib/membrane/rtp/session_bin.ex:567: Membrane.RTP.SessionBin.get_output_payload_type!/2
    (membrane_rtp_plugin 0.5.0) lib/membrane/rtp/session_bin.ex:341: Membrane.RTP.SessionBin.handle_pad_added/3

[mickel8]

Sorry so busy in last days. Please take a look at
membraneframework/membrane_demo#94
and
#9

You can try to run version from the first PR but it may still have some problems.

[jongirard]

@mickel8 thanks 👍🏻 I updated deps for membrane_rtp_aac_plugin from master branch to get the latest commit but it seems like the problem still persists, still with error

(RuntimeError) Cannot find default RTP payload type for encoding AAC

[mickel8]

@jongirard We have merged a new version of rtp_to_hls (membraneframework/membrane_demo#94) demo and it should work pretty well I think. Could you please send your pipeline code once again?

[jongirard]

@mickel8 basically following the demo (other than using Membrane.Bin) https://gist.github.com/jongirard/9d0784172c4109d14f76647e1b40d247

There is a slightly new error now, with ** (RuntimeError) Cannot find payloader for encoding AAC

I can add

      rtp: %Membrane.RTP.SessionBin{
        custom_payloaders: %{:AAC => Membrane.MP4.Payloader.AAC}
      },

to the RTP SessionBin, which makes the error go away, but then it complains of invalid caps.

13:24:01.115 [error] GenServer #PID<0.1042.0> terminating
** (stop) exited in: :cannot_handle_message.invalid_caps({:message, {Membrane.Core.Message, :caps, %Membrane.AAC{channels: 2, encapsulation: :ADTS, frames_per_buffer: 1, mpeg_version: 2, profile: :LC, sample_rate: 48000, samples_per_frame: 1024}, [for_pad: :input]}}, {:mode, :info})
    ** (EXIT) :error
Last message: {Membrane.Core.Message, :caps, %Membrane.AAC{channels: 2, encapsulation: :ADTS, frames_per_buffer: 1, mpeg_version: 2, profile: :LC, sample_rate: 48000, samples_per_frame: 1024}, [for_pad: :input]}

13:38:45.910 [warn]  [:payloader] Terminating element possibly not prepared for termination as it was in state :playing.
Reason: {:error, {:cannot_handle_message, :invalid_caps, [message: {Membrane.Core.Message, :caps, %Membrane.AAC{channels: 2, encapsulation: :ADTS, frames_per_buffer: 1, mpeg_version: 2, profile: :LC, sample_rate: 48000, samples_per_frame: 1024}, [for_pad: :input]}, mode: :info]}}",
State: %Membrane.Core.Element.State{
  controlling_pid: #PID<0.1126.0>,
  delayed_demands: %{},
  internal_state: %{avg_bit_rate: 0, max_bit_rate: 0},
  module: Membrane.MP4.Payloader.AAC,
  name: :payloader,
  pads: %{
    data: %{
      input: %Membrane.Pad.Data{
        accepted_caps: {Membrane.AAC, [encapsulation: :none]},
        availability: :always,

The payloader is set in the notification as in the demo so I'm not sure why it says it can't find the payloader.

[mickel8]

Oh I didn't see it before, my bad. The problem is that we don't have AAC payloader so its impossible to create RTP stream containing AAC payload at this moment :(

[mickel8]

We only have AAC depayloader and rtp_to_hls demo use it but it doesn't use payloader

[jongirard]

Oh I see.. 🙂 I guess I'm a little confused then, because there is no MP4 muxer either I thought I needed to "manually" get the .h264/.aac file extractions via ffmpeg in order to use in my flow of file source -> rtp -> realtimer -> hls. Is there some other audio format I could use or some way to get this working?

I see there is an RTP.Opus.Payloader but there is no Membrane.MP4.Payloader.OPUS plugin. For my flow (HLS) I think it needs to be containerized into CMAF.

Could it be possible to go opus (parser) -> Opus Payloader -> RTP -> Membrane.RTP.AAC.Depayloader -> Membrane.MP4.Payloader.AAC?

[jongirard]

Could something like this work? Basically parsing OPUS input file, through RTP, decode OPUS to Membrane.Caps.Audio.Raw then use Membrane.AAC.FDK.Encoder to convert raw to AAC?

def handle_init(options) do
      audio_parser: Membrane.Opus.Parser,
      rtp: Membrane.RTP.SessionBin,
      ....
end

links = [
      link_bin_input(:audio_input, buffer: [fail_size: 20_000])
      |> to(:audio_parser)
      |> via_in(audio_ip, buffer: [fail_size: 800])
      |> to(:rtp)
      |> via_out(Pad.ref(:rtp_output, audio_ssrc), options: [encoding: :OPUS])
      |> to(:audio_realtimer)
      |> via_in(audio_rtp_ip, buffer: [fail_size: 800])
      |> to(:rtp)
    ]

@impl true
  def handle_notification({:new_rtp_stream, ssrc, 120}, :rtp, _ctx, state) do
    children = %{
      opus_decoder: Membrane.Opus.Decoder,
      fdk_aac: Membrane.AAC.FDK.Encoder,
      audio_payloader: Membrane.MP4.Payloader.AAC,
      audio_cmaf_muxer: Membrane.MP4.CMAF.Muxer
    }

    links = [
      link(:rtp)
      |> via_out(Pad.ref(:output, ssrc))
      |> to(:opus_decoder)
      |> to(:fdk_aac)
      |> to(:audio_payloader)
      |> to(:audio_cmaf_muxer)
      |> via_in(:input)
      |> to(:hls)
    ]
    ....

If so, the only thing that's going wrong is "corrupted stream" from the membrane opus decoder.

** (stop) "corrupted stream"
    (membrane_opus_plugin 0.2.1) Membrane.Opus.Decoder.Native.Nif.unifex_decode_packet(#Reference<0.2202285139.1524236295.189824>, <<79, 103, 103, 83, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 182, 219, 5, 22, 0, 0, 0, 0, 14, 14, 137, 132, 1, 19, 79, 112, 117, 115, 72, 101, 97, 100, 1, 2, 56, 1, 128, 187, 0, 0, 0, 0, 0, 79, 103, 103, ...>>)
    (membrane_opus_plugin 0.2.1) lib/membrane_opus/decoder.ex:70: Membrane.Opus.Decoder.handle_process/4

[mickel8]

Could it be possible to go opus (parser) -> Opus Payloader -> RTP -> Membrane.RTP.AAC.Depayloader -> Membrane.MP4.Payloader.AAC?

Why do we want to combine Opus elements with AAC elements? If audio is compressed with Opus, encapsulated in RTP packets with Opus Payloader then it should be extracted from RTP packets with Opus Depayloader.

Could something like this work? Basically parsing OPUS input file, through RTP, decode OPUS to Membrane.Caps.Audio.Raw then use Membrane.AAC.FDK.Encoder to convert raw to AAC?

I think that it might be possible 😉

** (stop) "corrupted stream"
(membrane_opus_plugin 0.2.1) Membrane.Opus.Decoder.Native.Nif.unifex_decode_packet(#Reference<0.2202285139.1524236295.189824>, <<79, 103, 103, 83, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 182, 219, 5, 22, 0, 0, 0, 0, 14, 14, 137, 132, 1, 19, 79, 112, 117, 115, 72, 101, 97, 100, 1, 2, 56, 1, 128, 187, 0, 0, 0, 0, 0, 79, 103, 103, ...>>)
(membrane_opus_plugin 0.2.1) lib/membrane_opus/decoder.ex:70: Membrane.Opus.Decoder.handle_process/4

Are you sure you have Opus stream not AAC one? We have similar example here https://github.com/membraneframework/membrane_demo/blob/master/rtp/lib/membrane_demo_rtp/receive_pipeline.ex#L73-L75 and we do it in the same way you did 🤔

[jongirard]

Are you sure you have Opus stream not AAC one?

Indeed, I'm using ffmpeg -i caminandes_llamigos_1080p.mp4 -c:a libopus -b:a 192K caminandes_llamigos_1080p.opus to encode the opus file.

Interestingly, I tried using the "beep.opus" as used in the rtp send_pipeline demo which seems to make it past the decoder, only to fail due to a CMAF issue in the last step.. any ideas how to fix that? I see that Membrane.RTP.Timestamper was used in a demo, but it seems like it doesn't exist in the new membrane_rtp_plugin version.

00:38:26.803 [error] GenServer #PID<0.593.0> terminating
** (KeyError) key :timestamp not found in: %{}
    (membrane_mp4_plugin 0.4.0) lib/membrane_mp4/cmaf/muxer.ex:50: Membrane.MP4.CMAF.Muxer.handle_process/4

00:38:26.802 [warn]  [:audio_cmaf_muxer] Terminating element possibly not prepared for termination as it was in state :playing.
Reason: {{:badkey, :timestamp, %{}}, [{Membrane.MP4.CMAF.Muxer, :handle_process, 4, [file: 'lib/membrane_mp4/cmaf/muxer.ex', line: 50]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: 'lib/membrane/core/callback_handler.ex', line: 128]}, {Membrane.Core.CallbackHandler, :"-exec_and_handle_splitted_callback/6-fun-1-", 7, [file: 'lib/membrane/core/callback_handler.ex', line: 97]}, {Bunch.Enum, :"-try_reduce_while/3-fun-0-", 3, [file: 'lib/bunch/enum.ex', line: 159]}, {Enumerable.List, :reduce, 3, [file: 'lib/enum.ex', line: 3764]}, {Enum, :reduce_while, 3, [file: 'lib/enum.ex', line: 2231]}, {Membrane.Core.Element.ActionHandler, :handle_action, 4, [file: 'lib/membrane/core/element/action_handler.ex', line: 24]}, {Bunch.Enum, :"-try_reduce/3-fun-0-", 3, [file: 'lib/bunch/enum.ex', line: 121]}]}",
State: %Membrane.Core.Element.State{
  controlling_pid: #PID<0.541.0>,
  delayed_demands: %{},
  internal_state: %{
    elapsed_time: 0,
    samples: [],
    segment_duration: 2000000000,
    seq_num: 0
  },
  module: Membrane.MP4.CMAF.Muxer,

As for why my own opus file was claiming to be a "corrupted stream", maybe it's a problem with the ffmpeg encode of my opus file?

[mickel8]

As for why my own opus file was claiming to be a "corrupted stream", maybe it's a problem with the ffmpeg encode of my opus file?

This can be possible but I don't have any ideas what you could set differently while encoding your audio.

Anyway, beep.opus seems to work pretty well except MP4 is not able to mux it with video into mp4 file right?

@mat-hek do you have any ideas?
As @jongirard pointed out it seems that there is no timestamp key.

[mickel8]

The only thing I can see is that you are setting audio bitrate to 192k by -b:a 192K. Maybe this should be changed 🤔

[mat-hek]

@jongirard It seems you shouldn't parse Opus after getting it from RTP, as RTP contains a non-delimited Opus stream. Check the audio part of https://github.com/membraneframework/membrane_demo/blob/webrtc_to_hls/webrtc_to_hls/lib/webrtc_to_hls/pipeline.ex , the WebRTC Endpoint bin uses RTP bin underneath, so the output is the same as from RTP.