Streaming audio and video in sync for mp4 container using Gstreamer framework

Question

I am new to gstreamer and I want to stream a mp4 video which is having audio and video both from my Host(Ubuntu PC) to target board. I have successfully streamed only video. But streaming both(audio and video) is not happening.

Pipeline to stream only video is as below

Sender:

gst-launch-1.0 -v filesrc location = video.mp4 ! decodebin ! x264enc ! rtph264pay ! udpsink host=192.168.2.136 port=5000

receiver:

gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96, ssrc=3394826012, timestamp-offset=2215812541, seqnum-offset=46353" ! rtph264depay ! avdec_h264 ! videoconvert ! ximagesink sync=false

Hence, can see video on target.

Did streaming of only audio from mp4 file.

Pipeline to stream only video is as below

Sender:

gst-launch-1.0 -v filesrc location=video.mp4 ! qtdemux name=demuxer demuxer. ! queue ! rtpmp4apay pt=97 ! udpsink host=192.168.2.69 port=5000

Receiver:

gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=3386264266, timestamp-offset=2822202855, seqnum-offset=17719" ! rtpmp4adepay ! decodebin ! alsasink &

This too runs successfully.

But when I tried below pipeline to stream both audio/video in sync:

Sender:

gst-launch-1.0 -v filesrc location=sample.mp4 ! qtdemux name=demux demux. ! queue ! rtph264pay pt=96 ! udpsink host=192.168.3.101 port=5000 demux. ! queue ! rtpmp4apay pt=97 ! udpsink host=192.168.3.101 port=5001

Receiver:

gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96, ssrc=2226494377, timestamp-offset=3242004369, seqnum-offset=17021" ! rtph264depay ! decodebin ! ximagesink udpsrc port=5001 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=1499706502, timestamp-offset=190741668, seqnum-offset=24774" ! rtpmp4adepay ! decodebin ! alsasink

But: get an output as below:

ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin1/GstFaad:faad0: Could not decode stream.

fails. Please can someone suggest me a pipeline which can stream audio/video both on target device in sync.

Thanks in advance.

Answer 1

There seem to be issues with AAC in RTP as well as other RTP payloaders in gstreamer.

As mentioned in the other answer, it is anyway the best strategy to not demux and split video and audio if it is desired to have a synchronized playback on the receiver side. Additionally, a container format that is streamable can also improve overall functionality.

My proposal is therefore to remux the data streams into MPEG2-TS and transport that via RTP. This has several advantages, like good re-synchronization after packet loss and that A/V always stays sync.

Example

This works for me with gstreamer 1.2.4:

Sender

gst-launch-1.0 -v filesrc location=/foo/bar.mp4 ! \
  qtdemux name=demux demux. ! h264parse config-interval=10 ! queue ! mux. \
  mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=10.0.0.1 port=5003 \
  demux. ! aacparse ! queue ! mux.

Receiver (file)

gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" ! \
  rtpmp2tdepay ! tsparse !  filesink location=/foo/bar.ts

Receiver (output)

This command is based on the commands in the question, could not test it with ALSA:

gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" ! \
  rtpmp2tdepay ! tsparse ! tsdemux name=demux \
  demux. ! queue ! audioconvert ! alsasink \
  demux. ! queue ! xvimagesink

Answer 2

There is no easy way to stream in sync with gstreamer if you demux the video and audio stream on the sender side. Timestamp information is kept in the container (MP4 in this case), which is lost when demuxing.

Try to send the MP4 over to the receiver, and then demux there. A basic example may be found here: Streaming Mp4 video through RTP protocol using Gstreamer in Ubuntu

Answer 3

Sender

gst-launch-1.0 filesrc location= ~/1.mp4 ! qtdemux name=d ! queue ! h264parse ! rtph264pay config-interval=10 pt=96 ! udpsink host=10.24.217.30 port=5000 d. ! queue ! aacparse ! rtpmp4apay pt=97 ! udpsink host=10.24.217.30 port=5001 -v

Receiver

gst-launch-1.0 udpsrc port=5000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, payload=(int)96, encoding-name=(string)H264" ! rtph264depay ! identity silent=0 ! avdec_h264 ! videoconvert ! ximagesink sync=0 async=0 udpsrc port=5001 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=1499706502, timestamp-offset=190741668, seqnum-offset=24774" ! rtpmp4adepay ! avdec_aac ! identity silent=0 ! alsasink sync=0 async=0

This works properly.

Note: Starting receiver early before starting sender gives error at the receiver end when sender is started. ** (gst-launch-1.0:26959): CRITICAL **: 15:15:51.191: gst_audio_decoder_set_output_format: assertion 'GST_AUDIO_INFO_IS_VALID (info)' failed ERROR: from element /GstPipeline:pipeline0/avdec_aac:avdec_aac0: GStreamer error: negotiation problem. Additional debug info: gstavauddec.c(449): gst_ffmpegauddec_negotiate (): /GstPipeline:pipeline0/avdec_aac:avdec_aac0: Could not set caps for libav decoder (aac), not fixed?

This could be a bug in GST in caps negotiations. WAR is to start receiver after sender.