Synchronize two RTSP/RTP H264 video streams capture using GStreamer

Question

I have two AXIS IP cameras streaming H264 stream over RTSP/RTP. Both cameras are set to synchronize with same NTP server so I assume both cameras will have same exact clock (may be minor diff in ms).

In my application, both cameras are pointing to same view and its required to process both camera images of same time. Thus, I want to synchronize the image capture using GStreamer.

I have tried invoking two pipelines separately on different cmd prompts but the videos are 2-3 seconds apart .

1. gst-launch rtspsrc location=rtsp://192.168.16.136:554/live ! rtph264depay ! h264parse ! splitmuxsink max-size-time=100000000 location=cam1_video_%d.mp4
2. gst-launch rtspsrc location=rtsp://192.168.16.186:554/live ! rtph264depay ! h264parse ! splitmuxsink max-size-time=100000000 location=cam2_video_%d.mp4

Can someone suggest a gstreamer pipeline to synchronize both H264 streams and record them into separate video files?

Thanks! ARM

Answer 1

I am able to launch a pipeline using gst-launch as shown below. It shows good improvement on captured frame synchronization compare to lanuching two pipelines. Most times they differ by 0-500 msec. Though, I still want to synchronize them less than 150 msec accuracy.

rtspsrc location=rtsp://192.168.16.136:554/axis-media/media.amp?videocodec=h264 \
    ! rtph264depay ! h264parse \
    ! splitmuxsink max-size-time=10000000000 location=axis/video_136_%d.mp4   \
rtspsrc location=rtsp://192.168.16.186:554/axis-media/media.amp?videocodec=h264 \
    ! rtph264depay ! h264parse \
    ! splitmuxsink max-size-time=10000000000 location=axis/video_186_%d.mp4

Appreciate if someone can point other ideas!

~Arm

Answer 2

What do you mean synchronize? if you record to separate video files you do not need any synchronization.. as this is going to totaly separate them.. each RT(S)P stream will contain different timestamps, if you want to align them somehow to the same time (I mean real human time.. like "both should start from 15:00") then you have to configure them this way somehow (this is just idea)..

Also you did not tell us whats inside those rtp/rtsp streams (is it MPEG ts or pure IP.. etc). So I will give example of mpeg ts encapsulated rtp streams. We will go step by step:

Suppose this is one camera just to demonstrate how it may look like:

gst-launch-1.0 -v videotestsrc ! videoconvert ! x264enc ! mpegtsmux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=8888

Then this would be reciever (it must use rtmp2tdepay. We are encapsulating metadata inside MPEG container):

gst-launch-1.0 udpsrc port=8888 caps=application/x-rtp\,\ media\=\(string\)video\,\ encoding-name\=\(string\)MP2T ! rtpmp2tdepay ! decodebin ! videoconvert ! autovideosink

If you test this with your camera .. the autovideosink means that new window will popup displaying your camera..

Then you can try to store it inside file.. we will use mp4mux.. So for same camera input we do:

gst-launch-1.0 -e udpsrc port=8888 caps=application/x-rtp\,\ media\=\(string\)video\,\ encoding-name\=\(string\)MP2T ! rtpmp2tdepay ! tsdemux ! h264parse ! mp4mux ! filesink location=test.mp4

Explanation: We do not decode and reencode(waste of processing power) so I will just demux the MPEG ts stream and then instead of decoding H264 I will just parse it for the mp4mux which accepts video/x-h264.

Now you could use the same pipeline for each camera.. or you can just copypaste all elements into the same pipeline..

Now as you did not provide any - at least partial - attempt to make something out this is going to be your homework :) or make yourself more clear about the synchronization as I do not understand it..

UPDATE After your update to question this answer is not very useful, but I will keep it here as reference. I have no idea how to synchronize that..

Another advise.. try to look at timestamps after udpsrc.. maybe they are synchronized already.. in that case you can use streamsynchronizer to synchronize two streams.. or maybe video/audio mixer:

gst-launch-1.0 udpsrc -v port=8888 ! identity silent=false ! fakesink

This should print the timestamps (PTS, DTS, Duration ..):

/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (1328 bytes, dts: 0:00:02.707033598, pts:0:00:02.707033598, duration: none, offset: -1, offset_end: -1, flags: 00004000 tag-memory ) 0x7f57dc016400

Compare PTS of each stream.. maybe you could combine two udpsrc in one pipeline and after each udpsrc put identity (with different name=something1) to make them start reception together..

HTH