Gstreamer issue with adding timeoverlay on RTMP stream

1
votes

I needed to add a time overlay to an rtmp stream, and save to disk. The following pipeline, without the overlay, works fine:

gst-launch-1.0 -v \
rtmpsrc location=rtmp://192.168.x.x/live/0 do-timestamp=true ! queue2 ! flvdemux name=demux \
flvmux name=mux \
demux.video ! queue ! decodebin \
    ! nvvidconv \
    ! 'video/x-raw(memory:NVMM),width=1920,height=1080, format=(string)I420, framerate=50/1' \
    ! nvv4l2h264enc ! h264parse \
    ! mux.video \
demux.audio ! queue name="dmx_aud_q" ! mux.audio \
mux.src ! queue name="mux_q" ! filesink location="rtmp.flv"

Once I add the timeoverlay (or even clockoverlay), the pipeline doesn't run:

gst-launch-1.0 -v \
rtmpsrc location=rtmp://192.168.0.168/x.x do-timestamp=true ! queue2 ! flvdemux name=demux \
flvmux name=mux \
demux.video ! queue ! decodebin \
    ! timeoverlay \
    ! nvvidconv \
    ! 'video/x-raw(memory:NVMM),width=1920,height=1080, format=(string)I420, framerate=50/1' \
    ! nvv4l2h264enc ! h264parse \
    ! mux.video \
demux.audio ! queue name="dmx_aud_q" ! mux.audio \
mux.src ! queue name="mux_q" ! filesink location="rtmp.flv"

The GST_DEBUG=3 logs are as added at the end. The Pipeline graph is as below.

Failed Pipeline for Adding Overlay to RTMP stream

From what I understand, at the point where the decodebin hands over to timeoverlay, there is some issue with caps negotiation. I'm not able to figure out how to make timeoverlay accept or output data in a way that the pipeline can continue to mux. Any help to understand what is happening here, and how to find a solution, would be great.

This is on an Nvidia Jetson Nano board, running Ubuntu 18.0. (The plugins starting with "nv" can usually be replaced with regular ones on other systems, I believe - nvvidconv with videoconvert, nvv4l2h264enc with omxh264enc etc.

Setting pipeline to PAUSED ... Opening in BLOCKING MODE 0:00:00.119491546 834 0x55b9d05600 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360 Failed to determine interlace mode 0:00:00.119570298 834
0x55b9d05600 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360 Failed to determine interlace mode 0:00:00.119623164 834
0x55b9d05600 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x55b9cf2360 Failed to determine interlace mode 0:00:00.119721552 834
0x55b9d05600 WARN v4l2 gstv4l2object.c:4408:gst_v4l2_object_probe_caps: Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1 Pipeline is PREROLLING ... 0:00:00.406155973 834 0x55b9a72370 WARN flvdemux gstflvdemux.c:659:gst_flv_demux_parse_tag_script: failed reading a tag, skipping /GstPipeline:pipeline0/GstQueue:dmx_aud_q.GstPad:sink: caps = audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, rate=(int)44100, channels=(int)2, codec_data=(buffer)1210 /GstPipeline:pipeline0/GstQueue:dmx_aud_q.GstPad:src: caps = audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, rate=(int)44100, channels=(int)2, codec_data=(buffer)1210 /GstPipeline:pipeline0/GstFlvMux:mux.GstFlvMuxPad:sink_1: caps = audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, rate=(int)44100, channels=(int)2, codec_data=(buffer)1210 /GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink: caps = video/x-h264, stream-format=(string)avc, width=(int)1920, height=(int)1080, codec_data=(buffer)0142002affe100166742002a95a81e0089f961000003000100000300648401000468ce3c80 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstH264Parse:h264parse1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 Opening in BLOCKING MODE 0:00:00.823485062 834
0x55b9ce8a30 WARN v4l2 gstv4l2object.c:4408:gst_v4l2_object_probe_caps: Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1 0:00:00.823543397 834 0x55b9ce8a30 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x7f6807a640 Failed to determine interlace mode NvMMLiteOpen : Block : BlockType = 261 NVMEDIA: Reading vendor.tegra.display-size : status: 6 NvMMLiteBlockCreate : Block : BlockType = 261 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 0:00:00.932089228 834 0x55b9ce8a30 WARN
v4l2 gstv4l2object.c:4408:gst_v4l2_object_probe_caps: Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1 0:00:00.932560124 834 0x55b9ce8a30 WARN v4l2 gstv4l2object.c:2370:gst_v4l2_object_add_interlace_mode:0x7f6807a640 Failed to determine interlace mode /GstPipeline:pipeline0/GstDecodeBin:decodebin0/nvv4l2decoder:nvv4l2decoder0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)50/1

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.065: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.066: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.067: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.067: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.068: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.069: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed 0:00:00.942959420 834 0x55b9ce8a30 WARN GST_PADS gstpad.c:4226:gst_pad_peer_query: could not send sticky events 0:00:00.943568965 834 0x55b9ce8a30 WARN
v4l2videodec gstv4l2videodec.c:1433:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency 0:00:00.944316482 834
0x55b9ce8a30 WARN v4l2bufferpool gstv4l2bufferpool.c:1054:gst_v4l2_buffer_pool_start: Uncertain or not enough buffers, enabling copy threshold

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.073: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.074: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed 0:00:00.948613871 834 0x55b9ce8a30 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-h264, stream-format=(string)byte-stream, width=(int)1920, height=(int)1080, framerate=(fraction)50/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, alignment=(string)au, profile=(string)baseline, level=(string)4.2 in anything we support 0:00:00.948674601 834 0x55b9ce8a30 WARN basetransform gstbasetransform.c:1415:gst_base_transform_reconfigure: warning: not negotiated 0:00:00.948709446 834 0x55b9ce8a30 WARN
basetransform gstbasetransform.c:1415:gst_base_transform_reconfigure: warning: not negotiated WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1: not negotiated Additional debug info: gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter1: not negotiated 0:00:00.971426937 834 0x7f70004a80 WARN
basesrc gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error. 0:00:00.971545793 834 0x7f70004a80 WARN
basesrc gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4) 0:00:00.978535326 834 0x7f6807c8f0 WARN v4l2bufferpool gstv4l2bufferpool.c:1518:gst_v4l2_buffer_pool_dqbuf: Driver should never set v4l2_buffer.field to ANY

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.105: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed ERROR: from element /GstPipeline:pipeline0/GstRTMPSrc:rtmpsrc0: Internal data stream error.

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.106: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed Additional debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstRTMPSrc:rtmpsrc0: streaming stopped, reason not-negotiated (-4) ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ...

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.107: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.108: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.109: gst_structure_copy: assertion 'structure != NULL' failed

(gst-launch-1.0:834): GStreamer-CRITICAL **: 14:19:35.109: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed Freeing pipeline ...

1
gstreamernvidia-jetson-nano
You can check caps negotiation with GST_DEBUG=GST_CAPS:5. Shot in a dark - isnt the problem that nvv4l2decoder produces DMA buffers, but timeoverlay works with just normal buffers? I guess there is no such thing as nvtimeoverlay . What happens when you add videoconvert before the timeoverlay (but I guess you would loose the DMA stuff)? – nayana
Hello @nayana, thanks for the response. You guessed correctly, that is indeed how I ended up solving it - even though timeoverlay source caps say ANY, it cannot handle NVVM memory. decodebin uses nvv4l2decoder internally, which produces 'video/x-raw(memory:NVMM)' on it's src. I added an nvvidconv ! 'video/x-raw' before the timeoverlay, and it works now. Can you please add it as an answer here, so that I can accept it as an answer? – Aswin Kumar
I've got some idea.. what about using the nvcompositor to do the composition of your decoded video and timeoverlay and that way it may be faster? The timeoverlay should produce the full screen frames with alpha.. I have no idea if it indeed speeds up things somehow. Maybe it will just be overcomplicated. – nayana

1 Answers

3
votes

So as Aswin said, it was solved by adding convert before timeoverlay. Its because timeoverlay cannot work with DMA buffers (thats the (memory:NVMM) means) So the pipeline looks like original except for this change:

... decodebin ! nvvidconv ! 'video/x-raw' !  timeoverlay ! nvvidconv ! 'video/x-raw(memory:NVMM)

The errors are basicaly about timeoverlay not being able to link to nvv4l2decoder which outputs the buffers as DMA. We need nvvidconv to copy those buffers so we can work with them in user space.

Longer explanation with details:

With my faint understanding of DMA/CPU buffers (please correct)- all nv* elements are able to work with DMA buffers (more directly I guess) which speeds up the process. But between nvv4l2decoder and nvv4l2h264enc we put the timeoverlay to add time into the video..

But unfortunately timeoverlay element does not have ability to work with those buffers. So we added the nvviddonv which slowly copies the buffers (it goes through CPU) into user space, slowly we do the changes, and then it slowly goes back to the faster "DMA buffer space". But I dont see some way around this. Unless there is some fancy custom element that could work with the buffers directly.

There was some work in GStreamer to better work with these buffers in past few years - I saw some presentations by ndufresne about DMA fences (working with DMA buffers asynchronosely) and zero copy pipelines (not sure if it applies): Zero copy: https://youtu.be/kNaa1fPv_uo DMA fences: https://youtu.be/HpmzJGHqObs

Some general Linux article: https://01.org/linuxgraphics/gfx-docs/drm/driver-api/dma-buf.html