Using a QUdpSocket for Custom FFMPEG I/O (raw H.264)

Question

I'm trying to create a custom I/O layer to pass raw H.264 frame data to FFMPEG from QUdpSocket datagrams. The H.264 frames don't have any sort of container and are simply streamed from the frame grabbing device. Something is going wrong, because for some reason FFMPEG thinks the access unit has a size of 1 when I call avcodec_decode_video2():

[h264 @ 0x1030dd000] missing picture in access unit with size 1
[h264 @ 0x1030dd000] no frame!

My first step is to probe the first frame for the AVInputFormat:

socket->readDatagram(datagram.data(), datagram.size());

AVProbeData probeData;
probeData.filename = udpUrl.toStdString().c_str();
probeData.buf_size = datagram.size() + AVPROBE_PADDING_SIZE;
probeData.buf = (unsigned char *)malloc(datagram.size() + AVPROBE_PADDING_SIZE);
memcpy(&probeData.buf[AVPROBE_PADDING_SIZE], datagram.data(), datagram.size());

inputFormat = av_probe_input_format(&probeData, 1);

Next I allocate an AVIOContext:

buffer = (unsigned char *)malloc(bufferSize);
ioContext = avio_alloc_context(buffer, bufferSize, 0, socket, &readFrame, NULL, NULL);

After which I call avformat_open_input_stream():

if (av_open_input_stream(&formatContext, ioContext, "", inputFormat, NULL) != 0)

Which seems to return successfully. However, the video stream is missing important information such as width, height, and pixel format, which then causes av_find_stream_info() to fail. I can set these parameters manually, but it doesn't result in a successful decode, and it makes me wonder what else I'm missing.

From what I can tell, the NAL units are intact:

First Frame
00:00:00:01:27:64:00:28:ad:84:09:26:6e:23:34:90:81:24:cd:c4:66:92:10:24:99:b8:8c:d2:42:04:93:37:11:9a:48:40:92:66:e2:33:49:08:12:4c:dc:46:69:21:05:5a:eb:d7:d7:e4:fe:bf:27:d7:ae:b5:50:82:ad:75:eb:eb:f2:7f:5f:93:eb:d7:5a:ab:40:50:1e:c8:
00:00:00:01:28:ee:3c:b0:
00:00:00:01:25:88:84:27:...

Second Frame
00:00:00:01:21:9a:59:15:...

Am I missing an API call? Can anyone see something that I might be doing wrong?

If you run ffmpeg locally on the computer where your frame grabbing device is, and just pipe the output to it, do you get the same problem? — sashoalm
"ffmpeg -i /dev/video0" and "cat /dev/video0 | ffmpeg -i pipe:0" both decode the video without any complaints. — Colin Ray

sashoalm sashoalm · Accepted Answer · 2012-04-19T20:24:07

This is just a guess but the problem might be because you're using UDP, instead of, say TCP. The problem is that UDP is unreliable, packets might go missing, be duplicated, or arrive out of order. And packets might not correspond to frames. So you might get bits and pieces of frames and garbled information. I don't think that ffmpeg would handle that well.

From Wikipedia:

UDP uses a simple transmission model without implicit handshaking dialogues for providing reliability, ordering, or data integrity. Thus, UDP provides an unreliable service and datagrams may arrive out of order, appear duplicated, or go missing without notice. UDP assumes that error checking and correction is either not necessary or performed in the application, avoiding the overhead of such processing at the network interface level

So maybe you should try using TCP instead, and see if it works. TCP should ensure that you receive exactly what you sent, so it shouldn't be different than piping the output locally to ffmpeg.

Using a QUdpSocket for Custom FFMPEG I/O (raw H.264)

1 Answers