ffmpeg producing strange NAL suffixes for MPEG-TS with h264

Question

I am making h264 stream analyzer and have found strange ffmpeg behaviour while writing and reading same frame packetized with MPEG-TS.

I have dumped both encoding and decoding frame data in HEX and found these differences:

At the beginning of frame there is NAL delimiter added:

+ 00
+ 00
+ 00
+ 01
+ 09
+ f0

that looks absolutely legit to me.

But here comes strange things:

If current packet is not the last one than some suffix is added (i.e. there is no this suffix in the last written frame). And this suffix is added to current read frame (i.e. this is not prefix to next frame).

+ e0
+ 00
+ 00
+ 00
+ 01
+ ce
+ 8c
+ 4d
+ 9d
+ 10
+ 8e
+ 25
+ e9
+ fe

(exactly starting with e0 byte added)

I am trying to understand what that could mean. If 00000001 is NAL header than it is followed by ce byte with forbidden zero-bit equals 1 that is absolutely broken.

I am using fork from ffmpeg github repo last merged with 89092fafdde894c6ba4d4f8e3cd1cce0d68bfc22 commit.

ffmpeg is built with --disable-everything --enable-encoder=libopenh264 --enable-muxer=mpegts --enable-demuxer=mpegts --enable-protocol=file --enable-parser=h264 --enable-decoder=libopenh264 --enable-shared --disable-static --disable-programs --disable-doc --enable-libopenh264 options

Yeah, that file may have some corruption. Can you post it someplace and I can confirm? — szatmary
@szatmary yeah, sure. Here it goes. I have checked that I still have problems on this video, though vlc with ffmpeg reads it normaly — ElDorado
The 00 00 00 01 ce sequence is not present in the file you posted. — szatmary
@ElDorado I confirm szatmary's findings. Can you give us the offset? — aergistal
@aergistal I have tried to build project on another PC to give you offset and provide more info but haven't found this pattern any more. Looks like it is environment dependent bug. Maybe memory alignment malfunction on that PC I was working before. As soon as I will be able to reproduce bug - I'll give you more info. Anyway pattern was not present in file and I knew it (sorry if I have not made it clear). It was present only in AVPacket while reading file with ffmpeg. — ElDorado

ElDorado ElDorado · Accepted Answer · 2016-12-13T09:50:47

So I have found the problem.

TL;DR:

pInputFormatContext->flags |= AVFMT_FLAG_KEEP_SIDE_DATA;

right after avformat_open_input

Explanation:

I've found while debbuging that AVPacket was spoiled at utils.c:1661 while performing

av_packet_merge_side_data(pkt);

Even more, I have found subsequence of that additional data seen before:

#define FF_MERGE_MARKER 0x8c4d9d108e25e9feULL

used in av_packet_merge_side_data

Side's data type in AVPacket is AV_PKT_DATA_MPEGTS_STREAM_ID (78) which is used only in demuxers and muxers, but not in codec.

I was confused than I've read annotation to av_read_frame that I was using to read frame data from file:

Return the next frame of a stream.
This function returns what is stored in the file, and does not validate
that what is there are valid frames for the decoder. It will split what is
stored in the file into frames and return one for each call. It will not
omit invalid data between valid frames so as to give the decoder the maximum
information possible for decoding.

I have absolutely no idea of why streams flags where different on two PCs I've used, but disabling merging side data into packet's data did the trick.

So i think it is ffmpeg's documentation issue that it has assured me that "I will be returned what is stored in file" without mentioning that some additional data can be added.

ffmpeg producing strange NAL suffixes for MPEG-TS with h264

1 Answers

TL;DR:

Explanation: