Why is FFMpeg outputting wrong NAL unit types? (javascript h264 livestream)

Question

I am trying to set up a livestream in the browser using h264 encoding, in which javascript decodes the h264 frames and paints it on a Canvas-element (or using WebGL).

Both Broadway and Prism implement decoding NAL units of type 1, 5, 7, and 8.

My current setup is as follows:

FFMpeg outputs an MPEG-TS stream with h264 data
The stream is piped to netcat which listens on port 8084
A websocket server in NodeJS pipes data from port 8084 to clients on 8085
The jsmpeg library decodes MPEG-TS into separate NAL units
The separate NAL units are decoded by Broadway or Prism which outputs to a canvas

I am using this FFMpeg command:

ffmpeg -f v4l2 -i /dev/video0 -r 15 -c:v h264_nvenc -pix_fmt yuv420p -b:v 500k -profile:v baseline -tune zerolatency -f mpegts - | nc -l -p 8084 127.0.0.1

The problem is that the NAL units I'm getting are of type 9 (or maybe 6?), here is the header of one of the NAL units that javascript is receiving, in Base64 and binary formatting:

echo "AAAAAQnwAAAAAQYBBAAECBCAAAAAAWHg" | base64 -d | xxd -b
00000000: 00000000 00000000 00000000 00000001 00001001 11110000  ......
00000006: 00000000 00000000 00000000 00000001 00000110 00000001  ......
0000000c: 00000100 00000000 00000100 00001000 00010000 10000000  ......
00000012: 00000000 00000000 00000000 00000001 01100001 11100000  ....a.

Neither Broadway nor Prism supports these NAL unit types. How can I configure FFMpeg to only output NAL units of type 1, 5, 7, and 8?

EDIT: I have also tried the following command:

ffmpeg -f v4l2 -i /dev/video0 -r 15 -c:v h264_nvenc -pix_fmt yuv420p \
    -b:v 500k -profile:v baseline -tune zerolatency \
    -movflags frag_keyframe+empty_moov -g 52 -f mp4 - \
    | nc -l -p 8084 127.0.0.1

Which encodes to mp4, and from there I try to parse NAL units starting with three zero bytes. The lines look all similar to the following:

echo "AAAACAYBBABOCBCAAAARemHk4f8df1Su" | base64 -d | xxd -b
00000000: 00000000 00000000 00000000 00001000 00000110 00000001  ......
00000006: 00000100 00000000 01001110 00001000 00010000 10000000  ..N...
0000000c: 00000000 00000000 00010001 01111010 01100001 11100100  ...za.
00000012: 11100001 11111111 00011101 01111111 01010100 10101110  ....T.

That is type 6 (00110 in the 5th byte), still not the desired NAL unit type.

UPDATE: The reason it didn't work for me was a matter of an encoding/decoding issue between chars and bytes in Javascript. I have put the working code on github for others who may want to do a similar thing.

Concerning the NAL units, it turns out the raw video of FFMpeg output contained type 6 of only a few bytes, followed by type 1 that has the frame data. The type 6 can be discarded. Thanks to the comments and accepted answer for the insight into this.

Those are Access Unit Delimiter NALU required by MPEG-TS. Since your decoders don't support them, you need to strip them out. — Gyan
@Gyan These are the only packets in the stream, there are no other types. However, I did notice that after 6 bytes into the header, the header seemingly starts again, as type 6? I don't know enough about this. How do I go about that stripping, if they're all type 9 or 6? — Yeti
@Gyan I've already tried using a raw 264 stream using ... -f h264 - | nc ... without the MPEG-TS demuxer, however, this did not work out of the box either, thus it probably did not contain the right types either (I might need to double-check this). It should work for -f mp4, but that is not suitable for streaming. — Yeti

micha137 micha137 · Accepted Answer · 2018-07-11T05:57:45

There is a misconception in "jsmpeg library decodes MPEG-TS into separate NAL units", because it really decodes the transport stream into PES packets. PES packets can contain several NAL units.

Removing the Access Unit delimiters is easy, because they are really just one byte long. So you could just skip 0x0000000109f0 (from your first dump) and process the next NAL unit at offset 6.

That one is of type 6, which means Supplemental Enhancement Information. It needs to be skipped as well, because your decoder doesn't support it (it is not neccessary for decoding). SEI NAL units are also normally quite short, ~10-50 bytes. So just look for the next start code 0x00000001...

Why is FFMpeg outputting wrong NAL unit types? (javascript h264 livestream)

1 Answers