Android mediacodec decoding h264 stream in real time with about 1 second latency

Question

I am trying to decode h264 live stream from WIFI camera by madiacodec in Galaxy S3, the video plays OK, but it seems that mediacode low level always buffers 1 second of video frames, a new frame can be decoded only by a new incoming NAL unit. This causes about 1 second of latency.

When MediaCodec.createDecoderByType("video/avc"); is called I can see from logcat that 22 buffers are allocated,

01-04 15:39:02.799: I/ExtendedCodec(13374): Smoothstreaming Enabled
01-04 15:39:02.809: I/ACodec(13374): [OMX.qcom.video.decoder.avc] Now Loaded->Idle
01-04 15:39:02.809: I/ACodec(13374): [OMX.qcom.video.decoder.avc] Allocating 22 buffers from a native window of size 245760 on output port
01-04 15:39:02.889: D/DecodeActivity(13374): Decoder started at --- 1388867942894
01-04 15:39:02.889: I/ACodec(13374): [OMX.qcom.video.decoder.avc] Now Idle->Executing
01-04 15:39:03.019: I/ACodec(13374): [OMX.qcom.video.decoder.avc] Now Executing

And I can see about 16 NAL units need to be pushed into decoder before I can get any frame out.

My question is that why this video buffering? any way I can try to allocate less buffer if that is the problem. Is there anyway I can reduce this video buffering?

Answer 1

First of all, are you sure that each NAL unit corresponds to one frame? H.264 doesn't dictates how many NAL unit will be there for each frame, it depends on encoder implementation. Second thing, I'm having similar issues when using a device with Qualcomm chipset. I'm not using MediaCodec, but IOMX interface, but basically I get the same latency. I wonder if you you have prints in logcat saying how mnay input buffers are allocated. Those will be more relevant to the issue, and not the buffers on the output port.