MediaCodec decoding h264 wrong on one device

Question

I'm trying to decode h.264 stream using Android's MediaCodec interface. Everything is working just fine on my test devices, but on one customer device that I dont' have access to (Samsung Tab S) there are strange issues.

When I decode the stream I don't send any SPS/PPS NALs or an initial frame. I just start pushing the data from the live stream, chopped into blocks ending with a 0x09 NAL and the decoder will synchronize itself nicely without problems quite quickly.

The issue with at least this one device is that when I get a BufferInfo from the decoder it will claim it decoded 1413120 bytes of data, but the buffer size is only 1382400! So of course if I even try to get that much data out of the buffer it will crash.

The video is 1280x720 and is decoded into NV12, so the buffer size is just fine. The reported decoded output size isn't. If I force the size as 1382400 and convert the NV12 into RGB I get almost correct picture. The first 32 lines have a strong green color and the blue channel is shifted by quite a lot. This means that the UV block is decoded partially wrong on this device.

Has anyone run into this kind of an issue before? I have recorded the raw h264 stream from that specific device and it plays just fine with no green blocks or color shifts.

Should I really set up SPS/PPS and an initial frame before starting the streaming? The stream seems to contain everything needed since the decoder realizes the resolution to be correct, sets up buffers and decodes on every other device I've tested except this one. So I'm just wondering if Samsung has something special going on.

Another application decoding the same stream shows it without issues, but as far as I know they use ffmpeg internally, not MediaCodec. I would rather use built-in system codecs if possible.

Here is an example of the result. Don't have an image of just the stream, do note that the frame is rotated. The Y component in the green area is just fine and on the white block on the right you can clearly see the blue shift.

Edit: Even if I start the decoder with the SPS/PPS blocks in csd-0 the color problems persist. So it isn't due to that.

Also managed to test the exact stream with another device. No green bar, no color shifts. So it's a problem with the codec in that particular device/model.

Try setting sps/pps as we cannot expect the same behavior on all the devices. Set it and check, it might help. — Karthik Kumar
Hmm, so to clarify MediaCodec.BufferInfo.size is 1413120 but the capacity of the ByteBuffer is only 1382400? What does the output MediaFormat contain? — mstorsjo
Yes, they're different in how/when you have access to the NALs (and you can e.g. pass SPS/PPS either via csd-0 only, or csd-0 and csd-1 via MediaFormat, or as a normal input buffer with the CODEC_CONFIG flag). Also, how to handle multiple NALs if the transport doesn't indicate frame boundaries clearly is of course an issue as well. But in any case I think both of them are quite orthogonal issue to malformed pixel data layout like the issue you're seeing (and which the CTS test tries to handle) - I doubt that something such would be related. — mstorsjo
Not having access to the device indeed makes things harder, sorry I forgot you had that situation. (I also checked that developer.samsung.com doesn't have any Galaxy Tab S, only Tab S2, available for remote testing, otherwise that would have been an option.) — mstorsjo
Yes, SPS/PPS shouldn't relate to this. If you have any sort of visible decoder output, the decoder has received the SPS/PPS somehow (and changing which way it is passed shouldn't matter much) - without this, it is impossible to get any sensible decoding output at all. — mstorsjo

Dean Wild Dean Wild · Accepted Answer · 2016-02-09T10:31:23

I had similar issues in the past (specifically on Samsung devices) and if I remember correctly it was due to the missing SPS/PPS data. You must feed in the SPS/PPS data if you want consistent results.

Not a direct solution to your issue but a possible workaround is to use an alternative decoder (if present) when running on that particular device.

I'm not sure how you are instantiating your decoder but often people use the mime type like so:

decoder = MediaCodec.createDecoderByType("video/avc");

The device will then choose the preferred decoder (probably hardware).

You can alternatively instantiate a specific decoder like so:

decoder = MediaCodec.createByCodecName("OMX.google.h264.decoder");

// OR

decoder = MediaCodec.createByCodecName("OMX.qcom.video.decoder.avc");

In my experience, most devices have at least 2 different H264 decoders available and you may find that an alternative decoder on this device performs without faults.

You can list all the available codecs using the following code:

static MediaCodecInfo[] getCodecs() {

    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
        MediaCodecList mediaCodecList = new MediaCodecList(MediaCodecList.ALL_CODECS);
        return mediaCodecList.getCodecInfos();
    } else {
        int numCodecs = MediaCodecList.getCodecCount();
        MediaCodecInfo[] mediaCodecInfo = new MediaCodecInfo[numCodecs];

        for (int i = 0; i < numCodecs; i++) {
            MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);
            mediaCodecInfo[i] = codecInfo;
        }

        return mediaCodecInfo;
    }       
}

MediaCodec decoding h264 wrong on one device

2 Answers