Android ImageReader YUV 420 888 Repeating Data

Question

I am trying to convert an Image received from ImageReader using the Camera 2 API to a OpenCV matrix and display it on screen using CameraBridgeViewBase, more specifically the function deliverAndDrawFrame. The ImageFormat for the reader is YUV_420_888, which, as far as I understand, has a Y plane with grayscale values for each pixel, and a U plane that has U/V every other with 1 for every 4 pixels. However, when I try to display this image it appears as if the image is repeating and is rotated 90 degrees. The code below is supposed to put the YUV data into a OpenCV matrix (just grayscale for now, not rgba):

/**
 * Takes an {@link Image} in the {@link ImageFormat#YUV_420_888} and puts it into a provided {@link Mat} in rgba format.
 *
 * @param yuvImage {@link Image} in the {@link ImageFormat#YUV_420_888} format.
 */
public static void yuv420888imageToRgbaMat(final Image yuvImage, final Mat rgbaMat) {

    final Image.Plane
            Yp     = yuvImage.getPlanes()[0],
            UandVp = yuvImage.getPlanes()[1];

    final ByteBuffer
            Ybb     = Yp    .getBuffer(),
            UandVbb = UandVp.getBuffer();

    Ybb    .get(mYdata    , 0, 480*640        );
    UandVbb.get(mUandVData, 0, 480*640 / 2 - 8);

    for (int i = 0; i < 640*480; i++) {
        for (int j = 0; j < 4; j++) {
            mRawRGBAFrameData[i + 640*480*j] = mYdata[i];
        }
        mRawRGBAFrameData[i*4  ] = mYdata[i];
        mRawRGBAFrameData[i*4+1] = mYdata[i];
        mRawRGBAFrameData[i*4+2] = mYdata[i];
        mRawRGBAFrameData[i*4+3] = -1;
    }
}

Here is my code for the OpenCV frame:

private class CameraFrame implements CvCameraViewFrame {

    private Mat mRgba;

    @Override
    public Mat gray() {

        return null;
    }

    @Override
    public Mat rgba() {

        mRgbaMat.put(0, 0, mRawRGBAFrameData);

        return mRgba;
    }

    public CameraFrame(final Mat rgba) {

        super();

        mRgba = rgba;
    }
}

The code for receiving drawing the frame:

private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {

    @Override
    public void onImageAvailable(ImageReader reader) {

        final Image yuvImage = reader.acquireLatestImage();

        yuv420888imageToRgbaMat(yuvImage, mRgbaMat);

        deliverAndDrawFrame(mFrame);

        yuvImage.close();
    }
};

And, this is the code for making the image reader:

mRgbaMat = new Mat(mFrameHeight, mFrameWidth, CvType.CV_8UC4);

mFrame = new CameraFrame(mRgbaMat);

mImageReader = ImageReader.newInstance(mFrameWidth, mFrameHeight, ImageFormat.YUV_420_888, 1);

mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);

AllocateCache();

This is the initialization of the arrays:

protected static byte[] mRawRGBAFrameData = new byte[640*480*4], mYdata = new byte[640*480], mUandVData = new byte[640*480 / 2];

Notes: mFrameWidth is 480 and mFrameHeight is 640. One weird thing is that the height and width for ImageReader and the Image received from it have inverted dimensions.

Here is the image with the code above: https://i.stack.imgur.com/lcdzf.png

Here is the image with this instead in yuv420888imageToRgbaMat https://i.stack.imgur.com/T2MOI.png

for (int i = 0; i < 640*480; i++) {
    mRawRGBAFrameData[i] = mYdata[i];
}

We can see that data is repeating in the Y frame and for some reason this gives an actual good looking image.

quintin quintin · Accepted Answer · 2018-02-08T18:19:17

For anyone having the same problem of trying to use OpenCV with the Camera 2 API, I have come up with a solution. The first thing that I discovered was the fact that there is padding in the ByteBuffer that the ImageReader supplies, so this can cause distortion in the output if you do not account for it. Another thing that I chose do to was to create my own SurfaceView and draw to it using a Bitmap instead of using CameraViewBase, and so far it has worked out great. OpenCV has a function Util.matToBitmap that takes a BGR matrix and converts it to an android Bitmap, so that has been useful. I obtain the BGR matrix by putting information from the first two Image.Planes supplied by the ImageReader into an OpenCV one channel matrix that is formatted as YUV 420, and using Imgproc.cvtColor with Imgproc.COLOR_YUV420p2BGR. The important thing to know is that the Y plane of the image has full pixels, but the second UV plane has interleaved pixels that map one to four Y pixels, so the total length of the UV plane is half of the Y plane. See here. Anyways, here is some code:

Initialization of matrices

m_BGRMat = new Mat(Constants.VISION_IMAGE_HEIGHT, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC3);
m_Yuv420FrameMat = new Mat(Constants.VISION_IMAGE_HEIGHT * 3 / 2, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC1);

Every frame:

// Convert image to YUV 420 matrix
ImageUtils.imageToMat(image, m_Yuv420FrameMat, m_RawFrameData, m_RawFrameRowData);
// Convert YUV matrix to BGR matrix
Imgproc.cvtColor(m_Yuv420FrameMat, m_BGRMat, Imgproc.COLOR_YUV420p2BGR);
// Flip width and height then mirror vertically
Core.transpose(m_BGRMat, m_BGRMat);
Core.flip(m_BGRMat, m_BGRMat, 0);
// Draw to Surface View
m_PreviewView.drawImageMat(m_BGRMat);

Here is the conversion to YUV 420 matrix:

/**
 * Takes an Android {@link Image} in the {@link ImageFormat#YUV_420_888} format and returns an OpenCV {@link Mat}.
 *
 * @param image {@link Image} in the {@link ImageFormat#YUV_420_888} format
 */
public static void imageToMat(final Image image, final Mat mat, byte[] data, byte[] rowData) {
    ByteBuffer buffer;
    int rowStride, pixelStride, width = image.getWidth(), height = image.getHeight(), offset = 0;
    Image.Plane[] planes = image.getPlanes();
    if (data == null || data.length != width * height) data = new byte[width * height * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
    if (rowData == null || rowData.length != planes[0].getRowStride()) rowData = new byte[planes[0].getRowStride()];
    for (int i = 0; i < planes.length; i++) {
        buffer = planes[i].getBuffer();
        rowStride = planes[i].getRowStride();
        pixelStride = planes[i].getPixelStride();
        int
            w = (i == 0) ? width : width / 2,
            h = (i == 0) ? height : height / 2;
        for (int row = 0; row < h; row++) {
            int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
            if (pixelStride == bytesPerPixel) {
                int length = w * bytesPerPixel;
                buffer.get(data, offset, length);
                // Advance buffer the remainder of the row stride, unless on the last row.
                // Otherwise, this will throw an IllegalArgumentException because the buffer
                // doesn't include the last padding.
                if (h - row != 1)
                    buffer.position(buffer.position() + rowStride - length);
                offset += length;
            } else {
                // On the last row only read the width of the image minus the pixel stride
                // plus one. Otherwise, this will throw a BufferUnderflowException because the
                // buffer doesn't include the last padding.
                if (h - row == 1)
                    buffer.get(rowData, 0, width - pixelStride + 1);
                else
                    buffer.get(rowData, 0, rowStride);
                for (int col = 0; col < w; col++)
                    data[offset++] = rowData[col * pixelStride];
            }
        }
    }
    mat.put(0, 0, data);
}

And finally, drawing

/**
 * Given an {@link Mat} that represents a BGR image, draw it on the surface canvas.
 * use the OpenCV helper function {@link Utils#matToBitmap(Mat, Bitmap)} to create a {@link Bitmap}.
 *
 * @param bgrMat BGR frame {@link Mat}
 */
public void drawImageMat(final Mat bgrMat) {
    if (m_HolderReady) {
        // Create bitmap from BGR matrix
        Utils.matToBitmap(bgrMat, m_Bitmap);
        // Obtain the canvas and draw the bitmap on top of it
        final SurfaceHolder holder = getHolder();
        final Canvas canvas = holder.lockCanvas();
        canvas.drawBitmap(m_Bitmap, null, new Rect(0, 0, m_HolderWidth, m_HolderHeight), null);
        holder.unlockCanvasAndPost(canvas);
    }
}

This way works, but I imagine the best way to do it is to set up an OpenGL rendering context and write some sort of simple shader to display the matrix.

Android ImageReader YUV 420 888 Repeating Data

1 Answers