How to access all Images in the ImageReader Queue using Android Camera2 API

Question

In my project, I need to capture the frames of the camera streams continuously. Here is the current code snippet I used.

To set up the ImageReader, I set the maxImages to 20. Let is every time when callback is triggered, there would have 20 frames in the ImageReader Queue.

imageReader = ImageReader.newInstance(
            optimumSize.getWidth(),
            optimumSize.getHeight(),
            ImageFormat.YUV_420_888,
            20
);

Then to access each image of these 20 frames. I used the following snippet.

imageReader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
    @Override
    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireNextImage();
        while (image != null) {
            // some processing here.....
            image.close();
            image = reader.acquireNextImage();
        }
        if (image != null) {
            image.close();
        }
    }
}, processingHandler);

The key obstacle here is to be able to access each of 20 frames in a callback, for further image processing. However the aforementioned code seems have some problems (I can only access the latest image in the underlying queue). In fact, I only need to access a small patch (50 x 50 pixels) in each frames, specified by users.

The reason for doing this is that I need to get the 20 continuous frames data with sampling frequency being ~60Hz. This seems really hard to achieve if we can only access single frame in each callback, which can only achieve up to 30fps.

Any suggestions would be super welcome! Thanks!

onImageAvailable is called when a single frame is available, so you should remove the while loop and rely on the callback. Furthermore, the actual processing should be done on another thread. Preallocate plenty of memory, copy the small patch and close the image. Wait in the other thread for enough data to be available and start processing if there is. Perhaps Renderscript can be used for fast image processing. — Michiel
Thank you so much @Michiel. One quick follow-up question. If I set the maxImages to 20, which is the last argument of ImageReader.newInstance(), in theory once the callback is triggered, should there be 20 images in the Queue? if so how shall access, let's say, the non-latest 19 images? Cheers! — Chen Chen
I'm pretty certain that's the case, but to be certain you should test it out. Having separate Handler for your ImageReader and MediaProjection will help as well. — Michiel

Eddy Talvala Eddy Talvala · Accepted Answer · 2020-07-28T18:04:04

Setting maxImages to 20 just means the queue will allow you to acquire 20 Images at the same time; it does not mean the onImageAvailable callback will fire only once 20 images are queued. The callback fires as soon as a single image is present.

Most camera devices run at 30fps max, so it's not surprising that's the speed you're seeing. Some cameras do have 60fps modes, but you have to explicitly switch to a CONTROL_AE_TARGET_FPS_RANGE of (60,60) to get that, and that's only if the device's CONTROL_AE_AVAILABLE_TARGET_FPS_RANGE values include that range.

60fps may also be resolution-limited (check the StreamConfigurationMap for minimum frame durations to find what resolutions can support 60fps, if you want to double-check).

How to access all Images in the ImageReader Queue using Android Camera2 API

1 Answers