How to use a reprocessCaptureRequest with camera2 API

Question

I'm trying to update a camera project to Android N and in consequence I'm moving my old CameraCaptureSession to a ReprocessableCaptureSession. I did it and it is working fine, but with this new feature I can use the CameraDevice.TEMPLATE_ZERO_SHUTTER_LAG template in my device and I can reprocess frames with the reprocessCaptureRequest.

Here is where my problem appear. Because I don't find any example, and I don't really understand the little documentation about how to use a reprocessCaptureRequest:

Each reprocess CaptureRequest processes one buffer from CameraCaptureSession's input Surface to all output Surfaces included in the reprocess capture request. The reprocess input images must be generated from one or multiple output images captured from the same camera device. The application can provide input images to camera device via queueInputImage(Image). The application must use the capture result of one of those output images to create a reprocess capture request so that the camera device can use the information to achieve optimal reprocess image quality. For camera devices that support only 1 output Surface, submitting a reprocess CaptureRequest with multiple output targets will result in a CaptureFailure.

I tried to have a look to the CTS tests about the camera in google.sources but they do the same than me. Using multiples imageReaders, saving the TotalCaptureResult of the pictures in a LinkedBlockingQueue<TotalCaptureResult>. And later just calling:

TotalCaptureResult totalCaptureResult = state.captureCallback.getTotalCaptureResult();
CaptureRequest.Builder reprocessCaptureRequest = cameraStore.state().cameraDevice.createReprocessCaptureRequest(totalCaptureResult);
reprocessCaptureRequest.addTarget(state.yuvImageReader.getSurface());
sessionStore.state().session.capture(reprocessCaptureRequest.build(), null, this.handlers.bg());

But it always throw me a RuntimeException: java.lang.RuntimeException: Capture failed: Reason 0 in frame 170,

I just want to know which is the right way to work with the ReprocessableCaptureSession because I already tried everything and I don't know what I'm doing wrong.

Answer 1

Finally I found the solution to make my reprocessableCaptureSession work. I use with Flux architecture so don't be confused when you see Dispatcher.dispatch(action), just see it as a callback. So, here is my code:

First How the session is created:

 //Configure preview surface
     Size previewSize = previewState.previewSize;
     previewState.previewTexture.setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());

     ArrayList<Surface> targets = new ArrayList<>();
     for (SessionOutputTarget outputTarget : state.outputTargets) {
        Surface surface = outputTarget.getSurface();
        if (surface != null) targets.add(surface);
     }
     targets.add(previewState.previewSurface);
     CameraCharacteristics cameraCharacteristics = cameraStore.state().availableCameras.get(cameraStore.state().selectedCamera);
     Size size = CameraCharacteristicsUtil.getYuvOutputSizes(cameraCharacteristics).get(0);

     InputConfiguration inputConfiguration = new InputConfiguration(size.getWidth(),
        size.getHeight(), ImageFormat.YUV_420_888);

     CameraCaptureSession.StateCallback sessionStateCallback = new CameraCaptureSession.StateCallback() {
        @Override
        public void onConfigured(@NonNull CameraCaptureSession session) {
           if (sessionId != currentSessionId) {
              Timber.e("Session opened for an old open request, skipping. Current %d, Request %d", currentSessionId, sessionId);
              //performClose(session);
              return;
           }

           try {
              session.getInputSurface();
              //This call is irrelevant,
              //however session might have closed and this will throw an IllegalStateException.
              //This happens if another camera app (or this one in another PID) takes control
              //of the camera while its opening
           } catch (IllegalStateException e) {
              Timber.e("Another process took control of the camera while creating the session, aborting!");
           }

           Dispatcher.dispatchOnUi(new SessionOpenedAction(session));
        }

        @Override
        public void onConfigureFailed(@NonNull CameraCaptureSession session) {
           if (sessionId != currentSessionId) {
              Timber.e("Configure failed for an old open request, skipping. Current %d, request %d", currentSessionId, sessionId);
              return;
           }

           Timber.e("Failed to configure the session");
           Dispatcher.dispatchOnUi(new SessionFailedAction(session, new IllegalStateException("onConfigureFailed")));
        }
     };

     if (state.outputMode == OutputMode.PHOTO) {
        cameraState.cameraDevice.createReprocessableCaptureSession(inputConfiguration, targets, sessionStateCallback, handlers.bg());
     } else if (state.outputMode == OutputMode.VIDEO) {
        cameraState.cameraDevice.createCaptureSession(targets, sessionStateCallback, handlers.bg());
     }

  } catch (IllegalStateException | IllegalArgumentException e) {
     Timber.e(e, "Something went wrong trying to start the session");
  } catch (CameraAccessException e) {
     //Camera will throw CameraAccessException if another we try to open / close the
     //session very fast.
     Timber.e("Failed to access camera, it was closed");
  }

Photo session as been created with 4 surfaces(Preview, YUV(input), JPEG and RAW). After it, I configure my imageWriter:

   Dispatcher.subscribe(Dispatcher.VERY_HIGH_PRIORITY, SessionOpenedAction.class)
     .filter(a -> isInPhotoMode())
     .subscribe(action -> {
           PhotoState newState = new PhotoState(state());
           newState.zslImageWriter = ImageWriter.newInstance(action.session.getInputSurface(), MAX_REPROCESS_IMAGES);
           setState(newState);
     });

Ok, now we have the ImageWriter and the session created. No we start the streaming with the repeating request:

         CaptureRequest.Builder captureRequestBuilder =
            cameraStore.state().cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_ZERO_SHUTTER_LAG);

         captureRequestBuilder.addTarget(previewStore.state().previewSurface);
         captureRequestBuilder.addTarget(photoStore.state().yuvImageReader.getSurface());
         state.session.setRepeatingRequest(captureRequestBuilder.build(), state.zslCaptureCallback, handlers.bg());

To don't add a lot of code, just say that the zslCaptureCallback is a custom callback which save in a LinkedBlockingQueue<TotalCaptureRequest> the X last TotalCaptureRequests. Also, I do the same with the yuvImageReader(input one) saving the last X images in a queue.

Finally here is my "take photo" method:

try {
        //Retrieve the last image stored by the zslImageReader
        Image image = zslImageReaderListener.getImage();
        //Retrieve the last totalCaptureResult from the zslCaptureCallback and create a reprocessableCaptureRequest with it
        TotalCaptureResult captureResult = sessionStore.state().zslCaptureCallback.getCaptureResult(image.getTimestamp());
        CaptureRequest.Builder captureRequest = cameraStore.state().cameraDevice.createReprocessCaptureRequest(captureResult);
        //Add the desired target and values to the captureRequest
        captureRequest.addTarget(state().jpegImageReader.getSurface());
        //Queued back to ImageWriter for future consumption.
        state.zslImageWriter.queueInputImage(image);
        //Drain all the unused and queued CapturedResult from the CaptureCallback
        sessionStore.state().zslCaptureCallback.drain();
        //Capture the desired frame
        CaptureRequest futureCaptureResult = captureRequest.build();
        sessionStore.state().session.capture(futureCaptureResult, new CameraCaptureSession.CaptureCallback() {
           @Override
           public void onCaptureCompleted(@NonNull CameraCaptureSession session,
                                          @NonNull CaptureRequest request,
                                          @NonNull TotalCaptureResult result) {
              Dispatcher.dispatchOnUi(new PhotoStatusChangedAction(PhotoState.Status.SUCCESS));
           }

           @Override
           public void onCaptureFailed(@NonNull CameraCaptureSession session,
                                       @NonNull CaptureRequest request,
                                       @NonNull CaptureFailure failure) {
              super.onCaptureFailed(session, request, failure);
              Exception captureFailedException = new RuntimeException(
                 String.format("Capture failed: Reason %s in frame %d, was image captured? -> %s",
                    failure.getReason(),
                    failure.getFrameNumber(),
                    failure.wasImageCaptured()));
              Timber.e(captureFailedException, "Cannot take mediaType, capture failed!");

              Dispatcher.dispatchOnUi(new PhotoStatusChangedAction(PhotoState.Status.ERROR, captureFailedException));
           }
        }, this.handlers.bg());

        //Capture did not blow up, we are taking the photo now.
        newState.status = PhotoState.Status.TAKING;

     } catch (CameraAccessException | InterruptedException| IllegalStateException | IllegalArgumentException | SecurityException e) {
        Timber.e(e, "Cannot take picture, capture error!");
        newState.status = PhotoState.Status.ERROR;
     }