Get ByteBuffer from Image for TensorFlow Lite Model

Question

I am creating an android app to run on Google Glass Enterprise Edition 2 that does Real-time Face Recognition. I am using Camera X as my Camera API and TensorFlow Lite (TFLite) as my classification model. However, the TFLite model input requires ByteBuffer which I am unable to convert into from the image retrieved from CameraX.

How do I get my Image from CameraX into ByteBuffer class for my TFLite Model?

Camera X Image Analysis: Reference

            val imageAnalysis = ImageAnalysis.Builder()
                    .setTargetResolution(Size(640, 360))
                    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                    .build()

            imageAnalysis.setAnalyzer(AsyncTask.THREAD_POOL_EXECUTOR, ImageAnalysis.Analyzer { imageProxy ->
                val rotationDegrees = imageProxy.imageInfo.rotationDegrees
                val mediaImage = imageProxy.image

                if (mediaImage != null) {
                    val image = InputImage.fromMediaImage(mediaImage, rotationDegrees)

                    /* Classify the Image using TensorFlow Lite Model */

                }

            })

TensorFlow Model Sample Code

val model = FaceRecognitionModel.newInstance(context)

// Creates inputs for reference.
val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
inputFeature0.loadBuffer(byteBuffer)

// Runs model inference and gets result.
val outputs = model.process(inputFeature0)
val outputFeature0 = outputs.outputFeature0AsTensorBuffer

// Releases model resources if no longer used.
model.close()

yyoon yyoon · Accepted Answer · 2021-02-17T01:23:44

Try using the TensorImage class from the TensorFlow Lite Support Library.

Roughly, you can follow these steps.

Convert the Image object into Bitmap. There should be other Stackoverflow questions on how to do this (e.g., this answer)
Create a TensorImage object from the Bitmap object using TensorImage.fromBitmap() factory.
Call getBuffer() method on the TensorImage object to get the underlying ByteBuffer.

You might also want to do some image pre-processing, in case the image from CameraX doesn't exactly match the format expected by the model. For this, you can explore the ImageProcessor utility.

Get ByteBuffer from Image for TensorFlow Lite Model

1 Answers