Android TensorFlowLite buffer with 76800 bytes and a ByteBuffer with 307200 bytes

Question

I have been trying to use facenet with ml_kit and I was able to generate the .tflite file following this tutorial but when I try to use it on Android I am getting this error message

TensorFlowLite buffer with 76800 bytes and a ByteBuffer with 307200 bytes

my model is as expected

INPUTS:

[{'name': 'input', 'index': 451, 'shape': array([  1, 160, 160,   3], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (0.0078125, 128)}]

OUTPUTS:

   [{'name': 'embeddings', 'index': 450, 'shape': array([  1, 512], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (0.0235294122248888, 0)}]

And the way that I use the model interpreter is as follow

val input = convertBitmapToByteBuffer(Bitmap.createScaledBitmap(bitmap, IMAGE_WIDTH, IMAGE_HEIGHT, true))
    //val input = convertBitmap(bitmap)
    val inputOutputOptions = createInputOutputOptions()

    // [START mlkit_run_inference]
    val inputs = FirebaseModelInputs.Builder()
        .add(input) // add() as many input arrays as your model requires
        .build()
    firebaseInterpreter.run(inputs, inputOutputOptions)


@Throws(FirebaseMLException::class)
private fun createInputOutputOptions(): FirebaseModelInputOutputOptions {
    // [START mlkit_create_io_options]
    val inputOutputOptions = FirebaseModelInputOutputOptions.Builder()
        .setInputFormat(0, FirebaseModelDataType.INT32, intArrayOf(1, IMAGE_WIDTH, IMAGE_HEIGHT, 3))
        .setOutputFormat(0, FirebaseModelDataType.INT32, intArrayOf(1, 512))
        .build()
    // [END mlkit_create_io_options]
    return inputOutputOptions
}

fun processImage(bitmap: Bitmap){

    val input = convertBitmapToByteBuffer(Bitmap.createScaledBitmap(bitmap, IMAGE_WIDTH, IMAGE_HEIGHT, true))

    val inputOutputOptions = createInputOutputOptions()

    // [START mlkit_run_inference]
    val inputs = FirebaseModelInputs.Builder()
        .add(input) // add() as many input arrays as your model requires
        .build()
    firebaseInterpreter.run(inputs, inputOutputOptions)
        .addOnSuccessListener { result ->
            // [START_EXCLUDE]
            // [START mlkit_read_result]              
            // [END mlkit_read_result]
            // [END_EXCLUDE]
           // listener?.onSuccess(probabilities)
        }
        .addOnFailureListener(
            object : OnFailureListener {
                override fun onFailure(e: Exception) {
                    // Task failed with an exception
                    // ...
                    listener?.onFailure(e)
                }
            })
}

private fun convertBitmapToByteBuffer(bitmap: Bitmap): ByteBuffer {

    val height = bitmap.getHeight()
    val width = bitmap.getWidth()

    val byteBuffer: ByteBuffer = ByteBuffer.allocateDirect(BYTES_PER_CHANNEL * DIM_BATCH_SIZE * width * height * DIM_PIXEL_SIZE)
    byteBuffer.order(ByteOrder.nativeOrder())

    val intValues = IntArray(width * height)
    bitmap.getPixels(intValues, 0, width, 0, 0, width, height);
    // Convert the image to floating point.
    var pixel = 0
    for (i in 0 until width) {
        for (j in 0 until height) {
            val `val` = intValues[pixel++]
            addPixelValueInt(byteBuffer, `val`)
        }
    }
    byteBuffer.rewind()
    return byteBuffer
}


protected fun addPixelValueInt(byteBuffer: ByteBuffer, pixelValue: Int) {
    byteBuffer.putInt((pixelValue shr 16 and 0xFF))
    byteBuffer.putInt((pixelValue shr 8 and 0xFF))
    byteBuffer.putInt((pixelValue and 0xFF))
}

My Config values

private val IMAGE_WIDTH : Int = 160
    private val IMAGE_HEIGHT : Int = 160

    private val DIM_BATCH_SIZE = 1
    private val DIM_PIXEL_SIZE = 3
    private val BYTES_PER_CHANNEL = 4

Any Idea what I am doing wrong?

Jose Luis Larraz Jose Luis Larraz · Accepted Answer · 2019-09-18T10:33:27

After many hours I found that ml_kit can't handle the type QUANTIZED_UINT8 (or at least I don't know hoe to do it). So I have modified my .tflite file to use FLOAT instead

tflite_convert --output_file model_mobile/my_facenet.tflite --graph_def_file facenet_frozen.pb --input_arrays "input" --input_shapes "1,160,160,3" --output_arrays "embeddings" --output_format TFLITE --mean_values 128 --std_dev_values 128 --default_ranges_min 0 --default_ranges_max 6 --inference_type QUANTIZED_UINT8 --inference_input_type QUANTIZED_UINT8

Also I have changed the way that I convert a bitmap to something that can be used by ml_kit

private fun bitmapToInputArray(originalBitmap: Bitmap, quantified : Boolean = true): Array<Array<Array<FloatArray>>> {
    var bitmap = originalBitmap
    // [START mlkit_bitmap_input]
    //be sure it's the right size
    bitmap = Bitmap.createScaledBitmap(bitmap, IMAGE_WIDTH, IMAGE_HEIGHT, true)

    val batchNum = 0
    val input = Array(DIM_BATCH_SIZE) { Array(bitmap.width) { Array(bitmap.height) { FloatArray(DIM_PIXEL_SIZE) } } }
    for (x in 0..bitmap.width-1) {
        for (y in 0..bitmap.height-1) {
            val pixelValue = bitmap.getPixel(x, y)
            var red:Float
            var green:Float
            var blue:Float
            if(quantified) {
                red = (pixelValue shr 16 and 0xFF) / 255f
                green = (pixelValue shr 8 and 0xFF) / 255f
                blue = (pixelValue and 0xFF) / 255f
            } else {
                red = ((pixelValue shr 16 and 0xFF) - IMAGE_MEAN) / IMAGE_STD
                green= ((pixelValue shr 8 and 0xFF) - IMAGE_MEAN) / IMAGE_STD
                blue= ((pixelValue and 0xFF) - IMAGE_MEAN) / IMAGE_STD
            }

            input[batchNum][x][y][0] = red
            input[batchNum][x][y][1] = green
            input[batchNum][x][y][2] = blue
        }
    }
    // [END mlkit_bitmap_input]

    return input
}

And finally my ml_kit options looks like

@Throws(FirebaseMLException::class)
private fun createInputOutputOptions(): FirebaseModelInputOutputOptions {
    // [START mlkit_create_io_options]
    val inputOutputOptions = FirebaseModelInputOutputOptions.Builder()
        .setInputFormat(0, FirebaseModelDataType.FLOAT32, intArrayOf(DIM_BATCH_SIZE, IMAGE_WIDTH, IMAGE_HEIGHT, DIM_PIXEL_SIZE))
        .setOutputFormat(0, FirebaseModelDataType.FLOAT32, intArrayOf(DIM_BATCH_SIZE, 512))
        .build()
    // [END mlkit_create_io_options]
    return inputOutputOptions
}

And those are my final config settings

private val IMAGE_WIDTH : Int = 160
    private val IMAGE_HEIGHT : Int = 160

    private val DIM_BATCH_SIZE = 1
    private val DIM_PIXEL_SIZE = 3
    private val BYTES_PER_CHANNEL = 4

    private val IMAGE_MEAN = 127
    private val IMAGE_STD = 128.0f

Android TensorFlowLite buffer with 76800 bytes and a ByteBuffer with 307200 bytes

1 Answers