CoreML Model spec - change output type to dictionary

Question

For the past week I have been struggling to run inference on a classifier I built using Google's AutoML Vision tool.

At first I thought everything would go smoothly because Google allows to export a CoreML version of the final model. I assumed I would only need to use Apple's CoreML library to make it work. When I export the model Google provides a .mlmodel file and a dict.txt file with the classification labels. For the current model I have 100 labels.

This is my Swift code to run inference on the model.

private lazy var classificationRequest: VNCoreMLRequest = {
        do {

            let classificationModel = try VNCoreMLModel(for: NewGenusModel().model)

            let request = VNCoreMLRequest(model: classificationModel, completionHandler: { [weak self] request, error in
                self?.processClassifications(for: request, error: error)
            })

            request.imageCropAndScaleOption = .scaleFit
            return request
        }
        catch {
            fatalError("Error! Can't use Model.")
        }
    }()

    func classifyImage(receivedImage: UIImage) {

        let orientation = CGImagePropertyOrientation(rawValue: UInt32(receivedImage.imageOrientation.rawValue))

        if let image = CIImage(image: receivedImage) {
            DispatchQueue.global(qos: .userInitiated).async {

                let handler = VNImageRequestHandler(ciImage: image, orientation: orientation!)
                do {
                    try handler.perform([self.classificationRequest])
                }
                catch {
                    fatalError("Error classifying image!")
                }
            }
        }
    }

The problem started when I tried to pass a UIImage to run inference on the model. The input type of the original model was MultiArray (Float32 1 x 224 x 224 x 3). Using Coremltools library I was able to convert the input type to Image (Color 224 x 224) using Python.

This worked and here is my code:

import coremltools
import coremltools.proto.FeatureTypes_pb2 as ft

spec = coremltools.utils.load_spec("model.mlmodel")

input = spec.description.input[0]
input.type.imageType.colorSpace = ft.ImageFeatureType.RGB
input.type.imageType.height = 224
input.type.imageType.width = 224

coremltools.utils.save_spec(spec, "newModel.mlmodel")

My problem now is with the output type. I want to be able to access the confidence of the classification as well as the result label of the classification. Again using coremltools I was able to to access the output description and I got this.

name: "scores"
type {
  multiArrayType {
    dataType: FLOAT32
  }
}

I am trying to change it this way:

f = open("dict.txt", "r")
labels = f.read()

class_labels = labels.splitlines()
print(class_labels)
class_labels = class_labels[1:] # remove the first class which is background
assert len(class_labels) == 57

# make sure entries of class_labels are strings
for i, label in enumerate(class_labels):
  if isinstance(label, bytes):
    class_labels[i] = label.decode("utf8")

#classifier_config = ct.ClassifierConfig(class_labels)

output = spec.description.output[0]
output.type = ft.DictionaryFeatureType

Unfortunately this is not working and I can't find any information that can help me.

Did you find solution regarding this above issue, i have same issue with google autoML with use or coreML model — Akshay Savaliya

Matthijs Hollemans Matthijs Hollemans · Accepted Answer · 2020-09-19T16:52:12

The ct.ClassifierConfig is only useful at the point you are converting your model, i.e. when you call ct.convert(...).

If you already have an mlmodel, you cannot change the class labels this way. You have to change them inside the mlmodel object using the spec.

Given that spec, you can do:

labels = spec.neuralNetworkClassifier.stringClassLabels 
del labels.vector[:]
new_labels = [ ... ] 
labels.vector.extend(new_labels)

coremltools.utils.save_spec(spec, "YourNewModel.mlmodel")

Where new_labels is a list of the labels.

CoreML Model spec - change output type to dictionary

1 Answers