CoreML frameworks generated different results from coremltools

Question

I have been developing image recognition app in ios 11 following the CoreML examples. However i notice that there is difference of the results when calling the model in ios, and the ones using coremltools in mac/python. I think the difference may lie on the image loading part. Python code use Pillow to load image, but xcode use CoreImage. I pasted the key codes as below. Hopefully somebody can help to point out the issue.

Also the input image is a 299*299 jpg. So should not any resizing happened in either of the implementation. Thank you.

python codes

import coremltools  
from PIL import Image  
from keras.preprocessing import image  
import numpy as np  

IMG_PATH='./test.jpg'  
img = image.load_img(IMG_PATH)  
model=coremltools.models.MLModel("./Inceptionv3.mlmodel")  
res = model.predict({'image':img})

ios codes

self.image = [CIImage imageWithContentsOfURL:fileURL];  
self.model = [[[Inceptionv3 alloc] init] model];  

VNCoreMLModel *m = [VNCoreMLModel modelForMLModel: self.model error:nil];  
VNCoreMLRequest *rq = [[VNCoreMLRequest alloc] initWithModel: m completionHandler: (VNRequestCompletionHandler) ^(VNRequest *request, NSError *error){  
    NSArray *results = [request.results copy];  
    NSString *top_results = @"";  
    for(int index = 0; index < kNumResults; index++)  
    {  
        VNClassificationObservation *res = ((VNClassificationObservation *)(results[index]));  
         NSString *tmp = [top_results stringByAppendingFormat: @"- %d %.4f %@\n ", index, res.confidence,res.identifier];  
         top_results = [tmp copy];  
    }  
    self.label_prob = [top_results copy];  
}];  

NSDictionary *d = [[NSDictionary alloc] init];  
NSArray *a = @[rq];  
VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCIImage:self.image options:d];  

dispatch_queue_t myCustomQueue;  
myCustomQueue = dispatch_queue_create("com.example.MyCustomQueue", NULL);  

dispatch_sync(myCustomQueue, ^{  
    [handler performRequests:a error:nil];  
});

The differences:

CoreML top-5 military uniform: 0.254365 suit, suit of clothes: 0.198099 Windsor tie: 0.077577 bulletproof vest: 0.068461 comic book: 0.022226

coremltools top-5 military uniform: 0.458214044571 bulletproof vest: 0.115854650736 suit, suit of clothes: 0.115854650736 Windsor tie: 0.0413092523813 pickelhaube: 0.0201325211674

Test image original was pre-resized to 299*299 for testing.

"However i notice that there is difference of the results..." Can you elaborate on what this difference is? — Knowledge Cube
#CoreML top-5 military uniform: 0.254365 suit, suit of clothes: 0.198099 Windsor tie: 0.077577 bulletproof vest: 0.068461 comic book: 0.022226 #coremltools top-5 military uniform: 0.458214044571 bulletproof vest: 0.115854650736 suit, suit of clothes: 0.115854650736 Windsor tie: 0.0413092523813 pickelhaube: 0.0201325211674 Test image is at link, which was pre-resized to 299*299 for testing. — Simon Chen

rob rob · Accepted Answer · 2018-07-06T21:11:03

I had a similar issue, but with a model created with Xcode 10's CreateML tools. While CreateML gave me excellent precision and recall, I saw very degraded performance when using the model with the Vision framework.

By chance I discovered that I could get better performance if I converted the image to data before I passed it to the request handler. That is:

Poor performance: let handler = VNImageRequestHandler(cgImage: myCGImage, options: [:])

Good performance: let imageData = UIImagePNGRepresentation(UIImage(cgImage: myCGImage)!)! let handler = VNImageRequestHandler(data: imageData, options: [:])

Not sure why this is.

CoreML frameworks generated different results from coremltools

python codes

ios codes

2 Answers