Converting UIImage to MLMultiArray for Keras Model

Question

In Python, I trained an image classification model with keras to receive input as a [224, 224, 3] array and output a prediction (1 or 0). When I load the save the model and load it into xcode, it states that the input has to be in MLMultiArray format.

Is there a way for me to convert a UIImage into MLMultiArray format? Or is there a way for me change my keras model to accept CVPixelBuffer type objects as an input.

Answer 1

In your Core ML conversion script you can supply the parameter image_input_names='data' where data is the name of your input.

Now Core ML will treat this input as an image (CVPixelBuffer) instead of a multi-array.

Answer 2

When you convert the caffe model to MLModel, you need to add this line:

image_input_names = 'data'

Take my own transfer script as an example, the script should be like this:

import coremltools
coreml_model = coremltools.converters.caffe.convert(('gender_net.caffemodel', 
'deploy_gender.prototxt'),
image_input_names = 'data',
class_labels = 'genderLabel.txt')
coreml_model.save('GenderMLModel.mlmodel')

And then your MLModel's input data will be CVPixelBufferRef instead of MLMultiArray. Transferring UIImage to CVPixelBufferRef would be an easy thing.

Answer 3

Did not tried this, but here is how its done for the FOOD101 sample

func preprocess(image: UIImage) -> MLMultiArray? {
        let size = CGSize(width: 299, height: 299)


        guard let pixels = image.resize(to: size).pixelData()?.map({ (Double($0) / 255.0 - 0.5) * 2 }) else {
            return nil
        }

        guard let array = try? MLMultiArray(shape: [3, 299, 299], dataType: .double) else {
            return nil
        }

        let r = pixels.enumerated().filter { $0.offset % 4 == 0 }.map { $0.element }
        let g = pixels.enumerated().filter { $0.offset % 4 == 1 }.map { $0.element }
        let b = pixels.enumerated().filter { $0.offset % 4 == 2 }.map { $0.element }

        let combination = r + g + b
        for (index, element) in combination.enumerated() {
            array[index] = NSNumber(value: element)
        }

        return array
    }

https://github.com/ph1ps/Food101-CoreML