Base64 images with Keras and Google Cloud ML

Question

I\'m predicting image classes using Keras. It works in Google Cloud ML (GCML), but for efficiency need change it to pass base64 strings instead of json array. Related Docu

Answer 1

Another answer suggested adding tf.placeholder with type of tf.string, which makes sense, but how to incorporate that into the Keras model?

In Keras you can access your selected Backend (in this case Tensorflow) by doing:

from keras import backend as K

This you already seem to import on your code. That will enable you to access some native methods and resources available on the backend of your choice. It is the case that Keras backend includes a method for creating placeholders, among other utilities. Regarding placeholders, we can see what the Keras docs indicates about them:

placeholder

keras.backend.placeholder(shape=None, ndim=None, dtype=None, sparse=False, name=None)

Instantiates a placeholder tensor and returns it.

It also gives some example on its use:

>>> from keras import backend as K
>>> input_ph = K.placeholder(shape=(2, 4, 5))
>>> input_ph._keras_shape
(2, 4, 5)
>>> input_ph
<tf.Tensor 'Placeholder_4:0' shape=(2, 4, 5) dtype=float32>

As you can see, this is returning a Tensorflow tensor, with shape (2,4,5) and of dtype float. If you had another backend while doing the example you would get another tensor object (a Theano one surely). You can therefore use this placeholder() to adapt the solution you got on your previous question.

In conclusion, you can use your backend imported as K (or whatever you want) to do calls on the methods and objects available on the backend of your choice, by doing K.foo.bar() on the desired method. I suggest you give a read to what the Keras Backend to explore more things that can be useful for you on future situations.

Update: As per your edit. Yes, this placeholder should be a layer in your model. Specifically, it should be the Input Layer of your model, as it holds your decoded image (as Keras needs it that way) to classify.

Answer 2

first of all I use tf.keras but this should not be a big problem. So here is an example of how you can read a base64 decoded jpeg:

def preprocess_and_decode(img_str, new_shape=[299,299]):
    img = tf.io.decode_base64(img_str)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.resize_images(img, new_shape, method=tf.image.ResizeMethod.BILINEAR, align_corners=False)
    # if you need to squeeze your input range to [0,1] or [-1,1] do it here
    return img
InputLayer = Input(shape = (1,),dtype="string")
OutputLayer = Lambda(lambda img : tf.map_fn(lambda im : preprocess_and_decode(im[0]), img, dtype="float32"))(InputLayer)
base64_model = tf.keras.Model(InputLayer,OutputLayer)   

The code above creates a model that takes a jpeg of any size, resizes it to 299x299 and returns as 299x299x3 tensor. This model can be exported directly to saved_model and used for Cloud ML Engine serving. It is a little bit stupid, since the only thing it does is the convertion of base64 to tensor.

If you need to redirect the output of this model to the input of an existing trained and compiled model (e.g inception_v3) you have to do the following:

base64_input = base64_model.input
final_output = inception_v3(base64_model.output)
new_model = tf.keras.Model(base64_input,final_output)

This new_model can be saved. It takes base64 jpeg and returns classes identified by the inception_v3 part.