More efficient way to send a request than JSON to deployed tensorflow model in Sagemaker?

问题

I have trained a tf.estimator based TensorFlow model in Sagemaker and deployed it and it works fine.

But I can only send requests to it in JSON format. I need to send some big input tensors and this seems very inefficient and also quickly breaks InvokeEndpoints 5MB request limit.

Is it possible to use a more effective format against the tensorflow serving based endpoint?

I tried sending a protobuf based request:

from sagemaker.tensorflow.serving import Model 
from sagemaker.tensorflow.tensorflow_serving.apis import predict_pb2 
from sagemaker.tensorflow.predictor import tf_serializer, tf_deserializer

role = 'xxx'

model = Model('s3://xxx/tmp/artifacts/sagemaker-tensorflow-scriptmode-xxx/output/model.tar.gz', role)

predictor = model.deploy(initial_instance_count=1, instance_type='ml.c5.xlarge', endpoint_name='test-endpoint')

# this predictor has json serializer, make a new one pred = 

RealTimePredictor('test-endpoint', serializer=tf_serializer, deserializer=tf_deserializer)

req = predict_pb2.PredictRequest()

req.inputs['instances'].CopyFrom(tf.make_tensor_proto(np.zeros((4, 36, 64)), shape=(4, 36, 64)))

predictor.predict(req)

Which results in the following error:

---------------------------------------------------------------------------
ModelError                                Traceback (most recent call last)
<ipython-input-40-5ba7f281bd0d> in <module>()
----> 1 predictor.predict(req)

~/anaconda3/envs/default/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args)
     76 
     77         request_args = self._create_request_args(data, initial_args)
---> 78         response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
     79         return self._handle_response(response)
     80 

~/anaconda3/envs/default/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    355                     "%s() only accepts keyword arguments." % py_operation_name)
    356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
    358 
    359         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/default/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    659             error_code = parsed_response.get("Error", {}).get("Code")
    660             error_class = self.exceptions.from_code(error_code)
--> 661             raise error_class(parsed_response, operation_name)
    662         else:
    663             return parsed_response

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (415) from model with message "{"error": "Unsupported Media Type: application/octet-stream"}".

Is JSON the only available query format for deployed TensorFlow models ?

回答1:

Have you looked at batch transform? If you don't need actually an HTTPS endpoint, this might solve your problem:

https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-batch-transform.html

来源：https://stackoverflow.com/questions/54090270/more-efficient-way-to-send-a-request-than-json-to-deployed-tensorflow-model-in-s

标签

python

tensorflow

amazon-sagemaker