Transcript
simple-tensorflow-servingDocumentation
tobe
Nov 17, 2020
Contents:
1 Introduction 1
2 Installation 32.1 Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Bazel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.4 Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.5 Docker Compose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.6 Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Quick Start 5
4 Advanced Usage 94.1 Multiple Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 GPU Acceleration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3 Generated Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.4 Image Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.5 Custom Op . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.6 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.7 TSL/SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5 API 155.1 RESTful API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2 Python Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6 Models 176.1 TensorFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.2 MXNET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.3 ONNX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.4 Scikit-learn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.5 XGBoost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.6 PMML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.7 H2o . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
7 Clients 217.1 Bash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217.2 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
i
7.3 Golang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217.4 Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.5 JavaScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.6 PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.7 Erlang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237.8 Lua . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237.9 Perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237.10 R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.11 Postman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
8 Image Model 258.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258.2 Export Image Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258.3 Inference With Uploaded Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268.4 Inference with Python Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
9 Performance 29
10 Development 3110.1 Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3110.2 Debug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
ii
CHAPTER 1
Introduction
Simple TensorFlow Serving is the generic and easy-to-use serving service for machine learning models.
• [x] Support distributed TensorFlow models
• [x] Support the general RESTful/HTTP APIs
• [x] Support inference with accelerated GPU
• [x] Support curl and other command-line tools
• [x] Support clients in any programming language
• [x] Support code-gen client by models without coding
1
simple-tensorflow-serving Documentation
• [x] Support inference with raw file for image models
• [x] Support statistical metrics for verbose requests
• [x] Support serving multiple models at the same time
• [x] Support dynamic online and offline for model versions
• [x] Support loading new custom op for TensorFlow models
• [x] Support secure authentication with configurable basic auth
• [x] Support multiple models of TensorFlow/MXNet/PyTorch/Caffe2/CNTK/ONNX/H2o/Scikit-learn/XGBoost/PMML
2 Chapter 1. Introduction
CHAPTER 2
Installation
2.1 Pip
Install the server with pip.
pip install simple_tensorflow_serving
2.2 Source
Install from source code.
git clone https://github.com/tobegit3hub/simple_tensorflow_serving
cd ./simple_tensorflow_serving/
python ./setup.py install
2.3 Bazel
Install with bazel.
git clone https://github.com/tobegit3hub/simple_tensorflow_serving
cd ./simple_tensorflow_serving/
bazel build simple_tensorflow_serving:server
3
simple-tensorflow-serving Documentation
2.4 Docker
Deploy with docker image.
docker run -d -p 8500:8500 tobegit3hub/simple_tensorflow_serving
docker run -d -p 8500:8500 tobegit3hub/simple_tensorflow_serving:latest-gpu
docker run -d -p 8500:8500 tobegit3hub/simple_tensorflow_serving:latest-hdfs
docker run -d -p 8500:8500 tobegit3hub/simple_tensorflow_serving:latest-py34
2.5 Docker Compose
Deploy with docker-compose.
wget https://raw.githubusercontent.com/tobegit3hub/simple_tensorflow_serving/master/→˓docker-compose.yml
docker-compose up -d
2.6 Kubernetes
Deploy in Kubernetes cluster.
wget https://raw.githubusercontent.com/tobegit3hub/simple_tensorflow_serving/master/→˓simple_tensorflow_serving.yaml
kubectl create -f ./simple_tensorflow_serving.yaml
4 Chapter 2. Installation
CHAPTER 3
Quick Start
Train or download the TensorFlow SavedModel.
import tensorflow as tf
export_dir = "./model/1"input_keys_placeholder = tf.placeholder(
tf.int32, shape=[None, 1], name="input_keys")output_keys = tf.identity(input_keys_placeholder, name="output_keys")
session = tf.Session()tf.saved_model.simple_save(
session,export_dir,inputs={"keys": input_keys_placeholder},outputs={"keys": output_keys})
This script will export the model in ./model.
5
simple-tensorflow-serving Documentation
Start serving to load the model.
simple_tensorflow_serving --model_base_path="./model"
Check out the dashboard in http://127.0.0.1:8500 in web browser.
dashboard
Generate the clients for testing without coding.
curl http://localhost:8500/v1/models/default/gen_client?language=python > client.py
python ./client.py
6 Chapter 3. Quick Start
simple-tensorflow-serving Documentation
7
simple-tensorflow-serving Documentation
8 Chapter 3. Quick Start
CHAPTER 4
Advanced Usage
4.1 Multiple Models
It supports serve multiple models and multiple versions of these models. You can run the server with this configuration.
{"model_config_list": [{
"name": "tensorflow_template_application_model","base_path": "./models/tensorflow_template_application_model/","platform": "tensorflow"
}, {"name": "deep_image_model","base_path": "./models/deep_image_model/","platform": "tensorflow"
}, {"name": "mxnet_mlp_model","base_path": "./models/mxnet_mlp/mx_mlp","platform": "mxnet"
}]
}
simple_tensorflow_serving --model_config_file="./examples/model_config_file.json"
9
simple-tensorflow-serving Documentation
4.2 GPU Acceleration
If you want to use GPU, try with the docker image with GPU tag and put cuda files in /usr/cuda_files/.
export CUDA_SO="-v /usr/cuda_files/:/usr/cuda_files/"export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}')export LIBRARY_ENV="-e LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:/usr/local/→˓nvidia/lib:/usr/local/nvidia/lib64:/usr/cuda_files"
docker run -it -p 8500:8500 $CUDA_SO $DEVICES $LIBRARY_ENV tobegit3hub/simple_→˓tensorflow_serving:latest-gpu
You can set session config and gpu options in command-line parameter or the model config file.
simple_tensorflow_serving --model_base_path="./models/tensorflow_template_application_→˓model" --session_config='{"log_device_placement": true, "allow_soft_placement":→˓true, "allow_growth": true, "per_process_gpu_memory_fraction": 0.5}'
{"model_config_list": [{
"name": "default","base_path": "./models/tensorflow_template_application_model/","platform": "tensorflow","session_config": {
"log_device_placement": true,"allow_soft_placement": true,"allow_growth": true,"per_process_gpu_memory_fraction": 0.5
}}
]}
10 Chapter 4. Advanced Usage
simple-tensorflow-serving Documentation
Here is the benchmark of CPU and GPU inference and y-coordinate is the latency(the lower the better).
4.3 Generated Client
You can generate the test json data for the online models.
curl http://localhost:8500/v1/models/default/gen_json
Or generate clients in different languages(Bash, Python, Golang, JavaScript etc.) for your model without writing anycode.
curl http://localhost:8500/v1/models/default/gen_client?language=python > client.pycurl http://localhost:8500/v1/models/default/gen_client?language=bash > client.shcurl http://localhost:8500/v1/models/default/gen_client?language=golang > client.gocurl http://localhost:8500/v1/models/default/gen_client?language=javascript > client.→˓js
The generated code should look like these which can be test immediately.
#!/usr/bin/env python
import requests
def main():endpoint = "http://127.0.0.1:8500"
input_data = {"keys": [[1.0], [1.0]], "features": [[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.→˓0, 1.0, 1.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]]}result = requests.post(endpoint, json=input_data)print(result.json())
(continues on next page)
4.3. Generated Client 11
simple-tensorflow-serving Documentation
(continued from previous page)
if __name__ == "__main__":main()
4.4 Image Model
For image models, we can request with the raw image files instead of constructing array data.
Now start serving the image model like deep_image_model.
simple_tensorflow_serving --model_base_path="./models/deep_image_model/"
Then request with the raw image file which has the same shape of your model.
curl -X POST -F 'image=@./images/mew.jpg' -F "model_version=1" 127.0.0.1:8500
4.5 Custom Op
If your models rely on new TensorFlow custom op, you can run the server while loading the so files.
simple_tensorflow_serving --model_base_path="./model/" --custom_op_paths="./foo_op/"
Please check out the complete example in ./examples/custom_op/.
4.6 Authentication
For enterprises, we can enable basic auth for all the APIs and any anonymous request is denied.
Now start the server with the configured username and password.
./server.py --model_base_path="./models/tensorflow_template_application_model/" --→˓enable_auth=True --auth_username="admin" --auth_password="admin"
If you are using the Web dashboard, just type your certification. If you are using clients, give the username andpassword within the request.
curl -u admin:admin -H "Content-Type: application/json" -X POST -d '{"data": {"keys":→˓[[11.0], [2.0]], "features": [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1,→˓1]]}}' http://127.0.0.1:8500
endpoint = "http://127.0.0.1:8500"input_data = {
"data": {"keys": [[11.0], [2.0]],"features": [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1]]
}}auth = requests.auth.HTTPBasicAuth("admin", "admin")result = requests.post(endpoint, json=input_data, auth=auth)
12 Chapter 4. Advanced Usage
simple-tensorflow-serving Documentation
4.7 TSL/SSL
It supports TSL/SSL and you can generate the self-signed secret files for testing.
openssl req -x509 -newkey rsa:4096 -nodes -out /tmp/secret.pem -keyout /tmp/secret.→˓key -days 365
Then run the server with certification files.
simple_tensorflow_serving --enable_ssl=True --secret_pem=/tmp/secret.pem --secret_→˓key=/tmp/secret.key --model_base_path="./models/tensorflow_template_application_→˓model"
4.7. TSL/SSL 13
simple-tensorflow-serving Documentation
14 Chapter 4. Advanced Usage
CHAPTER 5
API
5.1 RESTful API
The most import API is inference for the loaded models.
Endpoint: /Method: POSTJSON: {
"model_name": "default","model_version": 1,"data": {
"keys": [[11.0], [2.0]],"features": [[1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1]]}
}Response: {
"keys": [[1], [1]]}
5.2 Python Example
You can easily choose the specified model and version for inference.
import requests
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
(continues on next page)
15
simple-tensorflow-serving Documentation
(continued from previous page)
"keys": [[11.0], [2.0]],"features": [[1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1]]}
}result = requests.post(endpoint, json=input_data)
16 Chapter 5. API
CHAPTER 6
Models
6.1 TensorFlow
For TensorFlow models, you can load with commands and configuration like these.
simple_tensorflow_serving --model_base_path="./models/tensorflow_template_application_→˓model" --model_platform="tensorflow"
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
"data": [[12.0, 2.0]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
6.2 MXNET
For MXNet models, you can load with commands and configuration like these.
simple_tensorflow_serving --model_base_path="./models/mxnet_mlp/mx_mlp" --model_→˓platform="mxnet"
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
(continues on next page)
17
simple-tensorflow-serving Documentation
(continued from previous page)
"data": [[12.0, 2.0]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
6.3 ONNX
For ONNX models, you can load with commands and configuration like these.
simple_tensorflow_serving --model_base_path="./models/onnx_mnist_model/onnx_model.→˓proto" --model_platform="onnx"
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
"data": [[...]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
6.4 Scikit-learn
For Scikit-learn models, you can load with commands and configuration like these.
simple_tensorflow_serving --model_base_path="./models/scikitlearn_iris/model.joblib" -→˓-model_platform="scikitlearn"
simple_tensorflow_serving --model_base_path="./models/scikitlearn_iris/model.pkl" --→˓model_platform="scikitlearn"
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
"data": [[...]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
6.5 XGBoost
For XGBoost models, you can load with commands and configuration like these.
18 Chapter 6. Models
simple-tensorflow-serving Documentation
simple_tensorflow_serving --model_base_path="./models/xgboost_iris/model.bst" --model_→˓platform="xgboost"
simple_tensorflow_serving --model_base_path="./models/xgboost_iris/model.joblib" --→˓model_platform="xgboost"
simple_tensorflow_serving --model_base_path="./models/xgboost_iris/model.pkl" --model_→˓platform="xgboost"
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
"data": [[...]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
6.6 PMML
For PMML models, you can load with commands and configuration like these. This relies on Openscoring andOpenscoring-Python to load the models.
java -jar ./third_party/openscoring/openscoring-server-executable-1.4-SNAPSHOT.jar
simple_tensorflow_serving --model_base_path="./models/pmml_iris/DecisionTreeIris.pmml→˓" --model_platform="pmml"
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
"data": [[...]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
6.7 H2o
For H2o models, you can load with commands and configuration like these.
# Start H2o server with "java -jar h2o.jar"
simple_tensorflow_serving --model_base_path="./models/h2o_prostate_model/GLM_model_→˓python_1525255083960_17" --model_platform="h2o"
6.6. PMML 19
simple-tensorflow-serving Documentation
endpoint = "http://127.0.0.1:8500"input_data = {
"model_name": "default","model_version": 1,"data": {
"data": [[...]]}
}result = requests.post(endpoint, json=input_data)print(result.text)
20 Chapter 6. Models
CHAPTER 7
Clients
7.1 Bash
Here is the example client in Bash.
curl -H "Content-Type: application/json" -X POST -d '{"data": {"keys": [[1.0], [2.0]],→˓ "features": [[10, 10, 10, 8, 6, 1, 8, 9, 1], [6, 2, 1, 1, 1, 1, 7, 1, 1]]}}' http:/→˓/127.0.0.1:8500
7.2 Python
Here is the example client in Python.
endpoint = "http://127.0.0.1:8500"payload = {"data": {"keys": [[11.0], [2.0]], "features": [[1, 1, 1, 1, 1, 1, 1, 1, 1],→˓ [1, 1, 1, 1, 1, 1, 1, 1, 1]]}}
result = requests.post(endpoint, json=payload)
7.3 Golang
endpoint := "http://127.0.0.1:8500"dataByte := []byte(`{"data": {"keys": [[11.0], [2.0]], "features": [[1, 1, 1, 1, 1, 1,→˓ 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1]]}}`)var dataInterface map[string]interface{}json.Unmarshal(dataByte, &dataInterface)dataJson, _ := json.Marshal(dataInterface)
resp, err := http.Post(endpoint, "application/json", bytes.NewBuffer(dataJson))
21
simple-tensorflow-serving Documentation
7.4 Ruby
Here is the example client in Ruby.
endpoint = "http://127.0.0.1:8500"uri = URI.parse(endpoint)header = {"Content-Type" => "application/json"}input_data = {"data" => {"keys"=> [[11.0], [2.0]], "features"=> [[1, 1, 1, 1, 1, 1, 1,→˓ 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1]]}}http = Net::HTTP.new(uri.host, uri.port)request = Net::HTTP::Post.new(uri.request_uri, header)request.body = input_data.to_json
response = http.request(request)
7.5 JavaScript
Here is the example client in JavaScript.
var options = {uri: "http://127.0.0.1:8500",method: "POST",json: {"data": {"keys": [[11.0], [2.0]], "features": [[1, 1, 1, 1, 1, 1, 1, 1, 1],
→˓ [1, 1, 1, 1, 1, 1, 1, 1, 1]]}}};
request(options, function (error, response, body) {});
7.6 PHP
Here is the example client in PHP.
$endpoint = "127.0.0.1:8500";$inputData = array(
"keys" => [[11.0], [2.0]],"features" => [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1]],
);$jsonData = array(
"data" => $inputData,);$ch = curl_init($endpoint);curl_setopt_array($ch, array(
CURLOPT_POST => TRUE,CURLOPT_RETURNTRANSFER => TRUE,CURLOPT_HTTPHEADER => array(
"Content-Type: application/json"),CURLOPT_POSTFIELDS => json_encode($jsonData)
));
$response = curl_exec($ch);
22 Chapter 7. Clients
simple-tensorflow-serving Documentation
7.7 Erlang
Here is the example client in Erlang.
ssl:start(),application:start(inets),httpc:request(post,
{"http://127.0.0.1:8500", [],"application/json","{\"data\": {\"keys\": [[11.0], [2.0]], \"features\": [[1, 1, 1, 1, 1, 1, 1, 1, 1],
→˓[1, 1, 1, 1, 1, 1, 1, 1, 1]]}}"}, [], []).
7.8 Lua
Here is the example client in Lua.
local endpoint = "http://127.0.0.1:8500"keys_array = {}keys_array[1] = {1.0}keys_array[2] = {2.0}features_array = {}features_array[1] = {1, 1, 1, 1, 1, 1, 1, 1, 1}features_array[2] = {1, 1, 1, 1, 1, 1, 1, 1, 1}local input_data = {
["keys"] = keys_array,["features"] = features_array,
}local json_data = {
["data"] = input_data}request_body = json:encode (json_data)local response_body = {}
local res, code, response_headers = http.request{url = endpoint,method = "POST",headers =
{["Content-Type"] = "application/json";["Content-Length"] = #request_body;
},source = ltn12.source.string(request_body),sink = ltn12.sink.table(response_body),
}
7.9 Perl
Here is the example client in Perl.
my $endpoint = "http://127.0.0.1:8500";my $json = '{"data": {"keys": [[11.0], [2.0]], "features": [[1, 1, 1, 1, 1, 1, 1, 1,→˓1], [1, 1, 1, 1, 1, 1, 1, 1, 1]]}}'; (continues on next page)
7.7. Erlang 23
simple-tensorflow-serving Documentation
(continued from previous page)
my $req = HTTP::Request->new( 'POST', $endpoint );$req->header( 'Content-Type' => 'application/json' );$req->content( $json );$ua = LWP::UserAgent->new;
$response = $ua->request($req);
7.10 R
Here is the example client in R.
endpoint <- "http://127.0.0.1:8500"body <- list(data = list(a = 1), keys = 1)json_data <- list(
data = list(keys = list(list(1.0), list(2.0)), features = list(list(1, 1, 1, 1, 1, 1, 1, 1,
→˓1), list(1, 1, 1, 1, 1, 1, 1, 1, 1)))
)
r <- POST(endpoint, body = json_data, encode = "json")stop_for_status(r)content(r, "parsed", "text/html")
7.11 Postman
Here is the example with Postman.
24 Chapter 7. Clients
CHAPTER 8
Image Model
8.1 Introduction
Simple TensorFlow Serving has extra support for image models. You can deploy the image models easily and makeinferences by uploading the image files in web browser or using form-data. The best practice is accepting base64strings as input of model signature like this.
inputs {key: "images"value {name: "model_input_b64_images:0"dtype: DT_STRINGtensor_shape {
dim {size: -1
}}
}}
8.2 Export Image Model
Model images should be standard TensorFlow SavedModel as well. We do not use [batch_size, r, g, b]or [batch_size, r, b, g] as signature input because it is not compatible with arbitrary image files. We canaccept the base64 strings as input, then decode and resize the tensor for the required model input.
# Define modeldef inference(input):weights = tf.get_variable(
"weights", [784, 10], initializer=tf.random_normal_initializer())bias = tf.get_variable(
(continues on next page)
25
simple-tensorflow-serving Documentation
(continued from previous page)
"bias", [10], initializer=tf.random_normal_initializer())logits = tf.matmul(input, weights) + bias
return logits
# Define op for model signaturetf.get_variable_scope().reuse_variables()
model_base64_placeholder = tf.placeholder(shape=[None], dtype=tf.string, name="model_input_b64_images")
model_base64_string = tf.decode_base64(model_base64_placeholder)model_base64_input = tf.map_fn(lambda x: tf.image.resize_images(tf.image.decode_→˓jpeg(x, channels=1), [28, 28]), model_base64_string, dtype=tf.float32)model_base64_reshape_input = tf.reshape(model_base64_input, [-1, 28 * 28])model_logits = inference(model_base64_reshape_input)model_predict_softmax = tf.nn.softmax(model_logits)model_predict = tf.argmax(model_predict_softmax, 1)
# Export modelexport_dir = "./model/1"tf.saved_model.simple_save(
sess,export_dir,inputs={"images": model_base64_placeholder},outputs={
"predict": model_predict,"probability": model_predict_softmax
})
8.3 Inference With Uploaded Files
Now we can start Simple TensorFlow Serving and load the image models easily. Take the deep_image_model forexample.
git clone https://github.com/tobegit3hub/simple_tensorflow_serving
cd ./simple_tensorflow_serving/models/
simple_tensorflow_serving --model_base_path="./deep_image_model"
Then you can choose the local image file to make inference.
26 Chapter 8. Image Model
simple-tensorflow-serving Documentation
8.4 Inference with Python Client
If you want to make inferences with Python client. You can encode the image file with the base64 library.
import requestsimport base64
def main():image_string = base64.urlsafe_b64encode(open("./test.png", "rb").read())
endpoint = "http://127.0.0.1:8500"json_data = {"model_name": "default", "data": {"images": [image_string]} }result = requests.post(endpoint, json=json_data)print(result.json())
if __name__ == "__main__":main()
Here is the example data of one image’s base64 string.
{"images": ["_9j_4AAQSkZJRgABAQAASABIAAD_→˓4QCMRXhpZgAATU0AKgAAAAgABQESAAMAAAABAAEAAAEaAAUAAAABAAAASgEbAAUAAAABAAAAUgEoAAMAAAABAAIAAIdpAAQAAAABAAAAWgAAAAAAAABIAAAAAQAAAEgAAAABAAOgAQADAAAAAQABAACgAgAEAAAAAQAAACCgAwAEAAAAAQAAACAAAAAA_→˓-→˓EJIWh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8APD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4gPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPiA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPiA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIi8-→˓IDwvcmRmOlJERj4gPC94OnhtcG1ldGE-→˓ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPD94cGFja2V0IGVuZD0idyI_→˓PgD_7QA4UGhvdG9zaG9wIDMuMAA4QklNBAQAAAAAAAA4QklNBCUAAAAAABDUHYzZjwCyBOmACZjs-EJ-_→˓8AAEQgAIAAgAwEiAAIRAQMRAf_EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC__→˓EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29_→˓j5-v_EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC__→˓EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5-→˓jp6vLz9PX29_j5-v_→˓bAEMAAgICAgICAwICAwUDAwMFBgUFBQUGCAYGBgYGCAoICAgICAgKCgoKCgoKCgwMDAwMDA4ODg4ODw8PDw8PDw8PD_→˓_→˓bAEMBAgICBAQEBwQEBxALCQsQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEP_→˓dAAQAAv_aAAwDAQACEQMRAD8A_bzx547t_CFokNtH9s1W6B-z2wySe29gOdoPpye2ACRwWg-ItctVm1_→˓xXHCbxSUhUSkgI33m4BHsAMADnnPGz4c8S-→˓EdOW71nX9Sgt9e1F2NykjbbiFVYiO3VCN4Ea4GAPmbLc5zXkus3UtnfXbxW99fQtHCbZZbcxiTYu0sqPtdVc4OWUDrg4FW1ZWPoMtp03enKOvfv5eSPprwxr48Qac14U8pkkZCMjkA5VupxuUgjmuj3A9DXy18K9M8cx2EOiLcjRrGDahitIo3liTbhN8tyZC2MYJVevvXp-→˓uJe-Hjb2mm69f32tXzKtrbSmGQPgje7oI12xKMl2yMdAdxAK5ddDzsXheWq4bH_9D9-ti7t-BuHfvXJ-→˓Nbyz0nw5falckQoBHHJJjkI8iocn0-b8K66snXdJh1zSLrSp-FuUK564PVT-→˓BANBrQmozTe1z5mm8Q3UTzajoy3aYt2uI7mCAyKmR1JYbADwfm-UjrXvHgjw_o-m6cmr2dxJqd5qcccs-→˓oXDb57jIyMnoij-GNAEXoBU-→˓g6VqcfhY6RrCp5hSaJY1bzFWI5CKWIGcLx06cVF4Jvo57GW0jdZPszDIXGELjLIQOhVs8dQCMintoepmGL9unLa2nqj_→˓_2Q=="]}
(continues on next page)
8.4. Inference with Python Client 27
simple-tensorflow-serving Documentation
(continued from previous page)
28 Chapter 8. Image Model
CHAPTER 9
Performance
You can run SimpleTensorFlowServing with any WSGI server for better performance. We have benchmarked andcompare with TensorFlow Serving. Find more details in benchmark directory.
STFS(Simple TensorFlow Serving) and TFS(TensorFlow Serving) have similar performances for different models.Vertical coordinate is inference latency(microsecond) and the less is better.
Then we test with ab with concurrent clients in CPU and GPU. TensorFlow Serving works better especiallywith GPUs.
29
simple-tensorflow-serving Documentation
For simplest model, each request only costs ~1.9 microseconds and one instance of Simple TensorFlow Serving canachieve 5000+ QPS. With larger batch size, it can inference more than 1M instances per second.
30 Chapter 9. Performance
CHAPTER 10
Development
10.1 Principle
1. simple_tensorflow_serving starts the HTTP server with flask application.
2. Load the TensorFlow models with tf.saved_model.loader Python API.
3. Construct the feed_dict data from the JSON body of the request.
// Method: POST, Content-Type: application/json{"model_version": 1, // Optional"data": {
"keys": [[1], [2]],"features": [[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], [1.0, 2.0, 3.0, 4.
→˓0, 5.0, 6.0, 7.0, 8.0, 9.0]]}
}
4. Use the TensorFlow Python API to sess.run() with feed_dict data.
5. For multiple versions supported, it starts independent thread to load models.
6. For generated clients, it reads user’s model and render code with Jinja templates.
10.2 Debug
You can install the server with develop and test when code changes.
git clone https://github.com/tobegit3hub/simple_tensorflow_serving
cd ./simple_tensorflow_serving/
python ./setup.py develop
31
top related