To compile and use TensorFlow Serving, you need to set up some prerequisites.
Bazel (only if compiling source code)
TensorFlow Serving requires Bazel 0.5.4 or higher. You can find the Bazel installation instructions here.
If you have the prerequisites for Bazel, those instructions consist of the following steps:
Download the relevant binary from here. Let's say you downloaded bazel-0.5.4-installer-linux-x86_64.sh. You would execute:
cd ~/Downloads chmod +x bazel-0.5.4-installer-linux-x86_64.sh ./bazel-0.5.4-installer-linux-x86_64.sh --user
Set up your environment. Put this in your ~/.bashrc.
To install TensorFlow Serving dependencies, execute the following:
sudo apt-get update && sudo apt-get install -y \ automake \ build-essential \ curl \ libcurl3-dev \ git \ libtool \ libfreetype6-dev \ libpng12-dev \ libzmq3-dev \ pkg-config \ python-dev \ python-numpy \ python-pip \ software-properties-common \ swig \ zip \ zlib1g-dev
The list of packages needed to build TensorFlow changes over time, so if you
encounter any issues, refer TensorFlow's build
particular attention to
apt-get install and
pip install commands which you
may need to run.
TensorFlow Serving Python API PIP package
To run Python client code without the need to install Bazel, you can install the
tensorflow-serving-api PIP package using:
pip install tensorflow-serving-api
Installing using apt-get
The TensorFlow Serving ModelServer binary is available in two variants:
tensorflow-model-server: Fully optimized server that uses some platform specific compiler optimizations like SSE4 and AVX instructions. This should be the preferred option for most users, but may not work on some older machines.
tensorflow-model-server-universal: Compiled with basic optimizations, but
doesn't include platform specific instruction sets, so should work on most if
not all machines out there. Use this if
tensorflow-model-server does not work
for you. Note that the binary name is the same for both packages, so if you
already installed tensorflow-model-server, you should first uninstall it using
sudo apt-get remove tensorflow-model-server
Installing the ModelServer
Add TensorFlow Serving distribution URI as a package source (one time setup)
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
Install and update TensorFlow ModelServer
sudo apt-get update && sudo apt-get install tensorflow-model-server
Once installed, the binary can be invoked using the command
You can upgrade to a newer version of tensorflow-model-server with:
sudo apt-get upgrade tensorflow-model-server
Installing from source
Clone the TensorFlow Serving repository
git clone https://github.com/tensorflow/serving cd serving
Note that these instructions will install the latest master branch of TensorFlow
Serving. If you want to install a specific branch (such as a release branch),
-b <branchname> to the
git clone command.
Follow the Prerequisites section above to install all dependencies. Consult the TensorFlow install instructions if you encounter any issues with setting up TensorFlow or its dependencies.
TensorFlow Serving uses Bazel to build. Use Bazel commands to build individual targets or the entire source tree.
To build the entire tree, execute:
bazel build -c opt tensorflow_serving/...
Binaries are placed in the bazel-bin directory, and can be run using a command like:
To test your installation, execute:
bazel test -c opt tensorflow_serving/...
It's possible to compile using some platform specific instruction sets (e.g.
AVX) that can significantly improve performance. Wherever you see 'bazel build'
in the documentation, you can add the flags
-c opt --copt=-msse4.1
--copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3
--cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" (or some subset of these flags). For
bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" tensorflow_serving/...