TensorFlow 2.0 RC is available Learn more

Post-training integer quantization

View on TensorFlow.org Run in Google Colab View source on GitHub

Overview

TensorFlow Lite now supports converting an entire model (weights and activations) to 8-bit during model conversion from TensorFlow to TensorFlow Lite's flat buffer format. This results in a 4x reduction in model size and a 3 to 4x performance improvement on CPU performance. In addition, this fully quantized model can be consumed by integer-only hardware accelerators.

In contrast to post-training "on-the-fly" quantization , which only stores weights as 8-bit ints, in this technique all weights and activations are quantized statically during model conversion.

In this tutorial, you train an MNIST model from scratch, check its accuracy in TensorFlow, and then convert the saved model into a Tensorflow Lite flatbuffer with full quantization. Finally, check the accuracy of the converted model and compare it to the original saved model. The training script, mnist.py, is available from the TensorFlow official MNIST tutorial.

Build an MNIST model

Setup

! pip uninstall -y tensorflow
! pip install -q -U tf-nightly
WARNING: Skipping tensorflow as it is not installed.
ERROR: tensorflow-gpu 2.0.0b1 has requirement tb-nightly<1.14.0a20190604,>=1.14.0a20190603, but you'll have tb-nightly 1.15.0a20190802 which is incompatible.
import tensorflow as tf
tf.enable_eager_execution()
WARNING: Logging before flag parsing goes to stderr.
W0802 17:55:29.575603 139925293250304 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.enable_eager_execution is deprecated. Please use tf.compat.v1.enable_eager_execution instead.

! git clone --depth 1 https://github.com/tensorflow/models
Cloning into 'models'...
remote: Enumerating objects: 3224, done.
remote: Counting objects: 100% (3224/3224), done.
remote: Compressing objects: 100% (2726/2726), done.
remote: Total 3224 (delta 587), reused 2067 (delta 421), pack-reused 0
Receiving objects: 100% (3224/3224), 370.68 MiB | 42.83 MiB/s, done.
Resolving deltas: 100% (587/587), done.
Checking out files: 100% (3053/3053), done.
import sys
import os

if sys.version_info.major >= 3:
    import pathlib
else:
    import pathlib2 as pathlib

# Add `models` to the python path.
models_path = os.path.join(os.getcwd(), "models")
sys.path.append(models_path)

Train and export the model

saved_models_root = "/tmp/mnist_saved_model"
# The above path addition is not visible to subprocesses, add the path for the subprocess as well.
# Note: channels_last is required here or the conversion may fail. 
!PYTHONPATH={models_path} python models/official/mnist/mnist.py --train_epochs=1 --export_dir {saved_models_root} --data_format=channels_last
WARNING: Logging before flag parsing goes to stderr.
W0802 17:55:47.553759 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0802 17:55:47.556486 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

I0802 17:55:47.557640 140144997107456 run_config.py:558] Initializing RunConfig with distribution strategies.
I0802 17:55:47.557827 140144997107456 estimator_training.py:167] Not using Distribute Coordinator.
I0802 17:55:47.558345 140144997107456 estimator.py:209] Using config: {'_model_dir': '/tmp/mnist_model', '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategyV1 object at 0x7f75dc6854a8>, '_num_ps_replicas': 0, '_session_config': allow_soft_placement: true
, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_experimental_distribute': None, '_eval_distribute': None, '_task_type': 'worker', '_num_worker_replicas': 1, '_experimental_max_worker_delay_secs': None, '_service': None, '_device_fn': None, '_distribute_coordinator_mode': None, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_is_chief': True, '_global_id_in_cluster': 0, '_save_checkpoints_secs': 600, '_save_summary_steps': 100, '_tf_random_seed': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f75dc685240>, '_protocol': None, '_task_id': 0}
W0802 17:55:47.560091 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

W0802 17:55:47.591004 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

W0802 17:55:48.684050 140144997107456 deprecation.py:506] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1633: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
I0802 17:55:48.687785 140144997107456 estimator.py:1145] Calling model_fn.
W0802 17:55:48.805828 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0802 17:55:48.841855 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0802 17:55:48.851698 140144997107456 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/losses/losses_impl.py:121: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0802 17:55:48.860609 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.metrics.accuracy is deprecated. Please use tf.compat.v1.metrics.accuracy instead.

W0802 17:55:48.882839 140144997107456 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0802 17:55:49.189281 140144997107456 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/training/optimizer.py:172: BaseResourceVariable.constraint (from tensorflow.python.ops.resource_variable_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Apply a constraint manually following the optimizer update step.
W0802 17:55:49.214515 140144997107456 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/model_fn.py:337: scalar (from tensorflow.python.framework.tensor_shape) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.TensorShape([]).
I0802 17:55:49.214864 140144997107456 estimator.py:1147] Done calling model_fn.
I0802 17:55:49.248135 140144997107456 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
I0802 17:55:49.403864 140144997107456 monitored_session.py:240] Graph was finalized.
2019-08-02 17:55:49.404301: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-02 17:55:49.410149: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-08-02 17:55:49.410683: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6a4d6f0 executing computations on platform Host. Devices:
2019-08-02 17:55:49.410771: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
I0802 17:55:49.413596 140144997107456 saver.py:1284] Restoring parameters from /tmp/mnist_model/model.ckpt-600
2019-08-02 17:55:49.426334: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
IteratorToStringHandle: CPU XLA_CPU 
IteratorV2: CPU XLA_CPU 
IteratorGetNext: CPU XLA_CPU 
MakeIterator: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  IteratorV2 (IteratorV2) /replica:0/task:0/device:GPU:0
  MakeIterator (MakeIterator) /replica:0/task:0/device:GPU:0
  IteratorToStringHandle (IteratorToStringHandle) /replica:0/task:0/device:GPU:0
  IteratorGetNext (IteratorGetNext) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.426480: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
AssignAddVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  global_step/Initializer/zeros (Const) 
  global_step (VarHandleOp) /replica:0/task:0/device:GPU:0
  global_step/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  global_step/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  global_step/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Identity/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/AssignAddVariableOp (AssignAddVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_4 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp (VarIsInitializedOp) 
  save/AssignVariableOp_26 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.426607: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d/kernel/Initializer/random_uniform/shape (Const) 
  conv2d/kernel/Initializer/random_uniform/min (Const) 
  conv2d/kernel/Initializer/random_uniform/max (Const) 
  conv2d/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  conv2d/kernel/Initializer/random_uniform/sub (Sub) 
  conv2d/kernel/Initializer/random_uniform/mul (Mul) 
  conv2d/kernel/Initializer/random_uniform (Add) 
  conv2d/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_1 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_11 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_12 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_1 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_11 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_12 (VarIsInitializedOp) 
  save/AssignVariableOp_5 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_6 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_7 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.426937: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
Mul: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
ResourceApplyAdam: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d/bias/Initializer/zeros (Const) 
  conv2d/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  beta1_power/Initializer/initial_value (Const) /replica:0/task:0/device:GPU:0
  beta1_power (VarHandleOp) /replica:0/task:0/device:GPU:0
  beta1_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  beta1_power/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  beta1_power/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  beta2_power/Initializer/initial_value (Const) /replica:0/task:0/device:GPU:0
  beta2_power (VarHandleOp) /replica:0/task:0/device:GPU:0
  beta2_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  beta2_power/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  beta2_power/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/mul (Mul) /replica:0/task:0/device:GPU:0
  Adam/AssignVariableOp (AssignVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_2 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/mul_1 (Mul) /replica:0/task:0/device:GPU:0
  Adam/AssignVariableOp_1 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_3 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_2 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_9 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_10 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_13 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_14 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_2 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_9 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_10 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_13 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_14 (VarIsInitializedOp) 
  save/AssignVariableOp (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_1 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_2 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_3 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_4 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.427372: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
Fill: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d_1/kernel/Initializer/random_uniform/shape (Const) 
  conv2d_1/kernel/Initializer/random_uniform/min (Const) 
  conv2d_1/kernel/Initializer/random_uniform/max (Const) 
  conv2d_1/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  conv2d_1/kernel/Initializer/random_uniform/sub (Sub) 
  conv2d_1/kernel/Initializer/random_uniform/mul (Mul) 
  conv2d_1/kernel/Initializer/random_uniform (Add) 
  conv2d_1/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d_1/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_3 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_15 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_16 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_3 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_15 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_16 (VarIsInitializedOp) 
  save/AssignVariableOp_11 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_12 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_13 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.427600: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d_1/bias/Initializer/zeros (Const) 
  conv2d_1/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_4 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_17 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_18 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_4 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_17 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_18 (VarIsInitializedOp) 
  save/AssignVariableOp_8 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_9 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_10 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.427879: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
Fill: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense/kernel/Initializer/random_uniform/shape (Const) 
  dense/kernel/Initializer/random_uniform/min (Const) 
  dense/kernel/Initializer/random_uniform/max (Const) 
  dense/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  dense/kernel/Initializer/random_uniform/sub (Sub) 
  dense/kernel/Initializer/random_uniform/mul (Mul) 
  dense/kernel/Initializer/random_uniform (Add) 
  dense/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_5 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_19 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_20 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_5 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_19 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_20 (VarIsInitializedOp) 
  save/AssignVariableOp_17 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_18 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_19 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.428165: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Fill: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense/bias/Initializer/zeros/shape_as_tensor (Const) 
  dense/bias/Initializer/zeros/Const (Const) 
  dense/bias/Initializer/zeros (Fill) 
  dense/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_6 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_21 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_22 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_6 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_21 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_22 (VarIsInitializedOp) 
  save/AssignVariableOp_14 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_15 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_16 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.428509: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
Fill: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense_1/kernel/Initializer/random_uniform/shape (Const) 
  dense_1/kernel/Initializer/random_uniform/min (Const) 
  dense_1/kernel/Initializer/random_uniform/max (Const) 
  dense_1/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  dense_1/kernel/Initializer/random_uniform/sub (Sub) 
  dense_1/kernel/Initializer/random_uniform/mul (Mul) 
  dense_1/kernel/Initializer/random_uniform (Add) 
  dense_1/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense_1/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_7 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_23 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_24 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_7 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_23 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_24 (VarIsInitializedOp) 
  save/AssignVariableOp_23 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_24 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_25 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.428696: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense_1/bias/Initializer/zeros (Const) 
  dense_1/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_8 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_25 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_26 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_8 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_25 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_26 (VarIsInitializedOp) 
  save/AssignVariableOp_20 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_21 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_22 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-08-02 17:55:49.428949: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
AssignAddVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  accuracy/total/Initializer/zeros (Const) 
  accuracy/total (VarHandleOp) /replica:0/task:0/device:GPU:0
  accuracy/total/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  accuracy/total/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/total/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/AssignAddVariableOp (AssignAddVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/value/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/update_op/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_27 (VarIsInitializedOp) 

2019-08-02 17:55:49.429034: W tensorflow/core/common_runtime/colocation_graph.cc:960] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
AssignAddVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  accuracy/count/Initializer/zeros (Const) 
  accuracy/count (VarHandleOp) /replica:0/task:0/device:GPU:0
  accuracy/count/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  accuracy/count/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/count/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/AssignAddVariableOp_1 (AssignAddVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/Maximum/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/Maximum_1/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_28 (VarIsInitializedOp) 

W0802 17:55:49.503391 140144997107456 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/training/saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
I0802 17:55:49.544175 140144997107456 session_manager.py:500] Running local_init_op.
I0802 17:55:49.560537 140144997107456 session_manager.py:502] Done running local_init_op.
I0802 17:55:49.875860 140144997107456 basic_session_run_hooks.py:606] Saving checkpoints for 600 into /tmp/mnist_model/model.ckpt.
I0802 17:55:54.912791 140144997107456 basic_session_run_hooks.py:262] cross_entropy = 0.1765852, learning_rate = 1e-04, train_accuracy = 0.97
I0802 17:55:54.913434 140144997107456 basic_session_run_hooks.py:262] loss = 0.1765852, step = 600
I0802 17:56:11.858450 140144997107456 basic_session_run_hooks.py:692] global_step/sec: 5.90113
I0802 17:56:11.859460 140144997107456 basic_session_run_hooks.py:260] cross_entropy = 0.0818163, learning_rate = 1e-04, train_accuracy = 0.975 (16.947 sec)
I0802 17:56:11.859687 140144997107456 basic_session_run_hooks.py:260] loss = 0.0818163, step = 700 (16.946 sec)
I0802 17:56:28.067404 140144997107456 basic_session_run_hooks.py:692] global_step/sec: 6.16944
I0802 17:56:28.068272 140144997107456 basic_session_run_hooks.py:260] cross_entropy = 0.12053037, learning_rate = 1e-04, train_accuracy = 0.97 (16.209 sec)
I0802 17:56:28.068516 140144997107456 basic_session_run_hooks.py:260] loss = 0.12053037, step = 800 (16.209 sec)
I0802 17:56:44.242596 140144997107456 basic_session_run_hooks.py:692] global_step/sec: 6.1823
I0802 17:56:44.243608 140144997107456 basic_session_run_hooks.py:260] cross_entropy = 0.07436521, learning_rate = 1e-04, train_accuracy = 0.97 (16.175 sec)
I0802 17:56:44.243974 140144997107456 basic_session_run_hooks.py:260] loss = 0.07436521, step = 900 (16.175 sec)
I0802 17:57:00.547430 140144997107456 basic_session_run_hooks.py:692] global_step/sec: 6.13315
I0802 17:57:00.548309 140144997107456 basic_session_run_hooks.py:260] cross_entropy = 0.052296676, learning_rate = 1e-04, train_accuracy = 0.972 (16.305 sec)
I0802 17:57:00.548600 140144997107456 basic_session_run_hooks.py:260] loss = 0.052296676, step = 1000 (16.305 sec)
I0802 17:57:16.738272 140144997107456 basic_session_run_hooks.py:692] global_step/sec: 6.17633
I0802 17:57:16.739129 140144997107456 basic_session_run_hooks.py:260] cross_entropy = 0.055802494, learning_rate = 1e-04, train_accuracy = 0.9716667 (16.191 sec)
I0802 17:57:16.739343 140144997107456 basic_session_run_hooks.py:260] loss = 0.055802494, step = 1100 (16.191 sec)
I0802 17:57:32.770414 140144997107456 basic_session_run_hooks.py:606] Saving checkpoints for 1200 into /tmp/mnist_model/model.ckpt.
I0802 17:57:32.938885 140144997107456 estimator.py:368] Loss for final step: 0.11538864.
W0802 17:57:32.981930 140144997107456 deprecation.py:323] From models/official/mnist/mnist.py:204: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
I0802 17:57:32.997227 140144997107456 estimator.py:1145] Calling model_fn.
I0802 17:57:33.162929 140144997107456 estimator.py:1147] Done calling model_fn.
I0802 17:57:33.181160 140144997107456 evaluation.py:255] Starting evaluation at 2019-08-02T17:57:33Z
I0802 17:57:33.257117 140144997107456 monitored_session.py:240] Graph was finalized.
I0802 17:57:33.258520 140144997107456 saver.py:1284] Restoring parameters from /tmp/mnist_model/model.ckpt-1200
I0802 17:57:33.312685 140144997107456 session_manager.py:500] Running local_init_op.
I0802 17:57:33.327383 140144997107456 session_manager.py:502] Done running local_init_op.
I0802 17:57:38.268164 140144997107456 evaluation.py:275] Finished evaluation at 2019-08-02-17:57:38
I0802 17:57:38.268510 140144997107456 estimator.py:2039] Saving dict for global step 1200: accuracy = 0.9807, global_step = 1200, loss = 0.06370965
I0802 17:57:38.318452 140144997107456 estimator.py:2099] Saving 'checkpoint_path' summary for global step 1200: /tmp/mnist_model/model.ckpt-1200

Evaluation results:
    {'global_step': 1200, 'accuracy': 0.9807, 'loss': 0.06370965}

W0802 17:57:38.320664 140144997107456 deprecation.py:323] From models/official/mnist/mnist.py:228: Estimator.export_savedmodel (from tensorflow_estimator.python.estimator.estimator) is deprecated and will be removed in a future version.
Instructions for updating:
This function has been renamed, use `export_saved_model` instead.
I0802 17:57:38.327512 140144997107456 estimator.py:1145] Calling model_fn.
I0802 17:57:38.465367 140144997107456 estimator.py:1147] Done calling model_fn.
W0802 17:57:38.465671 140144997107456 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
I0802 17:57:38.466043 140144997107456 export_utils.py:170] Signatures INCLUDED in export for Predict: ['classify', 'serving_default']
I0802 17:57:38.466136 140144997107456 export_utils.py:170] Signatures INCLUDED in export for Train: None
I0802 17:57:38.466204 140144997107456 export_utils.py:170] Signatures INCLUDED in export for Eval: None
I0802 17:57:38.466246 140144997107456 export_utils.py:170] Signatures INCLUDED in export for Regress: None
I0802 17:57:38.466294 140144997107456 export_utils.py:170] Signatures INCLUDED in export for Classify: None
I0802 17:57:38.489985 140144997107456 saver.py:1284] Restoring parameters from /tmp/mnist_model/model.ckpt-1200
I0802 17:57:38.515160 140144997107456 builder_impl.py:662] Assets added to graph.
I0802 17:57:38.515433 140144997107456 builder_impl.py:457] No assets to write.
I0802 17:57:38.572014 140144997107456 builder_impl.py:422] SavedModel written to: /tmp/mnist_saved_model/temp-b'1564793858'/saved_model.pb

For the example, you train the model for just a single epoch, so it only trains to ~96% accuracy.

Convert to a TensorFlow Lite model

The savedmodel directory is named with a timestamp. Select the most recent one:

saved_model_dir = str(sorted(pathlib.Path(saved_models_root).glob("*"))[-1])
saved_model_dir
'/tmp/mnist_saved_model/1564793858'

Using the Python TFLiteConverter, the saved model can be converted into a TensorFlow Lite model.

First load the model using the TFLiteConverter:

import tensorflow as tf
tf.enable_eager_execution()
tf.logging.set_verbosity(tf.logging.DEBUG)

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
W0802 17:57:39.503450 139925293250304 module_wrapper.py:136] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/util/module_wrapper.py:163: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0802 17:57:39.513344 139925293250304 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/lite/python/convert_saved_model.py:60: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
I0802 17:57:39.562710 139925293250304 saver.py:1284] Restoring parameters from /tmp/mnist_saved_model/1564793858/variables/variables
I0802 17:57:39.602909 139925293250304 convert_saved_model.py:80] The given SavedModel MetaGraphDef contains SignatureDefs with the following keys: {'classify', 'serving_default'}
I0802 17:57:39.604528 139925293250304 convert_saved_model.py:99] input tensors info: 
I0802 17:57:39.605328 139925293250304 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: image
I0802 17:57:39.605972 139925293250304 convert_saved_model.py:43]  tensor name: Placeholder:0, shape: (-1, 28, 28), type: DT_FLOAT
I0802 17:57:39.606681 139925293250304 convert_saved_model.py:101] output tensors info: 
I0802 17:57:39.607440 139925293250304 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: probabilities
I0802 17:57:39.608120 139925293250304 convert_saved_model.py:43]  tensor name: Softmax:0, shape: (-1, 10), type: DT_FLOAT
I0802 17:57:39.608816 139925293250304 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: classes
I0802 17:57:39.609461 139925293250304 convert_saved_model.py:43]  tensor name: ArgMax:0, shape: (-1), type: DT_INT64
I0802 17:57:39.653901 139925293250304 saver.py:1284] Restoring parameters from /tmp/mnist_saved_model/1564793858/variables/variables
W0802 17:57:39.707692 139925293250304 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/lite/python/util.py:249: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W0802 17:57:39.708967 139925293250304 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/framework/graph_util_impl.py:275: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
I0802 17:57:39.732800 139925293250304 graph_util_impl.py:318] Froze 8 variables.
I0802 17:57:39.764792 139925293250304 graph_util_impl.py:372] Converted 8 variables to const ops.

Write it out to a .tflite file:

tflite_models_dir = pathlib.Path("/tmp/mnist_tflite_models/")
tflite_models_dir.mkdir(exist_ok=True, parents=True)
tflite_model_file = tflite_models_dir/"mnist_model.tflite"
tflite_model_file.write_bytes(tflite_model)
13101276

To instead quantize the model on export, first set the optimizations flag to optimize for size:

tf.logging.set_verbosity(tf.logging.INFO)
converter.optimizations = [tf.lite.Optimize.DEFAULT]

Now, construct and provide a representative dataset, this is used to get the dynamic range of activations.

mnist_train, _ = tf.keras.datasets.mnist.load_data()
images = tf.cast(mnist_train[0], tf.float32)/255.0
mnist_ds = tf.data.Dataset.from_tensor_slices((images)).batch(1)
def representative_data_gen():
  for input_value in mnist_ds.take(100):
    yield [input_value]

converter.representative_dataset = representative_data_gen

Finally, convert the model like usual. By default, the converted model will still use float input and outputs for invocation convenience.

tflite_quant_model = converter.convert()
tflite_model_quant_file = tflite_models_dir/"mnist_model_quant.tflite"
tflite_model_quant_file.write_bytes(tflite_quant_model)
3284000

Note how the resulting file is approximately 1/4 the size.

!ls -lh {tflite_models_dir}
total 22M
-rw-rw-r-- 1 kbuilder kbuilder 6.3M Aug  2 17:54 mnist_model_quant_f16.tflite
-rw-rw-r-- 1 kbuilder kbuilder 3.2M Aug  2 17:57 mnist_model_quant.tflite
-rw-rw-r-- 1 kbuilder kbuilder  13M Aug  2 17:57 mnist_model.tflite

Run the TensorFlow Lite models

Run the TensorFlow Lite model using the Python TensorFlow Lite Interpreter.

Load the test data

First, let's load the MNIST test data to feed to the model:

import numpy as np
_, mnist_test = tf.keras.datasets.mnist.load_data()
images, labels = tf.cast(mnist_test[0], tf.float32)/255.0, mnist_test[1]

mnist_ds = tf.data.Dataset.from_tensor_slices((images, labels)).batch(1)

Load the model into the interpreters

interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))
interpreter.allocate_tensors()
interpreter_quant = tf.lite.Interpreter(model_path=str(tflite_model_quant_file))
interpreter_quant.allocate_tensors()

Test the models on one image

for img, label in mnist_ds:
  break

interpreter.set_tensor(interpreter.get_input_details()[0]["index"], img)
interpreter.invoke()
predictions = interpreter.get_tensor(
    interpreter.get_output_details()[0]["index"])
import matplotlib.pylab as plt

plt.imshow(img[0])
template = "True:{true}, predicted:{predict}"
_ = plt.title(template.format(true= str(label[0].numpy()),
                              predict=str(predictions[0])))
plt.grid(False)
interpreter_quant.set_tensor(
    interpreter_quant.get_input_details()[0]["index"], img)
interpreter_quant.invoke()
predictions = interpreter_quant.get_tensor(
    interpreter_quant.get_output_details()[0]["index"])
plt.imshow(img[0])
template = "True:{true}, predicted:{predict}"
_ = plt.title(template.format(true= str(label[0].numpy()),
                              predict=str(predictions[0])))
plt.grid(False)

png

Evaluate the models

def eval_model(interpreter, mnist_ds):
  total_seen = 0
  num_correct = 0

  input_index = interpreter.get_input_details()[0]["index"]
  output_index = interpreter.get_output_details()[0]["index"]

  for img, label in mnist_ds:
    total_seen += 1
    interpreter.set_tensor(input_index, img)
    interpreter.invoke()
    predictions = interpreter.get_tensor(output_index)
    if predictions == label.numpy():
      num_correct += 1

    if total_seen % 500 == 0:
      print("Accuracy after %i images: %f" %
            (total_seen, float(num_correct) / float(total_seen)))

  return float(num_correct) / float(total_seen)
# Create smaller dataset for demonstration purposes
mnist_ds_demo = mnist_ds.take(2000)

print(eval_model(interpreter, mnist_ds_demo))
Accuracy after 500 images: 0.986000
Accuracy after 1000 images: 0.981000
Accuracy after 1500 images: 0.976000
Accuracy after 2000 images: 0.974500
0.9745

Repeat the evaluation on the fully quantized model to obtain:

# NOTE: Colab runs on server CPUs. At the time of writing this, TensorFlow Lite
# doesn't have super optimized server CPU kernels. For this reason this may be
# slower than the above float interpreter. But for mobile CPUs, considerable
# speedup can be observed.
# Only use 2000 for demonstration purposes
print(eval_model(interpreter_quant, mnist_ds_demo))
Accuracy after 500 images: 0.986000
Accuracy after 1000 images: 0.981000
Accuracy after 1500 images: 0.975333
Accuracy after 2000 images: 0.974000
0.974

In this example, you have fully quantized a model with no difference in the accuracy.