Post-training weight quantization

View on TensorFlow.org Run in Google Colab View source on GitHub

Overview

TensorFlow Lite now supports converting weights to 8 bit precision as part of model conversion from tensorflow graphdefs to TensorFlow Lite's flat buffer format. Weight quantization achieves a 4x reduction in the model size. In addition, TFLite supports on the fly quantization and dequantization of activations to allow for:

  1. Using quantized kernels for faster implementation when available.
  2. Mixing of floating-point kernels with quantized kernels for different parts of the graph.

The activations are always stored in floating point. For ops that support quantized kernels, the activations are quantized to 8 bits of precision dynamically prior to processing and are de-quantized to float precision after processing. Depending on the model being converted, this can give a speedup over pure floating point computation.

In contrast to quantization aware training , the weights are quantized post training and the activations are quantized dynamically at inference in this method. Therefore, the model weights are not retrained to compensate for quantization induced errors. It is important to check the accuracy of the quantized model to ensure that the degradation is acceptable.

This tutorial trains an MNIST model from scratch, checks its accuracy in TensorFlow, and then converts the saved model into a Tensorflow Lite flatbuffer with weight quantization. Finally, it checks the accuracy of the converted model and compare it to the original saved model. The training script, mnist.py, is from Tensorflow official mnist tutorial.

Build an MNIST model

Setup

! pip uninstall -y tensorflow
! pip install -q -U tf-nightly
WARNING: Skipping tensorflow as it is not installed.
import tensorflow as tf
tf.enable_eager_execution()
WARNING:tensorflow:

  TensorFlow's `tf-nightly` package will soon be updated to TensorFlow 2.0.

  Please upgrade your code to TensorFlow 2.0:
    * https://www.tensorflow.org/beta/guide/migration_guide

  Or install the latest stable TensorFlow 1.X release:
    * `pip install -U "tensorflow==1.*"`

  Otherwise your code may be broken by the change.

  
! git clone --depth 1 https://github.com/tensorflow/models
Cloning into 'models'...
remote: Enumerating objects: 3230, done.
remote: Counting objects: 100% (3230/3230), done.
remote: Compressing objects: 100% (2737/2737), done.
remote: Total 3230 (delta 598), reused 2028 (delta 417), pack-reused 0
Receiving objects: 100% (3230/3230), 370.70 MiB | 41.42 MiB/s, done.
Resolving deltas: 100% (598/598), done.
Checking out files: 100% (3055/3055), done.
import sys
import os

if sys.version_info.major >= 3:
    import pathlib
else:
    import pathlib2 as pathlib

# Add `models` to the python path.
models_path = os.path.join(os.getcwd(), "models")
sys.path.append(models_path)

Train and export the model

saved_models_root = "/tmp/mnist_saved_model"
# The above path addition is not visible to subprocesses, add the path for the subprocess as well.
# Note: channels_last is required here or the conversion may fail. 
!PYTHONPATH={models_path} python models/official/mnist/mnist.py --train_epochs=1 --export_dir {saved_models_root} --data_format=channels_last
WARNING:tensorflow:

  TensorFlow's `tf-nightly` package will soon be updated to TensorFlow 2.0.

  Please upgrade your code to TensorFlow 2.0:
    * https://www.tensorflow.org/beta/guide/migration_guide

  Or install the latest stable TensorFlow 1.X release:
    * `pip install -U "tensorflow==1.*"`

  Otherwise your code may be broken by the change.

  
WARNING:tensorflow:From models/official/mnist/mnist.py:242: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From models/official/mnist/mnist.py:168: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1001 10:49:43.824324 140606332913408 module_wrapper.py:137] From models/official/mnist/mnist.py:168: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

INFO:tensorflow:Initializing RunConfig with distribution strategies.
I1001 10:49:43.825517 140606332913408 run_config.py:566] Initializing RunConfig with distribution strategies.
INFO:tensorflow:Not using Distribute Coordinator.
I1001 10:49:43.825726 140606332913408 estimator_training.py:167] Not using Distribute Coordinator.
INFO:tensorflow:Using config: {'_master': '', '_protocol': None, '_tf_random_seed': None, '_service': None, '_task_id': 0, '_save_checkpoints_secs': 600, '_task_type': 'worker', '_eval_distribute': None, '_save_checkpoints_steps': None, '_is_chief': True, '_global_id_in_cluster': 0, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_experimental_distribute': None, '_keep_checkpoint_max': 5, '_experimental_max_worker_delay_secs': None, '_distribute_coordinator_mode': None, '_num_worker_replicas': 1, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategyV1 object at 0x7fe1447aa0f0>, '_session_creation_timeout_secs': 7200, '_log_step_count_steps': 100, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe1447aa2b0>, '_model_dir': '/tmp/mnist_model', '_device_fn': None, '_num_ps_replicas': 0, '_session_config': allow_soft_placement: true
, '_save_summary_steps': 100}
I1001 10:49:43.826283 140606332913408 estimator.py:212] Using config: {'_master': '', '_protocol': None, '_tf_random_seed': None, '_service': None, '_task_id': 0, '_save_checkpoints_secs': 600, '_task_type': 'worker', '_eval_distribute': None, '_save_checkpoints_steps': None, '_is_chief': True, '_global_id_in_cluster': 0, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_experimental_distribute': None, '_keep_checkpoint_max': 5, '_experimental_max_worker_delay_secs': None, '_distribute_coordinator_mode': None, '_num_worker_replicas': 1, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategyV1 object at 0x7fe1447aa0f0>, '_session_creation_timeout_secs': 7200, '_log_step_count_steps': 100, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe1447aa2b0>, '_model_dir': '/tmp/mnist_model', '_device_fn': None, '_num_ps_replicas': 0, '_session_config': allow_soft_placement: true
, '_save_summary_steps': 100}
WARNING:tensorflow:From /tmpfs/src/temp/tensorflow/lite/g3doc/performance/models/official/mnist/dataset.py:65: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

W1001 10:49:43.828074 140606332913408 module_wrapper.py:137] From /tmpfs/src/temp/tensorflow/lite/g3doc/performance/models/official/mnist/dataset.py:65: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

Downloading https://storage.googleapis.com/cvdf-datasets/mnist/train-images-idx3-ubyte.gz to /tmpfs/tmp/tmpd4wx804m.gz
Downloading https://storage.googleapis.com/cvdf-datasets/mnist/train-labels-idx1-ubyte.gz to /tmpfs/tmp/tmpdcvy_gw_.gz
WARNING:tensorflow:Entity <function dataset.<locals>.decode_image at 0x7fe14479ed08> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: No module named 'tensorflow_estimator.contrib'
W1001 10:49:45.519554 140606332913408 ag_logging.py:146] Entity <function dataset.<locals>.decode_image at 0x7fe14479ed08> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: No module named 'tensorflow_estimator.contrib'
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W1001 10:49:45.679697 140606332913408 deprecation.py:506] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:tensorflow:Calling model_fn.
I1001 10:49:45.684040 140606332913408 estimator.py:1147] Calling model_fn.
WARNING:tensorflow:From models/official/mnist/mnist.py:127: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W1001 10:49:45.815752 140606332913408 module_wrapper.py:137] From models/official/mnist/mnist.py:127: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From models/official/mnist/mnist.py:130: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W1001 10:49:45.856831 140606332913408 module_wrapper.py:137] From models/official/mnist/mnist.py:130: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/losses/losses_impl.py:121: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1001 10:49:45.867890 140606332913408 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/ops/losses/losses_impl.py:121: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From models/official/mnist/mnist.py:131: The name tf.metrics.accuracy is deprecated. Please use tf.compat.v1.metrics.accuracy instead.

W1001 10:49:45.877774 140606332913408 module_wrapper.py:137] From models/official/mnist/mnist.py:131: The name tf.metrics.accuracy is deprecated. Please use tf.compat.v1.metrics.accuracy instead.

WARNING:tensorflow:From models/official/mnist/mnist.py:140: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1001 10:49:45.904295 140606332913408 module_wrapper.py:137] From models/official/mnist/mnist.py:140: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

INFO:tensorflow:Done calling model_fn.
I1001 10:49:46.144005 140606332913408 estimator.py:1149] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I1001 10:49:46.181501 140606332913408 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I1001 10:49:46.352748 140606332913408 monitored_session.py:240] Graph was finalized.
2019-10-01 10:49:46.353229: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-01 10:49:46.359123: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-10-01 10:49:46.359489: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x771d650 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-10-01 10:49:46.359533: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2019-10-01 10:49:46.372781: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
IteratorToStringHandle: CPU XLA_CPU 
IteratorV2: CPU XLA_CPU 
IteratorGetNext: CPU XLA_CPU 
MakeIterator: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  IteratorV2 (IteratorV2) /replica:0/task:0/device:GPU:0
  MakeIterator (MakeIterator) /replica:0/task:0/device:GPU:0
  IteratorToStringHandle (IteratorToStringHandle) /replica:0/task:0/device:GPU:0
  IteratorGetNext (IteratorGetNext) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.372904: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
AssignAddVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  global_step/Initializer/zeros (Const) 
  global_step (VarHandleOp) /replica:0/task:0/device:GPU:0
  global_step/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  global_step/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  global_step/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Identity/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/AssignAddVariableOp (AssignAddVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_4 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp (VarIsInitializedOp) 
  save/AssignVariableOp_26 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.373070: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d/kernel/Initializer/random_uniform/shape (Const) 
  conv2d/kernel/Initializer/random_uniform/min (Const) 
  conv2d/kernel/Initializer/random_uniform/max (Const) 
  conv2d/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  conv2d/kernel/Initializer/random_uniform/sub (Sub) 
  conv2d/kernel/Initializer/random_uniform/mul (Mul) 
  conv2d/kernel/Initializer/random_uniform (Add) 
  conv2d/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_1 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_11 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_12 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_1 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_11 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_12 (VarIsInitializedOp) 
  save/AssignVariableOp_5 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_6 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_7 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.373316: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
Mul: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
ResourceApplyAdam: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d/bias/Initializer/zeros (Const) 
  conv2d/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  beta1_power/Initializer/initial_value (Const) /replica:0/task:0/device:GPU:0
  beta1_power (VarHandleOp) /replica:0/task:0/device:GPU:0
  beta1_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  beta1_power/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  beta1_power/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  beta2_power/Initializer/initial_value (Const) /replica:0/task:0/device:GPU:0
  beta2_power (VarHandleOp) /replica:0/task:0/device:GPU:0
  beta2_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  beta2_power/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  beta2_power/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/kernel/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/kernel/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/bias/ResourceApplyAdam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/bias/ResourceApplyAdam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/mul (Mul) /replica:0/task:0/device:GPU:0
  Adam/AssignVariableOp (AssignVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_2 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/mul_1 (Mul) /replica:0/task:0/device:GPU:0
  Adam/AssignVariableOp_1 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  Adam/ReadVariableOp_3 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_2 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_9 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_10 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_13 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_14 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_2 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_9 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_10 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_13 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_14 (VarIsInitializedOp) 
  save/AssignVariableOp (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_1 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_2 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_3 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_4 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.373533: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
Fill: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d_1/kernel/Initializer/random_uniform/shape (Const) 
  conv2d_1/kernel/Initializer/random_uniform/min (Const) 
  conv2d_1/kernel/Initializer/random_uniform/max (Const) 
  conv2d_1/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  conv2d_1/kernel/Initializer/random_uniform/sub (Sub) 
  conv2d_1/kernel/Initializer/random_uniform/mul (Mul) 
  conv2d_1/kernel/Initializer/random_uniform (Add) 
  conv2d_1/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d_1/Conv2D/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_3 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_15 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_16 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_3 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_15 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_16 (VarIsInitializedOp) 
  save/AssignVariableOp_11 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_12 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_13 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.373718: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  conv2d_1/bias/Initializer/zeros (Const) 
  conv2d_1/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/conv2d_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  conv2d_1/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_conv2d_1/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_4 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_17 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_18 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_4 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_17 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_18 (VarIsInitializedOp) 
  save/AssignVariableOp_8 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_9 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_10 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.373926: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
Fill: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense/kernel/Initializer/random_uniform/shape (Const) 
  dense/kernel/Initializer/random_uniform/min (Const) 
  dense/kernel/Initializer/random_uniform/max (Const) 
  dense/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  dense/kernel/Initializer/random_uniform/sub (Sub) 
  dense/kernel/Initializer/random_uniform/mul (Mul) 
  dense/kernel/Initializer/random_uniform (Add) 
  dense/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_5 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_19 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_20 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_5 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_19 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_20 (VarIsInitializedOp) 
  save/AssignVariableOp_17 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_18 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_19 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.374123: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Fill: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense/bias/Initializer/zeros/shape_as_tensor (Const) 
  dense/bias/Initializer/zeros/Const (Const) 
  dense/bias/Initializer/zeros (Fill) 
  dense/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_6 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_21 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_22 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_6 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_21 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_22 (VarIsInitializedOp) 
  save/AssignVariableOp_14 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_15 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_16 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.376233: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
RandomUniform: CPU XLA_CPU 
Fill: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
Mul: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
Sub: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 
Add: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense_1/kernel/Initializer/random_uniform/shape (Const) 
  dense_1/kernel/Initializer/random_uniform/min (Const) 
  dense_1/kernel/Initializer/random_uniform/max (Const) 
  dense_1/kernel/Initializer/random_uniform/RandomUniform (RandomUniform) 
  dense_1/kernel/Initializer/random_uniform/sub (Sub) 
  dense_1/kernel/Initializer/random_uniform/mul (Mul) 
  dense_1/kernel/Initializer/random_uniform (Add) 
  dense_1/kernel (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense_1/MatMul/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Initializer/zeros/shape_as_tensor (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Initializer/zeros/Const (Const) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Initializer/zeros (Fill) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/kernel/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/kernel/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_7 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_23 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_24 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_7 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_23 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_24 (VarIsInitializedOp) 
  save/AssignVariableOp_23 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_24 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_25 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.376417: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
ResourceApplyAdam: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  dense_1/bias/Initializer/zeros (Const) 
  dense_1/bias (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  sequential/dense_1/BiasAdd/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/Initializer/zeros (Const) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1 (VarHandleOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  dense_1/bias/Adam_1/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  Adam/update_dense_1/bias/ResourceApplyAdam (ResourceApplyAdam) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_8 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_25 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables/VarIsInitializedOp_26 (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_8 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_25 (VarIsInitializedOp) 
  report_uninitialized_variables_1/VarIsInitializedOp_26 (VarIsInitializedOp) 
  save/AssignVariableOp_20 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_21 (AssignVariableOp) /replica:0/task:0/device:GPU:0
  save/AssignVariableOp_22 (AssignVariableOp) /replica:0/task:0/device:GPU:0

2019-10-01 10:49:46.376655: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
AssignAddVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  accuracy/total/Initializer/zeros (Const) 
  accuracy/total (VarHandleOp) /replica:0/task:0/device:GPU:0
  accuracy/total/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  accuracy/total/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/total/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/AssignAddVariableOp (AssignAddVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/value/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/update_op/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_27 (VarIsInitializedOp) 

2019-10-01 10:49:46.376749: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
  /job:localhost/replica:0/task:0/device:CPU:0
  /job:localhost/replica:0/task:0/device:XLA_CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/replica:0/task:0/device:GPU:0' assigned_device_name_='' resource_device_name_='/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU, XLA_CPU] possible_devices_=[]
AssignAddVariableOp: CPU XLA_CPU 
ReadVariableOp: CPU XLA_CPU 
AssignVariableOp: CPU XLA_CPU 
VarIsInitializedOp: CPU XLA_CPU 
Const: CPU XLA_CPU 
VarHandleOp: CPU XLA_CPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  accuracy/count/Initializer/zeros (Const) 
  accuracy/count (VarHandleOp) /replica:0/task:0/device:GPU:0
  accuracy/count/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /replica:0/task:0/device:GPU:0
  accuracy/count/Assign (AssignVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/count/Read/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/AssignAddVariableOp_1 (AssignAddVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/ReadVariableOp_1 (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/Maximum/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  accuracy/Maximum_1/ReadVariableOp (ReadVariableOp) /replica:0/task:0/device:GPU:0
  report_uninitialized_variables_1/VarIsInitializedOp_28 (VarIsInitializedOp) 

INFO:tensorflow:Running local_init_op.
I1001 10:49:46.569829 140606332913408 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I1001 10:49:46.586072 140606332913408 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/mnist_model/model.ckpt.
I1001 10:49:46.914192 140606332913408 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /tmp/mnist_model/model.ckpt.
INFO:tensorflow:cross_entropy = 2.3084075, learning_rate = 1e-04, train_accuracy = 0.13
I1001 10:49:53.034571 140606332913408 basic_session_run_hooks.py:262] cross_entropy = 2.3084075, learning_rate = 1e-04, train_accuracy = 0.13
INFO:tensorflow:loss = 2.3084075, step = 0
I1001 10:49:53.035211 140606332913408 basic_session_run_hooks.py:262] loss = 2.3084075, step = 0
INFO:tensorflow:global_step/sec: 6.11835
I1001 10:50:09.378496 140606332913408 basic_session_run_hooks.py:692] global_step/sec: 6.11835
INFO:tensorflow:cross_entropy = 0.5173844, learning_rate = 1e-04, train_accuracy = 0.485 (16.345 sec)
I1001 10:50:09.379353 140606332913408 basic_session_run_hooks.py:260] cross_entropy = 0.5173844, learning_rate = 1e-04, train_accuracy = 0.485 (16.345 sec)
INFO:tensorflow:loss = 0.5173844, step = 100 (16.345 sec)
I1001 10:50:09.379731 140606332913408 basic_session_run_hooks.py:260] loss = 0.5173844, step = 100 (16.345 sec)
INFO:tensorflow:global_step/sec: 6.48642
I1001 10:50:24.795319 140606332913408 basic_session_run_hooks.py:692] global_step/sec: 6.48642
INFO:tensorflow:cross_entropy = 0.39411446, learning_rate = 1e-04, train_accuracy = 0.61 (15.417 sec)
I1001 10:50:24.796307 140606332913408 basic_session_run_hooks.py:260] cross_entropy = 0.39411446, learning_rate = 1e-04, train_accuracy = 0.61 (15.417 sec)
INFO:tensorflow:loss = 0.39411446, step = 200 (15.417 sec)
I1001 10:50:24.796691 140606332913408 basic_session_run_hooks.py:260] loss = 0.39411446, step = 200 (15.417 sec)
INFO:tensorflow:global_step/sec: 6.49431
I1001 10:50:40.193404 140606332913408 basic_session_run_hooks.py:692] global_step/sec: 6.49431
INFO:tensorflow:cross_entropy = 0.121275075, learning_rate = 1e-04, train_accuracy = 0.6975 (15.398 sec)
I1001 10:50:40.194319 140606332913408 basic_session_run_hooks.py:260] cross_entropy = 0.121275075, learning_rate = 1e-04, train_accuracy = 0.6975 (15.398 sec)
INFO:tensorflow:loss = 0.121275075, step = 300 (15.398 sec)
I1001 10:50:40.194679 140606332913408 basic_session_run_hooks.py:260] loss = 0.121275075, step = 300 (15.398 sec)
INFO:tensorflow:global_step/sec: 6.49934
I1001 10:50:55.579627 140606332913408 basic_session_run_hooks.py:692] global_step/sec: 6.49934
INFO:tensorflow:cross_entropy = 0.13345657, learning_rate = 1e-04, train_accuracy = 0.75 (15.386 sec)
I1001 10:50:55.580628 140606332913408 basic_session_run_hooks.py:260] cross_entropy = 0.13345657, learning_rate = 1e-04, train_accuracy = 0.75 (15.386 sec)
INFO:tensorflow:loss = 0.13345657, step = 400 (15.386 sec)
I1001 10:50:55.580921 140606332913408 basic_session_run_hooks.py:260] loss = 0.13345657, step = 400 (15.386 sec)
INFO:tensorflow:global_step/sec: 6.53193
I1001 10:51:10.889015 140606332913408 basic_session_run_hooks.py:692] global_step/sec: 6.53193
INFO:tensorflow:cross_entropy = 0.12333397, learning_rate = 1e-04, train_accuracy = 0.78833336 (15.309 sec)
I1001 10:51:10.889981 140606332913408 basic_session_run_hooks.py:260] cross_entropy = 0.12333397, learning_rate = 1e-04, train_accuracy = 0.78833336 (15.309 sec)
INFO:tensorflow:loss = 0.12333397, step = 500 (15.309 sec)
I1001 10:51:10.890268 140606332913408 basic_session_run_hooks.py:260] loss = 0.12333397, step = 500 (15.309 sec)
INFO:tensorflow:Saving checkpoints for 600 into /tmp/mnist_model/model.ckpt.
I1001 10:51:26.107749 140606332913408 basic_session_run_hooks.py:606] Saving checkpoints for 600 into /tmp/mnist_model/model.ckpt.
INFO:tensorflow:Loss for final step: 0.1076683.
I1001 10:51:26.251542 140606332913408 estimator.py:371] Loss for final step: 0.1076683.
Downloading https://storage.googleapis.com/cvdf-datasets/mnist/t10k-images-idx3-ubyte.gz to /tmpfs/tmp/tmptgfjppjh.gz
Downloading https://storage.googleapis.com/cvdf-datasets/mnist/t10k-labels-idx1-ubyte.gz to /tmpfs/tmp/tmpcdtp0iaf.gz
WARNING:tensorflow:From models/official/mnist/mnist.py:210: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
W1001 10:51:26.563744 140606332913408 deprecation.py:323] From models/official/mnist/mnist.py:210: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
INFO:tensorflow:Calling model_fn.
I1001 10:51:26.579970 140606332913408 estimator.py:1147] Calling model_fn.
INFO:tensorflow:Done calling model_fn.
I1001 10:51:26.764876 140606332913408 estimator.py:1149] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-10-01T10:51:26Z
I1001 10:51:26.785440 140606332913408 evaluation.py:255] Starting evaluation at 2019-10-01T10:51:26Z
INFO:tensorflow:Graph was finalized.
I1001 10:51:26.867923 140606332913408 monitored_session.py:240] Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/mnist_model/model.ckpt-600
I1001 10:51:26.869284 140606332913408 saver.py:1284] Restoring parameters from /tmp/mnist_model/model.ckpt-600
INFO:tensorflow:Running local_init_op.
I1001 10:51:26.920707 140606332913408 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I1001 10:51:26.933905 140606332913408 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-10-01-10:51:32
I1001 10:51:32.334335 140606332913408 evaluation.py:275] Finished evaluation at 2019-10-01-10:51:32
INFO:tensorflow:Saving dict for global step 600: accuracy = 0.9692, global_step = 600, loss = 0.10507722
I1001 10:51:32.334626 140606332913408 estimator.py:2049] Saving dict for global step 600: accuracy = 0.9692, global_step = 600, loss = 0.10507722
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 600: /tmp/mnist_model/model.ckpt-600
I1001 10:51:32.384645 140606332913408 estimator.py:2109] Saving 'checkpoint_path' summary for global step 600: /tmp/mnist_model/model.ckpt-600

Evaluation results:
    {'loss': 0.10507722, 'global_step': 600, 'accuracy': 0.9692}

WARNING:tensorflow:From models/official/mnist/mnist.py:234: Estimator.export_savedmodel (from tensorflow_estimator.python.estimator.estimator) is deprecated and will be removed in a future version.
Instructions for updating:
This function has been renamed, use `export_saved_model` instead.
W1001 10:51:32.386067 140606332913408 deprecation.py:323] From models/official/mnist/mnist.py:234: Estimator.export_savedmodel (from tensorflow_estimator.python.estimator.estimator) is deprecated and will be removed in a future version.
Instructions for updating:
This function has been renamed, use `export_saved_model` instead.
INFO:tensorflow:Calling model_fn.
I1001 10:51:32.393103 140606332913408 estimator.py:1147] Calling model_fn.
INFO:tensorflow:Done calling model_fn.
I1001 10:51:32.536799 140606332913408 estimator.py:1149] Done calling model_fn.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
W1001 10:51:32.537208 140606332913408 deprecation.py:323] From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Signatures INCLUDED in export for Train: None
I1001 10:51:32.537760 140606332913408 export_utils.py:170] Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['classify', 'serving_default']
I1001 10:51:32.537940 140606332913408 export_utils.py:170] Signatures INCLUDED in export for Predict: ['classify', 'serving_default']
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
I1001 10:51:32.538098 140606332913408 export_utils.py:170] Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
I1001 10:51:32.538248 140606332913408 export_utils.py:170] Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
I1001 10:51:32.538388 140606332913408 export_utils.py:170] Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Restoring parameters from /tmp/mnist_model/model.ckpt-600
I1001 10:51:32.565613 140606332913408 saver.py:1284] Restoring parameters from /tmp/mnist_model/model.ckpt-600
INFO:tensorflow:Assets added to graph.
I1001 10:51:32.594856 140606332913408 builder_impl.py:665] Assets added to graph.
INFO:tensorflow:No assets to write.
I1001 10:51:32.595102 140606332913408 builder_impl.py:460] No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/mnist_saved_model/temp-b'1569952292'/saved_model.pb
I1001 10:51:32.654433 140606332913408 builder_impl.py:425] SavedModel written to: /tmp/mnist_saved_model/temp-b'1569952292'/saved_model.pb

For the example, since you trained the model for just a single epoch, so it only trains to ~96% accuracy.

Convert to a TFLite model

The savedmodel directory is named with a timestamp. Select the most recent one:

saved_model_dir = str(sorted(pathlib.Path(saved_models_root).glob("*"))[-1])
saved_model_dir
'/tmp/mnist_saved_model/1569952292'

Using the python TFLiteConverter, the saved model can be converted into a TFLite model.

First load the model using the TFLiteConverter:

import tensorflow as tf
tf.enable_eager_execution()
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/lite/python/convert_saved_model.py:60: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from /tmp/mnist_saved_model/1569952292/variables/variables
INFO:tensorflow:The given SavedModel MetaGraphDef contains SignatureDefs with the following keys: {'classify', 'serving_default'}
INFO:tensorflow:input tensors info: 
INFO:tensorflow:Tensor's key in saved_model's tensor_map: image
INFO:tensorflow: tensor name: Placeholder:0, shape: (-1, 28, 28), type: DT_FLOAT
INFO:tensorflow:output tensors info: 
INFO:tensorflow:Tensor's key in saved_model's tensor_map: probabilities
INFO:tensorflow: tensor name: Softmax:0, shape: (-1, 10), type: DT_FLOAT
INFO:tensorflow:Tensor's key in saved_model's tensor_map: classes
INFO:tensorflow: tensor name: ArgMax:0, shape: (-1), type: DT_INT64
INFO:tensorflow:Restoring parameters from /tmp/mnist_saved_model/1569952292/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/lite/python/util.py:249: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
INFO:tensorflow:Froze 8 variables.
INFO:tensorflow:Converted 8 variables to const ops.

Write it out to a tflite file:

tflite_models_dir = pathlib.Path("/tmp/mnist_tflite_models/")
tflite_models_dir.mkdir(exist_ok=True, parents=True)
tflite_model_file = tflite_models_dir/"mnist_model.tflite"
tflite_model_file.write_bytes(tflite_model)
13101352

To quantize the model on export, set the optimizations flag to optimize for size:

# Note: If you don't have a recent tf-nightly installed, the
# "optimizations" line will have no effect.
tf.logging.set_verbosity(tf.logging.INFO)
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
tflite_quant_model = converter.convert()
tflite_model_quant_file = tflite_models_dir/"mnist_model_quant.tflite"
tflite_model_quant_file.write_bytes(tflite_quant_model)
3283312

Note how the resulting file, is approximately 1/4 the size.

!ls -lh {tflite_models_dir}
total 16M
-rw-rw-r-- 1 kbuilder kbuilder 3.2M Oct  1 10:51 mnist_model_quant.tflite
-rw-rw-r-- 1 kbuilder kbuilder  13M Oct  1 10:51 mnist_model.tflite

Run the TFLite models

Run the TensorFlow Lite model using the Python TensorFlow Lite Interpreter.

load the test data

First let's load the mnist test data to feed to it:

import numpy as np
mnist_train, mnist_test = tf.keras.datasets.mnist.load_data()
images, labels = tf.cast(mnist_test[0], tf.float32)/255.0, mnist_test[1]

# Note: If you change the batch size, then use 
# `tf.lite.Interpreter.resize_tensor_input` to also change it for
# the interpreter.
mnist_ds = tf.data.Dataset.from_tensor_slices((images, labels)).batch(1)
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step

Load the model into an interpreter

interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))
interpreter.allocate_tensors()
input_index = interpreter.get_input_details()[0]["index"]
output_index = interpreter.get_output_details()[0]["index"]
tf.logging.set_verbosity(tf.logging.DEBUG)
interpreter_quant = tf.lite.Interpreter(model_path=str(tflite_model_quant_file))
interpreter_quant.allocate_tensors()
input_index = interpreter_quant.get_input_details()[0]["index"]
output_index = interpreter_quant.get_output_details()[0]["index"]

Test the model on one image

for img, label in mnist_ds.take(1):
  break

interpreter.set_tensor(input_index, img)
interpreter.invoke()
predictions = interpreter.get_tensor(output_index)
import matplotlib.pylab as plt

plt.imshow(img[0])
template = "True:{true}, predicted:{predict}"
_ = plt.title(template.format(true= str(label[0].numpy()),
                              predict=str(predictions[0])))
plt.grid(False)

Evaluate the models

def eval_model(interpreter, mnist_ds):
  total_seen = 0
  num_correct = 0

  for img, label in mnist_ds:
    total_seen += 1
    interpreter.set_tensor(input_index, img)
    interpreter.invoke()
    predictions = interpreter.get_tensor(output_index)
    if predictions == label.numpy():
      num_correct += 1

    if total_seen % 500 == 0:
        print("Accuracy after %i images: %f" %
              (total_seen, float(num_correct) / float(total_seen)))

  return float(num_correct) / float(total_seen)
print(eval_model(interpreter, mnist_ds))
Accuracy after 500 images: 0.972000
Accuracy after 1000 images: 0.967000
Accuracy after 1500 images: 0.960000
Accuracy after 2000 images: 0.957500
Accuracy after 2500 images: 0.953600
Accuracy after 3000 images: 0.957000
Accuracy after 3500 images: 0.958571
Accuracy after 4000 images: 0.957000
Accuracy after 4500 images: 0.957333
Accuracy after 5000 images: 0.956600
Accuracy after 5500 images: 0.959636
Accuracy after 6000 images: 0.960000
Accuracy after 6500 images: 0.961538
Accuracy after 7000 images: 0.962429
Accuracy after 7500 images: 0.964267
Accuracy after 8000 images: 0.965875
Accuracy after 8500 images: 0.967176
Accuracy after 9000 images: 0.968889
Accuracy after 9500 images: 0.969895
Accuracy after 10000 images: 0.969200
0.9692

Repeat the evaluation on the weight quantized model to obtain:

print(eval_model(interpreter_quant, mnist_ds))
Accuracy after 500 images: 0.972000
Accuracy after 1000 images: 0.966000
Accuracy after 1500 images: 0.959333
Accuracy after 2000 images: 0.956500
Accuracy after 2500 images: 0.952800
Accuracy after 3000 images: 0.956333
Accuracy after 3500 images: 0.958000
Accuracy after 4000 images: 0.956750
Accuracy after 4500 images: 0.957111
Accuracy after 5000 images: 0.956400
Accuracy after 5500 images: 0.959455
Accuracy after 6000 images: 0.959833
Accuracy after 6500 images: 0.961385
Accuracy after 7000 images: 0.962286
Accuracy after 7500 images: 0.964133
Accuracy after 8000 images: 0.965750
Accuracy after 8500 images: 0.967059
Accuracy after 9000 images: 0.968778
Accuracy after 9500 images: 0.969789
Accuracy after 10000 images: 0.969100
0.9691

In this example, the compressed model has no difference in the accuracy.

Optimizing an existing model

Resnets with pre-activation layers (Resnet-v2) are widely used for vision applications. Pre-trained frozen graph for resnet-v2-101 is available at the Tensorflow Lite model repository.

You can convert the frozen graph to a TensorFLow Lite flatbuffer with quantization by:

archive_path = tf.keras.utils.get_file("resnet_v2_101.tgz", "https://storage.googleapis.com/download.tensorflow.org/models/tflite_11_05_08/resnet_v2_101.tgz", extract=True)
archive_path = pathlib.Path(archive_path)
archive_dir = str(archive_path.parent)
Downloading data from https://storage.googleapis.com/download.tensorflow.org/models/tflite_11_05_08/resnet_v2_101.tgz
831692800/831690572 [==============================] - 8s 0us/step

The info.txt file lists the input and output names. You can also find them using TensorBoard to visually inspect the graph.

! cat {archive_dir}/resnet_v2_101_299_info.txt
Model: resnet_v2_101
Input: input
Output: output
graph_def_file = pathlib.Path(archive_path).parent/"resnet_v2_101_299_frozen.pb"
input_arrays = ["input"] 
output_arrays = ["output"]
converter = tf.lite.TFLiteConverter.from_frozen_graph(
  str(graph_def_file), input_arrays, output_arrays, input_shapes={"input":[1,299,299,3]})
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
resnet_tflite_file = graph_def_file.parent/"resnet_v2_101_quantized.tflite"
resnet_tflite_file.write_bytes(converter.convert())
44997328
!ls -lh {archive_dir}/*.tflite
-rw-r--r-- 1 kbuilder kbuilder 171M Sep  5  2018 /home/kbuilder/.keras/datasets/resnet_v2_101_299.tflite
-rw-rw-r-- 1 kbuilder kbuilder  43M Oct  1 10:52 /home/kbuilder/.keras/datasets/resnet_v2_101_quantized.tflite
-rw-rw-r-- 1 kbuilder kbuilder 2.7M Oct  1 10:48 /home/kbuilder/.keras/datasets/style_predict.tflite
-rw-rw-r-- 1 kbuilder kbuilder 239K Oct  1 10:48 /home/kbuilder/.keras/datasets/style_transform.tflite

The model size reduces from 171 MB to 43 MB. The accuracy of this model on imagenet can be evaluated using the scripts provided for TFLite accuracy measurement.

The optimized model top-1 accuracy is 76.8, the same as the floating point model.