Join us at TensorFlow World, Oct 28-31. Use code TF20 for 20% off select passes. Register now

Build a Convolutional Neural Network using Estimators

View on TensorFlow.org Run in Google Colab View source on GitHub

The tf.layers module provides a high-level API that makes it easy to construct a neural network. It provides methods that facilitate the creation of dense (fully connected) layers and convolutional layers, adding activation functions, and applying dropout regularization. In this tutorial, you'll learn how to use layers to build a convolutional neural network model to recognize the handwritten digits in the MNIST data set.

handwritten digits 0–9 from the MNIST data set

The MNIST dataset comprises 60,000 training examples and 10,000 test examples of the handwritten digits 0–9, formatted as 28x28-pixel monochrome images.

Get Started

Let's set up the imports for our TensorFlow program:

from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf
import numpy as np

tf.logging.set_verbosity(tf.logging.INFO)

Intro to Convolutional Neural Networks

Convolutional neural networks (CNNs) are the current state-of-the-art model architecture for image classification tasks. CNNs apply a series of filters to the raw pixel data of an image to extract and learn higher-level features, which the model can then use for classification. CNNs contains three components:

  • Convolutional layers, which apply a specified number of convolution filters to the image. For each subregion, the layer performs a set of mathematical operations to produce a single value in the output feature map. Convolutional layers then typically apply a ReLU activation function to the output to introduce nonlinearities into the model.

  • Pooling layers, which downsample the image data extracted by the convolutional layers to reduce the dimensionality of the feature map in order to decrease processing time. A commonly used pooling algorithm is max pooling, which extracts subregions of the feature map (e.g., 2x2-pixel tiles), keeps their maximum value, and discards all other values.

  • Dense (fully connected) layers, which perform classification on the features extracted by the convolutional layers and downsampled by the pooling layers. In a dense layer, every node in the layer is connected to every node in the preceding layer.

Typically, a CNN is composed of a stack of convolutional modules that perform feature extraction. Each module consists of a convolutional layer followed by a pooling layer. The last convolutional module is followed by one or more dense layers that perform classification. The final dense layer in a CNN contains a single node for each target class in the model (all the possible classes the model may predict), with a softmax activation function to generate a value between 0–1 for each node (the sum of all these softmax values is equal to 1). We can interpret the softmax values for a given image as relative measurements of how likely it is that the image falls into each target class.

Building the CNN MNIST Classifier

Let's build a model to classify the images in the MNIST dataset using the following CNN architecture:

  1. Convolutional Layer #1: Applies 32 5x5 filters (extracting 5x5-pixel subregions), with ReLU activation function
  2. Pooling Layer #1: Performs max pooling with a 2x2 filter and stride of 2 (which specifies that pooled regions do not overlap)
  3. Convolutional Layer #2: Applies 64 5x5 filters, with ReLU activation function
  4. Pooling Layer #2: Again, performs max pooling with a 2x2 filter and stride of 2
  5. Dense Layer #1: 1,024 neurons, with dropout regularization rate of 0.4 (probability of 0.4 that any given element will be dropped during training)
  6. Dense Layer #2 (Logits Layer): 10 neurons, one for each digit target class (0–9).

The tf.layers module contains methods to create each of the three layer types above:

  • conv2d(). Constructs a two-dimensional convolutional layer. Takes number of filters, filter kernel size, padding, and activation function as arguments.
  • max_pooling2d(). Constructs a two-dimensional pooling layer using the max-pooling algorithm. Takes pooling filter size and stride as arguments.
  • dense(). Constructs a dense layer. Takes number of neurons and activation function as arguments.

Each of these methods accepts a tensor as input and returns a transformed tensor as output. This makes it easy to connect one layer to another: just take the output from one layer-creation method and supply it as input to another.

Add the following cnn_model_fn function, which conforms to the interface expected by TensorFlow's Estimator API (more on this later in Create the Estimator). This function takes MNIST feature data, labels, and mode (from tf.estimator.ModeKeys: TRAIN, EVAL, PREDICT) as arguments; configures the CNN; and returns predictions, loss, and a training operation:

def cnn_model_fn(features, labels, mode):
  """Model function for CNN."""
  # Input Layer
  input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])

  # Convolutional Layer #1
  conv1 = tf.layers.conv2d(
      inputs=input_layer,
      filters=32,
      kernel_size=[5, 5],
      padding="same",
      activation=tf.nn.relu)

  # Pooling Layer #1
  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)

  # Convolutional Layer #2 and Pooling Layer #2
  conv2 = tf.layers.conv2d(
      inputs=pool1,
      filters=64,
      kernel_size=[5, 5],
      padding="same",
      activation=tf.nn.relu)
  pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)

  # Dense Layer
  pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
  dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
  dropout = tf.layers.dropout(
      inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)

  # Logits Layer
  logits = tf.layers.dense(inputs=dropout, units=10)

  predictions = {
      # Generate predictions (for PREDICT and EVAL mode)
      "classes": tf.argmax(input=logits, axis=1),
      # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
      # `logging_hook`.
      "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
  }

  if mode == tf.estimator.ModeKeys.PREDICT:
    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

  # Calculate Loss (for both TRAIN and EVAL modes)
  loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)

  # Configure the Training Op (for TRAIN mode)
  if mode == tf.estimator.ModeKeys.TRAIN:
    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
    train_op = optimizer.minimize(
        loss=loss,
        global_step=tf.train.get_global_step())
    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)

  # Add evaluation metrics (for EVAL mode)
  eval_metric_ops = {
      "accuracy": tf.metrics.accuracy(
          labels=labels, predictions=predictions["classes"])
  }
  return tf.estimator.EstimatorSpec(
      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)

The following sections (with headings corresponding to each code block above) dive deeper into the tf.layers code used to create each layer, as well as how to calculate loss, configure the training op, and generate predictions. If you're already experienced with CNNs and TensorFlow Estimators, and find the above code intuitive, you may want to skim these sections or just skip ahead to "Training and Evaluating the CNN MNIST Classifier".

Input Layer

The methods in the layers module for creating convolutional and pooling layers for two-dimensional image data expect input tensors to have a shape of [batch_size, image_height, image_width, channels] by default. This behavior can be changed using the data_format parameter; defined as follows:

  • batch_size —Size of the subset of examples to use when performing gradient descent during training.
  • image_height —Height of the example images.
  • image_width —Width of the example images.
  • channels —Number of color channels in the example images. For color images, the number of channels is 3 (red, green, blue). For monochrome images, there is just 1 channel (black).
  • data_format —A string, one of channels_last (default) or channels_first. channels_last corresponds to inputs with shape (batch, ..., channels) while channels_first corresponds to inputs with shape (batch, channels, ...).

Here, our MNIST dataset is composed of monochrome 28x28 pixel images, so the desired shape for our input layer is [batch_size, 28, 28, 1].

To convert our input feature map (features) to this shape, we can perform the following reshape operation:

input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])

Note that we've indicated -1 for batch size, which specifies that this dimension should be dynamically computed based on the number of input values in features["x"], holding the size of all other dimensions constant. This allows us to treat batch_size as a hyperparameter that we can tune. For example, if we feed examples into our model in batches of 5, features["x"] will contain 3,920 values (one value for each pixel in each image), and input_layer will have a shape of [5, 28, 28, 1]. Similarly, if we feed examples in batches of 100, features["x"] will contain 78,400 values, and input_layer will have a shape of [100, 28, 28, 1].

Convolutional Layer #1

In our first convolutional layer, we want to apply 32 5x5 filters to the input layer, with a ReLU activation function. We can use the conv2d() method in the layers module to create this layer as follows:

conv1 = tf.layers.conv2d(
    inputs=input_layer,
    filters=32,
    kernel_size=[5, 5],
    padding="same",
    activation=tf.nn.relu)

The inputs argument specifies our input tensor, which must have the shape [batch_size, image_height, image_width, channels]. Here, we're connecting our first convolutional layer to input_layer, which has the shape [batch_size, 28, 28, 1].

The filters argument specifies the number of filters to apply (here, 32), and kernel_size specifies the dimensions of the filters as [<em>height</em>, <em>width</em>]</code> (here, <code>[5, 5]).

TIP: If filter height and width have the same value, you can instead specify a single integer for kernel_size—e.g., kernel_size=5.

The padding argument specifies one of two enumerated values (case-insensitive): valid (default value) or same. To specify that the output tensor should have the same height and width values as the input tensor, we set padding=same here, which instructs TensorFlow to add 0 values to the edges of the input tensor to preserve height and width of 28. (Without padding, a 5x5 convolution over a 28x28 tensor will produce a 24x24 tensor, as there are 24x24 locations to extract a 5x5 tile from a 28x28 grid.)

The activation argument specifies the activation function to apply to the output of the convolution. Here, we specify ReLU activation with tf.nn.relu.

Our output tensor produced by conv2d() has a shape of [batch_size, 28, 28, 32]: the same height and width dimensions as the input, but now with 32 channels holding the output from each of the filters.

Pooling Layer #1

Next, we connect our first pooling layer to the convolutional layer we just created. We can use the max_pooling2d() method in layers to construct a layer that performs max pooling with a 2x2 filter and stride of 2:

pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)

Again, inputs specifies the input tensor, with a shape of [batch_size, image_height, image_width, channels]. Here, our input tensor is conv1, the output from the first convolutional layer, which has a shape of [batch_size, 28, 28, 32].

The pool_size argument specifies the size of the max pooling filter as [height, width] (here, [2, 2]). If both dimensions have the same value, you can instead specify a single integer (e.g., pool_size=2).

The strides argument specifies the size of the stride. Here, we set a stride of 2, which indicates that the subregions extracted by the filter should be separated by 2 pixels in both the height and width dimensions (for a 2x2 filter, this means that none of the regions extracted will overlap). If you want to set different stride values for height and width, you can instead specify a tuple or list (e.g., stride=[3, 6]).

Our output tensor produced by max_pooling2d() (pool1) has a shape of [batch_size, 14, 14, 32]: the 2x2 filter reduces height and width by 50% each.

Convolutional Layer #2 and Pooling Layer #2

We can connect a second convolutional and pooling layer to our CNN using conv2d() and max_pooling2d() as before. For convolutional layer #2, we configure 64 5x5 filters with ReLU activation, and for pooling layer #2, we use the same specs as pooling layer #1 (a 2x2 max pooling filter with stride of 2):

conv2 = tf.layers.conv2d(
    inputs=pool1,
    filters=64,
    kernel_size=[5, 5],
    padding="same",
    activation=tf.nn.relu)

pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)

Note that convolutional layer #2 takes the output tensor of our first pooling layer (pool1) as input, and produces the tensor conv2 as output. conv2 has a shape of [batch_size, 14, 14, 64], the same height and width as pool1 (due to padding="same"), and 64 channels for the 64 filters applied.

Pooling layer #2 takes conv2 as input, producing pool2 as output. pool2 has shape [batch_size, 7, 7, 64] (50% reduction of height and width from conv2).

Dense Layer

Next, we want to add a dense layer (with 1,024 neurons and ReLU activation) to our CNN to perform classification on the features extracted by the convolution/pooling layers. Before we connect the layer, however, we'll flatten our feature map (pool2) to shape [batch_size, features], so that our tensor has only two dimensions:

pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])

In the reshape() operation above, the -1 signifies that the batch_size dimension will be dynamically calculated based on the number of examples in our input data. Each example has 7 (pool2 height) * 7 (pool2 width) * 64 (pool2 channels) features, so we want the features dimension to have a value of 7 * 7 * 64 (3136 in total). The output tensor, pool2_flat, has shape [batch_size, 3136].

Now, we can use the dense() method in layers to connect our dense layer as follows:

dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)

The inputs argument specifies the input tensor: our flattened feature map, pool2_flat. The units argument specifies the number of neurons in the dense layer (1,024). The activation argument takes the activation function; again, we'll use tf.nn.relu to add ReLU activation.

To help improve the results of our model, we also apply dropout regularization to our dense layer, using the dropout method in layers:

dropout = tf.layers.dropout(
    inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)

Again, inputs specifies the input tensor, which is the output tensor from our dense layer (dense).

The rate argument specifies the dropout rate; here, we use 0.4, which means 40% of the elements will be randomly dropped out during training.

The training argument takes a boolean specifying whether or not the model is currently being run in training mode; dropout will only be performed if training is True. Here, we check if the mode passed to our model function cnn_model_fn is TRAIN mode.

Our output tensor dropout has shape [batch_size, 1024].

Logits Layer

The final layer in our neural network is the logits layer, which will return the raw values for our predictions. We create a dense layer with 10 neurons (one for each target class 0–9), with linear activation (the default):

logits = tf.layers.dense(inputs=dropout, units=10)

Our final output tensor of the CNN, logits, has shape [batch_size, 10].

Generate Predictions {#generate_predictions}

The logits layer of our model returns our predictions as raw values in a [batch_size, 10]-dimensional tensor. Let's convert these raw values into two different formats that our model function can return:

  • The predicted class for each example: a digit from 0–9.
  • The probabilities for each possible target class for each example: the probability that the example is a 0, is a 1, is a 2, etc.

For a given example, our predicted class is the element in the corresponding row of the logits tensor with the highest raw value. We can find the index of this element using the tf.argmax function:

tf.argmax(input=logits, axis=1)

The input argument specifies the tensor from which to extract maximum values—here logits. The axis argument specifies the axis of the input tensor along which to find the greatest value. Here, we want to find the largest value along the dimension with index of 1, which corresponds to our predictions (recall that our logits tensor has shape [batch_size, 10]).

We can derive probabilities from our logits layer by applying softmax activation using tf.nn.softmax:

tf.nn.softmax(logits, name="softmax_tensor")

We compile our predictions in a dict, and return an EstimatorSpec object:

predictions = {
    "classes": tf.argmax(input=logits, axis=1),
    "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
}
if mode == tf.estimator.ModeKeys.PREDICT:
  return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

Calculate Loss {#calculating_loss}

For both training and evaluation, we need to define a loss function that measures how closely the model's predictions match the target classes. For multiclass classification problems like MNIST, cross entropy is typically used as the loss metric. The following code calculates cross entropy when the model runs in either TRAIN or EVAL mode:

loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)

Let's take a closer look at what's happening above.

Our labels tensor contains a list of prediction indices for our examples, e.g. [1, 9, ...]. logits contains the linear outputs of our last layer.

tf.losses.sparse_softmax_cross_entropy, calculates the softmax crossentropy (aka: categorical crossentropy, negative log-likelihood) from these two inputs in an efficient, numerically stable way.

Configure the Training Op

In the previous section, we defined loss for our CNN as the softmax cross-entropy of the logits layer and our labels. Let's configure our model to optimize this loss value during training. We'll use a learning rate of 0.001 and stochastic gradient descent as the optimization algorithm:

if mode == tf.estimator.ModeKeys.TRAIN:
  optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
  train_op = optimizer.minimize(
      loss=loss,
      global_step=tf.train.get_global_step())
  return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)

Add evaluation metrics

To add accuracy metric in our model, we define eval_metric_ops dict in EVAL mode as follows:

eval_metric_ops = {
    "accuracy": tf.metrics.accuracy(
        labels=labels, predictions=predictions["classes"])
}
return tf.estimator.EstimatorSpec(
    mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)

Training and Evaluating the CNN MNIST Classifier

We've coded our MNIST CNN model function; now we're ready to train and evaluate it.

Load Training and Test Data

First, let's load our training and test data with the following code:

# Load training and eval data
((train_data, train_labels),
 (eval_data, eval_labels)) = tf.keras.datasets.mnist.load_data()

train_data = train_data/np.float32(255)
train_labels = train_labels.astype(np.int32)  # not required

eval_data = eval_data/np.float32(255)
eval_labels = eval_labels.astype(np.int32)  # not required
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step

We store the training feature data (the raw pixel values for 55,000 images of hand-drawn digits) and training labels (the corresponding value from 0–9 for each image) as numpy arrays in train_data and train_labels, respectively. Similarly, we store the evaluation feature data (10,000 images) and evaluation labels in eval_data and eval_labels, respectively.

Create the Estimator {#create_the_estimator}

Next, let's create an Estimator (a TensorFlow class for performing high-level model training, evaluation, and inference) for our model. Add the following code to main():

# Create the Estimator
mnist_classifier = tf.estimator.Estimator(
    model_fn=cnn_model_fn, model_dir="/tmp/mnist_convnet_model")
WARNING: Logging before flag parsing goes to stderr.
I0625 16:03:31.535491 140067097196288 estimator.py:1790] Using default config.
I0625 16:03:31.537065 140067097196288 estimator.py:209] Using config: {'_save_summary_steps': 100, '_eval_distribute': None, '_model_dir': '/tmp/mnist_convnet_model', '_master': '', '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_device_fn': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_train_distribute': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f63dd60ec88>, '_service': None, '_tf_random_seed': None, '_global_id_in_cluster': 0, '_save_checkpoints_steps': None, '_is_chief': True, '_keep_checkpoint_max': 5, '_evaluation_master': '', '_keep_checkpoint_every_n_hours': 10000, '_task_type': 'worker', '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_log_step_count_steps': 100, '_save_checkpoints_secs': 600, '_protocol': None}

The model_fn argument specifies the model function to use for training, evaluation, and prediction; we pass it the cnn_model_fn we created in "Building the CNN MNIST Classifier." The model_dir argument specifies the directory where model data (checkpoints) will be saved (here, we specify the temp directory /tmp/mnist_convnet_model, but feel free to change to another directory of your choice).

Set Up a Logging Hook {#set_up_a_logging_hook}

Since CNNs can take a while to train, let's set up some logging so we can track progress during training. We can use TensorFlow's tf.train.SessionRunHook to create a tf.train.LoggingTensorHook that will log the probability values from the softmax layer of our CNN. Add the following to main():

# Set up logging for predictions
tensors_to_log = {"probabilities": "softmax_tensor"}

logging_hook = tf.train.LoggingTensorHook(
    tensors=tensors_to_log, every_n_iter=50)

We store a dict of the tensors we want to log in tensors_to_log. Each key is a label of our choice that will be printed in the log output, and the corresponding label is the name of a Tensor in the TensorFlow graph. Here, our probabilities can be found in softmax_tensor, the name we gave our softmax operation earlier when we generated the probabilities in cnn_model_fn.

Next, we create the LoggingTensorHook, passing tensors_to_log to the tensors argument. We set every_n_iter=50, which specifies that probabilities should be logged after every 50 steps of training.

Train the Model

Now we're ready to train our model, which we can do by creating train_input_fn and calling train() on mnist_classifier. In the numpy_input_fn call, we pass the training feature data and labels to x (as a dict) and y, respectively. We set a batch_size of 100 (which means that the model will train on minibatches of 100 examples at each step). num_epochs=None means that the model will train until the specified number of steps is reached. We also set shuffle=True to shuffle the training data. Then train the model a single step and log the output:

# Train the model
train_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": train_data},
    y=train_labels,
    batch_size=100,
    num_epochs=None,
    shuffle=True)

# train one step and display the probabilties
mnist_classifier.train(
    input_fn=train_input_fn,
    steps=1,
    hooks=[logging_hook])
W0625 16:03:31.578685 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0625 16:03:31.595290 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:62: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
W0625 16:03:31.597698 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:500: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
I0625 16:03:31.608231 140067097196288 estimator.py:1145] Calling model_fn.
W0625 16:03:31.610746 140067097196288 deprecation.py:323] From <ipython-input-3-16a10fb7f348>:12: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0625 16:03:31.613602 140067097196288 deprecation.py:506] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0625 16:03:31.841041 140067097196288 deprecation.py:323] From <ipython-input-3-16a10fb7f348>:15: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0625 16:03:31.996138 140067097196288 deprecation.py:323] From <ipython-input-3-16a10fb7f348>:28: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
W0625 16:03:32.307477 140067097196288 deprecation.py:323] From <ipython-input-3-16a10fb7f348>:30: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
W0625 16:03:32.522615 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I0625 16:03:32.632307 140067097196288 estimator.py:1147] Done calling model_fn.
I0625 16:03:32.635431 140067097196288 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
I0625 16:03:32.760613 140067097196288 monitored_session.py:240] Graph was finalized.
I0625 16:03:34.002932 140067097196288 session_manager.py:500] Running local_init_op.
I0625 16:03:34.011360 140067097196288 session_manager.py:502] Done running local_init_op.
W0625 16:03:34.035200 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py:875: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
I0625 16:03:34.196981 140067097196288 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /tmp/mnist_convnet_model/model.ckpt.
I0625 16:03:36.112093 140067097196288 basic_session_run_hooks.py:262] probabilities = [[0.09097217 0.09048758 0.0998596  0.09649529 0.09494466 0.10511343
  0.11121551 0.10612831 0.10476618 0.10001727]
 [0.10413646 0.0923803  0.09999727 0.11231551 0.09655558 0.0928714
  0.09915487 0.09720246 0.10958957 0.09579661]
 [0.09682328 0.09830856 0.10512023 0.1098512  0.09649026 0.0869299
  0.09935334 0.10295186 0.10354603 0.10062541]
 [0.08301835 0.0864645  0.09789587 0.11889298 0.09937818 0.09290722
  0.10593288 0.10331158 0.11751608 0.09468235]
 [0.08290824 0.09011602 0.09569915 0.10470317 0.08635037 0.08399618
  0.1131844  0.11127329 0.12014707 0.11162213]
 [0.09261602 0.09723736 0.09689444 0.1116029  0.09803282 0.09754211
  0.10951941 0.10310103 0.10552932 0.08792457]
 [0.10759767 0.09020805 0.09325988 0.09528298 0.10476671 0.09890428
  0.10416283 0.10910536 0.11133875 0.08537348]
 [0.09960581 0.10021449 0.09564672 0.10103224 0.09700084 0.09533523
  0.1069309  0.09552898 0.10777292 0.10093191]
 [0.08870408 0.09874253 0.09445115 0.10342291 0.09423342 0.09637831
  0.11554436 0.09469026 0.11679831 0.09703473]
 [0.09104782 0.09083612 0.10584343 0.1005907  0.08913624 0.0990207
  0.11114144 0.09671755 0.11921237 0.09645356]
 [0.09880769 0.09344607 0.09365786 0.10434978 0.10328294 0.09807172
  0.10978374 0.09509667 0.10347418 0.10002936]
 [0.10638998 0.09405933 0.10268114 0.10932288 0.09802047 0.09040378
  0.103087   0.09810701 0.10463331 0.0932951 ]
 [0.08653034 0.08752086 0.09305898 0.1001475  0.10557776 0.08722819
  0.11909232 0.10080797 0.11921199 0.10082408]
 [0.09889446 0.0915235  0.08730438 0.11313409 0.10067123 0.10187875
  0.1034969  0.10083218 0.10547388 0.09679066]
 [0.09827913 0.09018061 0.09160279 0.11531482 0.0953174  0.10265336
  0.1131882  0.09839744 0.10585345 0.08921284]
 [0.09731895 0.09171987 0.09797716 0.10263161 0.11258632 0.08519374
  0.11242405 0.08855369 0.11908621 0.09250839]
 [0.08666116 0.09471314 0.0951551  0.11840754 0.08882163 0.09604281
  0.10663031 0.10530084 0.11629172 0.09197569]
 [0.08654637 0.0971511  0.10373887 0.11384418 0.09635454 0.09430306
  0.10750031 0.09403692 0.12324966 0.08327496]
 [0.10752177 0.09912755 0.09427822 0.09700193 0.1053421  0.08638754
  0.10419965 0.10289543 0.09627241 0.10697338]
 [0.08560034 0.09962375 0.09402138 0.11016365 0.09424843 0.0903488
  0.10906228 0.10520479 0.10781861 0.10390803]
 [0.09114564 0.09595553 0.11759496 0.1085892  0.08735003 0.08971652
  0.11761052 0.09762517 0.10883282 0.08557963]
 [0.09374272 0.09406927 0.09784926 0.10128739 0.09682406 0.09411498
  0.11213344 0.1042427  0.11193311 0.09380311]
 [0.09230128 0.10132714 0.10648091 0.10289653 0.10502308 0.09651634
  0.10299165 0.09777021 0.10968333 0.08500956]
 [0.0972426  0.09216306 0.09444319 0.10873286 0.09367811 0.10160879
  0.09983595 0.10221524 0.11950661 0.09057362]
 [0.0962598  0.09252594 0.10033576 0.09899239 0.10204493 0.0993699
  0.10032111 0.1016204  0.10486152 0.10366822]
 [0.08954873 0.10393356 0.1000989  0.10308198 0.10945632 0.08699777
  0.1106786  0.10099205 0.1088796  0.08633245]
 [0.0983408  0.09310494 0.09760918 0.10663613 0.09991597 0.09055817
  0.10833436 0.10131598 0.10843059 0.09575388]
 [0.10161913 0.10471549 0.08948921 0.09876378 0.09701937 0.0913737
  0.10462677 0.10392483 0.11309955 0.09536824]
 [0.08857052 0.09306807 0.09623126 0.10214064 0.09933344 0.10022234
  0.10562577 0.10105801 0.12254796 0.09120193]
 [0.09716561 0.09085734 0.09788045 0.12576534 0.10049953 0.09140084
  0.10511465 0.08773641 0.10355029 0.10002953]
 [0.09110674 0.09904964 0.08644637 0.10003748 0.09525018 0.11005495
  0.10176451 0.10374242 0.11385167 0.09869597]
 [0.09828661 0.09240715 0.10423797 0.10623311 0.08587241 0.09704107
  0.11276066 0.10648152 0.1040477  0.09263182]
 [0.08754804 0.10920493 0.09301102 0.11490015 0.08536354 0.08474217
  0.11057325 0.10019831 0.10478798 0.10967062]
 [0.09362096 0.1028135  0.09856459 0.10414369 0.09921423 0.08601185
  0.11066504 0.10323316 0.09863988 0.10309309]
 [0.09445197 0.08662976 0.10207856 0.10731391 0.10205653 0.10269167
  0.10513522 0.10655745 0.09107671 0.10200815]
 [0.09676581 0.0923213  0.0923676  0.09527522 0.09659824 0.09458804
  0.1043456  0.11350614 0.12584312 0.08838893]
 [0.09336933 0.10312263 0.09373205 0.09438696 0.09312181 0.09452382
  0.10088845 0.12500872 0.10653479 0.09531144]
 [0.09593049 0.08909184 0.09506837 0.10833663 0.08886851 0.1070371
  0.10503263 0.10223669 0.10474315 0.10365468]
 [0.09701952 0.09117085 0.10260189 0.09963723 0.09703425 0.10723366
  0.09819555 0.11151092 0.10309505 0.09250106]
 [0.09204267 0.09197177 0.10421135 0.09975231 0.10167068 0.09725484
  0.10527208 0.10294405 0.10859209 0.09628814]
 [0.09406437 0.09577721 0.08611102 0.09907297 0.09426041 0.09797491
  0.11821763 0.09371059 0.1278954  0.09291547]
 [0.08559245 0.09868842 0.09683178 0.10840756 0.09819238 0.0872064
  0.11313252 0.11453568 0.10117117 0.09624162]
 [0.09802126 0.09564933 0.10530005 0.1018973  0.09851474 0.09366398
  0.10586388 0.10726001 0.10070077 0.09312868]
 [0.10805877 0.09590426 0.09590337 0.10442537 0.10455374 0.08533593
  0.10529568 0.08861348 0.10775486 0.10415447]
 [0.0939979  0.09376053 0.08689635 0.10818817 0.11057528 0.09002888
  0.11094701 0.10877799 0.10343443 0.09339351]
 [0.10848582 0.09213281 0.09640264 0.11481884 0.09359614 0.08314244
  0.10150737 0.09774991 0.1158528  0.09631129]
 [0.09813016 0.08919827 0.09308346 0.09753448 0.09254092 0.09803109
  0.10456806 0.11181977 0.12093796 0.0941558 ]
 [0.11057237 0.09966362 0.09575909 0.1106533  0.09791888 0.08493685
  0.10162352 0.09702307 0.11414289 0.08770645]
 [0.09408525 0.10689355 0.0955866  0.10244057 0.09376117 0.09345775
  0.1044287  0.101138   0.09885433 0.10935408]
 [0.10198554 0.09589057 0.09793003 0.1040794  0.09977134 0.09130817
  0.09738292 0.10249962 0.10791553 0.10123689]
 [0.08192728 0.10734563 0.10065275 0.08893523 0.10021116 0.09006823
  0.10906356 0.10346971 0.11356635 0.10476013]
 [0.09049617 0.08407087 0.09755797 0.12012581 0.09426241 0.09228384
  0.11017061 0.10072515 0.11587916 0.0944281 ]
 [0.09821519 0.09693043 0.09854466 0.10170706 0.10205024 0.09606478
  0.11014181 0.09666856 0.11348628 0.08619099]
 [0.10080083 0.09962792 0.09860586 0.1014617  0.09080013 0.09159436
  0.09066562 0.115434   0.12396654 0.08704298]
 [0.09130672 0.09280578 0.10077082 0.109362   0.09606735 0.09680945
  0.10387707 0.09282545 0.11506645 0.10110892]
 [0.10199423 0.08135835 0.10437623 0.10487282 0.08433328 0.10154983
  0.10741655 0.11277369 0.11019267 0.09113234]
 [0.08924133 0.10663446 0.09348728 0.10563488 0.08867469 0.08536047
  0.1032244  0.10334743 0.11700251 0.10739249]
 [0.08684957 0.09500177 0.0857219  0.10688379 0.09522104 0.09857043
  0.09537487 0.1139521  0.12577842 0.0966461 ]
 [0.09589642 0.10204976 0.08627791 0.10327102 0.09248792 0.10025216
  0.09962965 0.10230035 0.11665237 0.10118239]
 [0.09700378 0.09507774 0.09894656 0.10356716 0.10434826 0.10102426
  0.09948    0.10218389 0.10774951 0.09061878]
 [0.08980236 0.09677151 0.09365343 0.09800218 0.09876654 0.09282535
  0.11273904 0.1033929  0.12045822 0.09358855]
 [0.09856404 0.10045265 0.09050646 0.11032531 0.10199763 0.08883871
  0.10851467 0.09945282 0.10258818 0.09875948]
 [0.09477144 0.10418438 0.09876866 0.10418589 0.08243235 0.09713686
  0.11054488 0.10233277 0.11810756 0.0875352 ]
 [0.09941852 0.1004591  0.09628823 0.10143016 0.09691712 0.09860576
  0.10941321 0.10147849 0.10578059 0.09020879]
 [0.0969332  0.10200457 0.08624222 0.10479336 0.08613723 0.09984273
  0.11358702 0.09700589 0.1166586  0.09679519]
 [0.0935888  0.09502745 0.08950844 0.10740398 0.09994859 0.09377708
  0.1046567  0.10072508 0.10859803 0.10676583]
 [0.09791592 0.08802851 0.10553321 0.10762385 0.09092929 0.09676573
  0.11749914 0.09442197 0.11315301 0.08812949]
 [0.09778352 0.09634145 0.09930228 0.11217428 0.09073611 0.09514435
  0.09941217 0.09950389 0.10352556 0.10607629]
 [0.09594994 0.10749433 0.0994332  0.10910659 0.08904584 0.07973965
  0.11056878 0.10552227 0.10767118 0.09546824]
 [0.1072707  0.09528756 0.09302171 0.10874888 0.09488777 0.09029051
  0.11083339 0.08838288 0.10755372 0.10372289]
 [0.0985786  0.09485463 0.10189337 0.10349803 0.10005637 0.08647601
  0.09937482 0.10792945 0.11045362 0.09688503]
 [0.0876409  0.09295518 0.09671902 0.10142554 0.10294478 0.09868973
  0.11241304 0.1009692  0.10571536 0.10052729]
 [0.0969674  0.08881083 0.09371341 0.11511463 0.09415448 0.0875365
  0.11091228 0.10483094 0.11982792 0.0881316 ]
 [0.08739841 0.08986755 0.09927898 0.10688349 0.10533407 0.09223189
  0.11182262 0.10081982 0.11707404 0.0892892 ]
 [0.09151157 0.09645481 0.09525654 0.10140791 0.09739725 0.10448439
  0.10519508 0.10755968 0.1073114  0.09342137]
 [0.08221615 0.09782939 0.08974703 0.10721473 0.09045499 0.09689777
  0.11262052 0.1009571  0.12012634 0.10193598]
 [0.09516714 0.0896734  0.0960971  0.10270105 0.09412973 0.10069565
  0.10644829 0.10166651 0.10592985 0.10749131]
 [0.09571372 0.09280181 0.09385815 0.10968047 0.1041878  0.09368565
  0.10205988 0.10266703 0.10854593 0.09679952]
 [0.09887527 0.09708045 0.09124357 0.10495408 0.09649721 0.10095372
  0.10482283 0.09295216 0.1118442  0.10077656]
 [0.10732683 0.09678441 0.09144063 0.10380818 0.10505723 0.09520999
  0.10431528 0.09472112 0.10938089 0.09195554]
 [0.08825699 0.09800744 0.09935553 0.10148741 0.09547417 0.09665556
  0.10165371 0.1058194  0.11000937 0.10328037]
 [0.08758155 0.09645443 0.09734137 0.10060754 0.09910477 0.09572277
  0.10684495 0.0878213  0.12329368 0.10522771]
 [0.08690117 0.08820241 0.10743979 0.10824974 0.09668894 0.09743525
  0.10880229 0.09224957 0.11130298 0.10272782]
 [0.09346806 0.09380108 0.08965503 0.09513464 0.11067409 0.10044438
  0.10642345 0.10751949 0.11137477 0.091505  ]
 [0.10208255 0.09016952 0.10665252 0.1101231  0.10067504 0.09406389
  0.09911698 0.09480885 0.11159619 0.09071141]
 [0.09783906 0.09845503 0.08922756 0.09470557 0.11032111 0.09442204
  0.10127091 0.10563336 0.10871743 0.09940798]
 [0.09144014 0.09743812 0.09357249 0.09691977 0.10343023 0.09593147
  0.09740849 0.10519391 0.1138106  0.10485476]
 [0.09017909 0.0970815  0.1181006  0.08707509 0.10602573 0.08064039
  0.1126327  0.10422812 0.10677964 0.09725721]
 [0.0985798  0.09465363 0.08378213 0.11175994 0.09849033 0.09554493
  0.10098224 0.09984832 0.12056424 0.09579445]
 [0.10767455 0.09995147 0.09624223 0.09528501 0.10181282 0.09282509
  0.10995442 0.10084565 0.09559172 0.09981705]
 [0.09117978 0.09736143 0.09007578 0.11201868 0.09723372 0.09257448
  0.1063298  0.10005602 0.11541212 0.09775817]
 [0.10284792 0.09784615 0.09983233 0.09777229 0.08969183 0.09825333
  0.10486608 0.10622028 0.10184889 0.10082093]
 [0.10791031 0.10043948 0.09711371 0.09878362 0.09714545 0.09260079
  0.09830026 0.10651492 0.10469904 0.09649243]
 [0.09928722 0.09652156 0.10142608 0.10976052 0.09699388 0.09401559
  0.10270787 0.09448071 0.11393198 0.09087459]
 [0.08385981 0.09468715 0.09734431 0.10375801 0.097099   0.09665059
  0.10761864 0.11251785 0.10676543 0.09969918]
 [0.08878472 0.08922949 0.0897506  0.11246076 0.1063939  0.08480833
  0.10599921 0.10516755 0.1141916  0.10321388]
 [0.08034097 0.09674594 0.09526023 0.1056294  0.09574419 0.08988566
  0.11211209 0.10139988 0.12785338 0.09502822]
 [0.09208388 0.10139605 0.102355   0.10273887 0.09437061 0.09768627
  0.11518363 0.10494093 0.09870492 0.09053981]
 [0.10021859 0.10046367 0.10670132 0.1019779  0.10421495 0.0851501
  0.1009765  0.09938399 0.10290223 0.09801073]
 [0.0894952  0.0897708  0.0991473  0.10625338 0.09520555 0.083684
  0.11445771 0.10524634 0.12597914 0.0907606 ]]
I0625 16:03:36.114122 140067097196288 basic_session_run_hooks.py:262] loss = 2.3144946, step = 0
I0625 16:03:36.116354 140067097196288 basic_session_run_hooks.py:606] Saving checkpoints for 1 into /tmp/mnist_convnet_model/model.ckpt.
I0625 16:03:36.233864 140067097196288 estimator.py:368] Loss for final step: 2.3144946.

<tensorflow_estimator.python.estimator.estimator.Estimator at 0x7f63dd60ec50>

Now—without logging each step—set steps=1000 to train the model longer, but in a reasonable time to run this example. Training CNNs is computationally intensive. To increase the accuracy of your model, increase the number of steps passed to train(), like 20,000 steps.

mnist_classifier.train(input_fn=train_input_fn, steps=1000)
I0625 16:03:36.302984 140067097196288 estimator.py:1145] Calling model_fn.
I0625 16:03:36.523211 140067097196288 estimator.py:1147] Done calling model_fn.
I0625 16:03:36.526391 140067097196288 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
I0625 16:03:36.616043 140067097196288 monitored_session.py:240] Graph was finalized.
W0625 16:03:36.621677 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I0625 16:03:36.624543 140067097196288 saver.py:1280] Restoring parameters from /tmp/mnist_convnet_model/model.ckpt-1
W0625 16:03:36.664747 140067097196288 deprecation.py:323] From /home/kbuilder/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1066: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
I0625 16:03:36.692198 140067097196288 session_manager.py:500] Running local_init_op.
I0625 16:03:36.698947 140067097196288 session_manager.py:502] Done running local_init_op.
I0625 16:03:36.888452 140067097196288 basic_session_run_hooks.py:606] Saving checkpoints for 1 into /tmp/mnist_convnet_model/model.ckpt.
I0625 16:03:37.039489 140067097196288 basic_session_run_hooks.py:262] loss = 2.308238, step = 1
I0625 16:03:37.423969 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 259.7
I0625 16:03:37.426251 140067097196288 basic_session_run_hooks.py:260] loss = 2.2883432, step = 101 (0.387 sec)
I0625 16:03:37.743056 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 313.362
I0625 16:03:37.745036 140067097196288 basic_session_run_hooks.py:260] loss = 2.2824674, step = 201 (0.319 sec)
I0625 16:03:38.056281 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 319.262
I0625 16:03:38.058346 140067097196288 basic_session_run_hooks.py:260] loss = 2.2429779, step = 301 (0.313 sec)
I0625 16:03:38.373447 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 315.31
I0625 16:03:38.375437 140067097196288 basic_session_run_hooks.py:260] loss = 2.2049856, step = 401 (0.317 sec)
I0625 16:03:38.684617 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 321.41
I0625 16:03:38.686816 140067097196288 basic_session_run_hooks.py:260] loss = 2.2014468, step = 501 (0.311 sec)
I0625 16:03:39.010223 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 307.082
I0625 16:03:39.012194 140067097196288 basic_session_run_hooks.py:260] loss = 2.1512933, step = 601 (0.325 sec)
I0625 16:03:39.335533 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 307.383
I0625 16:03:39.337497 140067097196288 basic_session_run_hooks.py:260] loss = 2.1272545, step = 701 (0.325 sec)
I0625 16:03:39.645028 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 323.137
I0625 16:03:39.647317 140067097196288 basic_session_run_hooks.py:260] loss = 2.0234334, step = 801 (0.310 sec)
I0625 16:03:39.963176 140067097196288 basic_session_run_hooks.py:692] global_step/sec: 314.314
I0625 16:03:39.965477 140067097196288 basic_session_run_hooks.py:260] loss = 2.0139112, step = 901 (0.318 sec)
I0625 16:03:40.283510 140067097196288 basic_session_run_hooks.py:606] Saving checkpoints for 1001 into /tmp/mnist_convnet_model/model.ckpt.
I0625 16:03:40.365504 140067097196288 estimator.py:368] Loss for final step: 1.8406956.

<tensorflow_estimator.python.estimator.estimator.Estimator at 0x7f63dd60ec50>

Evaluate the Model

Once training is complete, we want to evaluate our model to determine its accuracy on the MNIST test set. We call the evaluate method, which evaluates the metrics we specified in eval_metric_ops argument in the model_fn. Add the following to main():

eval_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": eval_data},
    y=eval_labels,
    num_epochs=1,
    shuffle=False)

eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)
print(eval_results)
I0625 16:03:40.425752 140067097196288 estimator.py:1145] Calling model_fn.
I0625 16:03:40.554706 140067097196288 estimator.py:1147] Done calling model_fn.
I0625 16:03:40.579329 140067097196288 evaluation.py:255] Starting evaluation at 2019-06-25T16:03:40Z
I0625 16:03:40.674911 140067097196288 monitored_session.py:240] Graph was finalized.
I0625 16:03:40.680154 140067097196288 saver.py:1280] Restoring parameters from /tmp/mnist_convnet_model/model.ckpt-1001
I0625 16:03:40.739500 140067097196288 session_manager.py:500] Running local_init_op.
I0625 16:03:40.753200 140067097196288 session_manager.py:502] Done running local_init_op.
I0625 16:03:41.043090 140067097196288 evaluation.py:275] Finished evaluation at 2019-06-25-16:03:41
I0625 16:03:41.044250 140067097196288 estimator.py:2039] Saving dict for global step 1001: accuracy = 0.7311, global_step = 1001, loss = 1.8677098
I0625 16:03:41.082953 140067097196288 estimator.py:2099] Saving 'checkpoint_path' summary for global step 1001: /tmp/mnist_convnet_model/model.ckpt-1001

{'loss': 1.8677098, 'global_step': 1001, 'accuracy': 0.7311}

To create eval_input_fn, we set num_epochs=1, so that the model evaluates the metrics over one epoch of data and returns the result. We also set shuffle=False to iterate through the data sequentially.

Additional Resources

To learn more about TensorFlow Estimators and CNNs in TensorFlow, see the following resources:

  • Creating Estimators in tf.estimator provides an introduction to the TensorFlow Estimator API. It walks through configuring an Estimator, writing a model function, calculating loss, and defining a training op.
  • Advanced Convolutional Neural Networks walks through how to build a MNIST CNN classification model without estimators using lower-level TensorFlow operations.