Keras

View on TensorFlow.org Run in Google Colab View source on GitHub

Keras is a high-level API to build and train deep learning models. It's used for fast prototyping, advanced research, and production, with three key advantages:

  • User friendly
    Keras has a simple, consistent interface optimized for common use cases. It provides clear and actionable feedback for user errors.
  • Modular and composable
    Keras models are made by connecting configurable building blocks together, with few restrictions.
  • Easy to extend
    Write custom building blocks to express new ideas for research. Create new layers, loss functions, and develop state-of-the-art models.

Import tf.keras

tf.keras is TensorFlow's implementation of the Keras API specification. This is a high-level API to build and train models that includes first-class support for TensorFlow-specific functionality, such as eager execution, tf.data pipelines, and Estimators. tf.keras makes TensorFlow easier to use without sacrificing flexibility and performance.

To get started, import tf.keras as part of your TensorFlow program setup:

!pip install -q pyyaml  # pyyaml is an optional
import tensorflow as tf
from tensorflow.keras import layers

print(tf.VERSION)
print(tf.keras.__version__)
1.12.0
2.1.6-tf

tf.keras can run any Keras-compatible code, but keep in mind:

Build a simple model

Sequential model

In Keras, you assemble layers to build models. A model is (usually) a graph of layers. The most common type of model is a stack of layers: the tf.keras.Sequential model.

To build a simple, fully-connected network (i.e. multi-layer perceptron):

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation='relu'))
# Add another:
model.add(layers.Dense(64, activation='relu'))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation='softmax'))

Configure the layers

There are many tf.keras.layers available with some common constructor parameters:

  • activation: Set the activation function for the layer. This parameter is specified by the name of a built-in function or as a callable object. By default, no activation is applied.
  • kernel_initializer and bias_initializer: The initialization schemes that create the layer's weights (kernel and bias). This parameter is a name or a callable object. This defaults to the "Glorot uniform" initializer.
  • kernel_regularizer and bias_regularizer: The regularization schemes that apply the layer's weights (kernel and bias), such as L1 or L2 regularization. By default, no regularization is applied.

The following instantiates tf.keras.layers.Dense layers using constructor arguments:

# Create a sigmoid layer:
layers.Dense(64, activation='sigmoid')
# Or:
layers.Dense(64, activation=tf.sigmoid)

# A linear layer with L1 regularization of factor 0.01 applied to the kernel matrix:
layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l1(0.01))

# A linear layer with L2 regularization of factor 0.01 applied to the bias vector:
layers.Dense(64, bias_regularizer=tf.keras.regularizers.l2(0.01))

# A linear layer with a kernel initialized to a random orthogonal matrix:
layers.Dense(64, kernel_initializer='orthogonal')

# A linear layer with a bias vector initialized to 2.0s:
layers.Dense(64, bias_initializer=tf.keras.initializers.constant(2.0))

Train and evaluate

Set up training

After the model is constructed, configure its learning process by calling the compile method:

model = tf.keras.Sequential([
# Adds a densely-connected layer with 64 units to the model:
layers.Dense(64, activation='relu'),
# Add another:
layers.Dense(64, activation='relu'),
# Add a softmax layer with 10 output units:
layers.Dense(10, activation='softmax')])

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

tf.keras.Model.compile takes three important arguments:

The following shows a few examples of configuring a model for training:

# Configure a model for mean-squared error regression.
model.compile(optimizer=tf.train.AdamOptimizer(0.01),
              loss='mse',       # mean squared error
              metrics=['mae'])  # mean absolute error

# Configure a model for categorical classification.
model.compile(optimizer=tf.train.RMSPropOptimizer(0.01),
              loss=tf.keras.losses.categorical_crossentropy,
              metrics=[tf.keras.metrics.categorical_accuracy])

Input NumPy data

For small datasets, use in-memory NumPy arrays to train and evaluate a model. The model is "fit" to the training data using the fit method:

import numpy as np

data = np.random.random((1000, 32))
labels = np.random.random((1000, 10))

model.fit(data, labels, epochs=10, batch_size=32)
Epoch 1/10
1000/1000 [==============================] - 0s 218us/step - loss: 11.6670 - categorical_accuracy: 0.0950
Epoch 2/10
1000/1000 [==============================] - 0s 43us/step - loss: 11.5966 - categorical_accuracy: 0.1040
Epoch 3/10
1000/1000 [==============================] - 0s 41us/step - loss: 11.5854 - categorical_accuracy: 0.1160
Epoch 4/10
1000/1000 [==============================] - 0s 50us/step - loss: 11.5768 - categorical_accuracy: 0.0970
Epoch 5/10
1000/1000 [==============================] - 0s 53us/step - loss: 11.5758 - categorical_accuracy: 0.1000
Epoch 6/10
1000/1000 [==============================] - 0s 62us/step - loss: 11.5733 - categorical_accuracy: 0.1020
Epoch 7/10
1000/1000 [==============================] - 0s 61us/step - loss: 11.5774 - categorical_accuracy: 0.1050
Epoch 8/10
1000/1000 [==============================] - 0s 61us/step - loss: 11.5712 - categorical_accuracy: 0.1110
Epoch 9/10
1000/1000 [==============================] - 0s 66us/step - loss: 11.5688 - categorical_accuracy: 0.1220
Epoch 10/10
1000/1000 [==============================] - 0s 60us/step - loss: 11.5668 - categorical_accuracy: 0.1240

tf.keras.Model.fit takes three important arguments:

  • epochs: Training is structured into epochs. An epoch is one iteration over the entire input data (this is done in smaller batches).
  • batch_size: When passed NumPy data, the model slices the data into smaller batches and iterates over these batches during training. This integer specifies the size of each batch. Be aware that the last batch may be smaller if the total number of samples is not divisible by the batch size.
  • validation_data: When prototyping a model, you want to easily monitor its performance on some validation data. Passing this argument—a tuple of inputs and labels—allows the model to display the loss and metrics in inference mode for the passed data, at the end of each epoch.

Here's an example using validation_data:

import numpy as np

data = np.random.random((1000, 32))
labels = np.random.random((1000, 10))

val_data = np.random.random((100, 32))
val_labels = np.random.random((100, 10))

model.fit(data, labels, epochs=10, batch_size=32,
          validation_data=(val_data, val_labels))
Train on 1000 samples, validate on 100 samples
Epoch 1/10
1000/1000 [==============================] - 0s 106us/step - loss: 11.4519 - categorical_accuracy: 0.0850 - val_loss: 11.5329 - val_categorical_accuracy: 0.0800
Epoch 2/10
1000/1000 [==============================] - 0s 65us/step - loss: 11.4491 - categorical_accuracy: 0.0910 - val_loss: 11.5346 - val_categorical_accuracy: 0.1300
Epoch 3/10
1000/1000 [==============================] - 0s 68us/step - loss: 11.4479 - categorical_accuracy: 0.1030 - val_loss: 11.5337 - val_categorical_accuracy: 0.0700
Epoch 4/10
1000/1000 [==============================] - 0s 62us/step - loss: 11.4482 - categorical_accuracy: 0.0910 - val_loss: 11.5347 - val_categorical_accuracy: 0.0800
Epoch 5/10
1000/1000 [==============================] - 0s 63us/step - loss: 11.4473 - categorical_accuracy: 0.1120 - val_loss: 11.5360 - val_categorical_accuracy: 0.1000
Epoch 6/10
1000/1000 [==============================] - 0s 66us/step - loss: 11.4416 - categorical_accuracy: 0.1050 - val_loss: 11.5429 - val_categorical_accuracy: 0.0600
Epoch 7/10
1000/1000 [==============================] - 0s 67us/step - loss: 11.4402 - categorical_accuracy: 0.0950 - val_loss: 11.5387 - val_categorical_accuracy: 0.1000
Epoch 8/10
1000/1000 [==============================] - 0s 65us/step - loss: 11.4372 - categorical_accuracy: 0.0970 - val_loss: 11.5858 - val_categorical_accuracy: 0.0700
Epoch 9/10
1000/1000 [==============================] - 0s 61us/step - loss: 11.4317 - categorical_accuracy: 0.1190 - val_loss: 11.5696 - val_categorical_accuracy: 0.0500
Epoch 10/10
1000/1000 [==============================] - 0s 63us/step - loss: 11.4273 - categorical_accuracy: 0.1240 - val_loss: 11.5771 - val_categorical_accuracy: 0.1700

Input tf.data datasets

Use the Datasets API to scale to large datasets or multi-device training. Pass a tf.data.Dataset instance to the fit method:

# Instantiates a toy dataset instance:
dataset = tf.data.Dataset.from_tensor_slices((data, labels))
dataset = dataset.batch(32)
dataset = dataset.repeat()

# Don't forget to specify `steps_per_epoch` when calling `fit` on a dataset.
model.fit(dataset, epochs=10, steps_per_epoch=30)
Epoch 1/10
30/30 [==============================] - 0s 4ms/step - loss: 11.3898 - categorical_accuracy: 0.1240
Epoch 2/10
30/30 [==============================] - 0s 2ms/step - loss: 11.4143 - categorical_accuracy: 0.1302
Epoch 3/10
30/30 [==============================] - 0s 2ms/step - loss: 11.4311 - categorical_accuracy: 0.1448
Epoch 4/10
30/30 [==============================] - 0s 2ms/step - loss: 11.4180 - categorical_accuracy: 0.1375
Epoch 5/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3933 - categorical_accuracy: 0.1469
Epoch 6/10
30/30 [==============================] - 0s 2ms/step - loss: 11.4155 - categorical_accuracy: 0.1625
Epoch 7/10
30/30 [==============================] - 0s 2ms/step - loss: 11.4089 - categorical_accuracy: 0.1625
Epoch 8/10
30/30 [==============================] - 0s 2ms/step - loss: 11.4212 - categorical_accuracy: 0.1656
Epoch 9/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3842 - categorical_accuracy: 0.1833
Epoch 10/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3476 - categorical_accuracy: 0.1802

Here, the fit method uses the steps_per_epoch argument—this is the number of training steps the model runs before it moves to the next epoch. Since the Dataset yields batches of data, this snippet does not require a batch_size.

Datasets can also be used for validation:

dataset = tf.data.Dataset.from_tensor_slices((data, labels))
dataset = dataset.batch(32).repeat()

val_dataset = tf.data.Dataset.from_tensor_slices((val_data, val_labels))
val_dataset = val_dataset.batch(32).repeat()

model.fit(dataset, epochs=10, steps_per_epoch=30,
          validation_data=val_dataset,
          validation_steps=3)
Epoch 1/10
30/30 [==============================] - 0s 6ms/step - loss: 11.3420 - categorical_accuracy: 0.1781 - val_loss: 11.5709 - val_categorical_accuracy: 0.1042
Epoch 2/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3626 - categorical_accuracy: 0.1990 - val_loss: 11.9732 - val_categorical_accuracy: 0.0312
Epoch 3/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3862 - categorical_accuracy: 0.1927 - val_loss: 11.8544 - val_categorical_accuracy: 0.0625
Epoch 4/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3707 - categorical_accuracy: 0.1885 - val_loss: 12.0138 - val_categorical_accuracy: 0.1042
Epoch 5/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3455 - categorical_accuracy: 0.1906 - val_loss: 11.6441 - val_categorical_accuracy: 0.1042
Epoch 6/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3664 - categorical_accuracy: 0.2010 - val_loss: 12.0033 - val_categorical_accuracy: 0.0312
Epoch 7/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3659 - categorical_accuracy: 0.2021 - val_loss: 11.8532 - val_categorical_accuracy: 0.0729
Epoch 8/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3746 - categorical_accuracy: 0.1906 - val_loss: 12.0326 - val_categorical_accuracy: 0.0417
Epoch 9/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3393 - categorical_accuracy: 0.2042 - val_loss: 11.6516 - val_categorical_accuracy: 0.1146
Epoch 10/10
30/30 [==============================] - 0s 2ms/step - loss: 11.3080 - categorical_accuracy: 0.1906 - val_loss: 11.9671 - val_categorical_accuracy: 0.0417

Evaluate and predict

The tf.keras.Model.evaluate and tf.keras.Model.predict methods can use NumPy data and a tf.data.Dataset.

To evaluate the inference-mode loss and metrics for the data provided:

data = np.random.random((1000, 32))
labels = np.random.random((1000, 10))

model.evaluate(data, labels, batch_size=32)

model.evaluate(dataset, steps=30)
1000/1000 [==============================] - 0s 66us/step
30/30 [==============================] - 0s 3ms/step

[11.31696376800537, 0.209375]

And to predict the output of the last layer in inference for the data provided, as a NumPy array:

result = model.predict(data, batch_size=32)
print(result.shape)
(1000, 10)

Build advanced models

Functional API

The tf.keras.Sequential model is a simple stack of layers that cannot represent arbitrary models. Use the Keras functional API to build complex model topologies such as:

  • Multi-input models,
  • Multi-output models,
  • Models with shared layers (the same layer called several times),
  • Models with non-sequential data flows (e.g. residual connections).

Building a model with the functional API works like this:

  1. A layer instance is callable and returns a tensor.
  2. Input tensors and output tensors are used to define a tf.keras.Model instance.
  3. This model is trained just like the Sequential model.

The following example uses the functional API to build a simple, fully-connected network:

inputs = tf.keras.Input(shape=(32,))  # Returns a placeholder tensor

# A layer instance is callable on a tensor, and returns a tensor.
x = layers.Dense(64, activation='relu')(inputs)
x = layers.Dense(64, activation='relu')(x)
predictions = layers.Dense(10, activation='softmax')(x)

Instantiate the model given inputs and outputs.

model = tf.keras.Model(inputs=inputs, outputs=predictions)

# The compile step specifies the training configuration.
model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Trains for 5 epochs
model.fit(data, labels, batch_size=32, epochs=5)
Epoch 1/5
1000/1000 [==============================] - 0s 227us/step - loss: 11.5461 - acc: 0.1100
Epoch 2/5
1000/1000 [==============================] - 0s 61us/step - loss: 11.4875 - acc: 0.1090
Epoch 3/5
1000/1000 [==============================] - 0s 61us/step - loss: 11.4698 - acc: 0.1150
Epoch 4/5
1000/1000 [==============================] - 0s 69us/step - loss: 11.4593 - acc: 0.1140
Epoch 5/5
1000/1000 [==============================] - 0s 66us/step - loss: 11.4512 - acc: 0.1090

Model subclassing

Build a fully-customizable model by subclassing tf.keras.Model and defining your own forward pass. Create layers in the __init__ method and set them as attributes of the class instance. Define the forward pass in the call method.

Model subclassing is particularly useful when eager execution is enabled since the forward pass can be written imperatively.

The following example shows a subclassed tf.keras.Model using a custom forward pass:

class MyModel(tf.keras.Model):

  def __init__(self, num_classes=10):
    super(MyModel, self).__init__(name='my_model')
    self.num_classes = num_classes
    # Define your layers here.
    self.dense_1 = layers.Dense(32, activation='relu')
    self.dense_2 = layers.Dense(num_classes, activation='sigmoid')

  def call(self, inputs):
    # Define your forward pass here,
    # using layers you previously defined (in `__init__`).
    x = self.dense_1(inputs)
    return self.dense_2(x)

  def compute_output_shape(self, input_shape):
    # You need to override this function if you want to use the subclassed model
    # as part of a functional-style model.
    # Otherwise, this method is optional.
    shape = tf.TensorShape(input_shape).as_list()
    shape[-1] = self.num_classes
    return tf.TensorShape(shape)

Instantiate the new model class:

model = MyModel(num_classes=10)

# The compile step specifies the training configuration.
model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Trains for 5 epochs.
model.fit(data, labels, batch_size=32, epochs=5)
Epoch 1/5
1000/1000 [==============================] - 0s 195us/step - loss: 11.6099 - acc: 0.1000
Epoch 2/5
1000/1000 [==============================] - 0s 49us/step - loss: 11.5495 - acc: 0.0960
Epoch 3/5
1000/1000 [==============================] - 0s 47us/step - loss: 11.4888 - acc: 0.1000
Epoch 4/5
1000/1000 [==============================] - 0s 52us/step - loss: 11.4714 - acc: 0.0930
Epoch 5/5
1000/1000 [==============================] - 0s 44us/step - loss: 11.4649 - acc: 0.0940

Custom layers

Create a custom layer by subclassing tf.keras.layers.Layer and implementing the following methods:

  • build: Create the weights of the layer. Add weights with the add_weight method.
  • call: Define the forward pass.
  • compute_output_shape: Specify how to compute the output shape of the layer given the input shape.
  • Optionally, a layer can be serialized by implementing the get_config method and the from_config class method.

Here's an example of a custom layer that implements a matmul of an input with a kernel matrix:

class MyLayer(layers.Layer):

  def __init__(self, output_dim, **kwargs):
    self.output_dim = output_dim
    super(MyLayer, self).__init__(**kwargs)

  def build(self, input_shape):
    shape = tf.TensorShape((input_shape[1], self.output_dim))
    # Create a trainable weight variable for this layer.
    self.kernel = self.add_weight(name='kernel',
                                  shape=shape,
                                  initializer='uniform',
                                  trainable=True)
    # Be sure to call this at the end
    super(MyLayer, self).build(input_shape)

  def call(self, inputs):
    return tf.matmul(inputs, self.kernel)

  def compute_output_shape(self, input_shape):
    shape = tf.TensorShape(input_shape).as_list()
    shape[-1] = self.output_dim
    return tf.TensorShape(shape)

  def get_config(self):
    base_config = super(MyLayer, self).get_config()
    base_config['output_dim'] = self.output_dim
    return base_config

  @classmethod
  def from_config(cls, config):
    return cls(**config)

Create a model using your custom layer:

model = tf.keras.Sequential([
    MyLayer(10),
    layers.Activation('softmax')])

# The compile step specifies the training configuration
model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Trains for 5 epochs.
model.fit(data, labels, batch_size=32, epochs=5)
Epoch 1/5
1000/1000 [==============================] - 0s 189us/step - loss: 11.4600 - acc: 0.0940
Epoch 2/5
1000/1000 [==============================] - 0s 51us/step - loss: 11.4556 - acc: 0.0830
Epoch 3/5
1000/1000 [==============================] - 0s 51us/step - loss: 11.4534 - acc: 0.1040
Epoch 4/5
1000/1000 [==============================] - 0s 59us/step - loss: 11.4511 - acc: 0.1090
Epoch 5/5
1000/1000 [==============================] - 0s 56us/step - loss: 11.4494 - acc: 0.1010

Callbacks

A callback is an object passed to a model to customize and extend its behavior during training. You can write your own custom callback, or use the built-in tf.keras.callbacks that include:

To use a tf.keras.callbacks.Callback, pass it to the model's fit method:

callbacks = [
  # Interrupt training if `val_loss` stops improving for over 2 epochs
  tf.keras.callbacks.EarlyStopping(patience=2, monitor='val_loss'),
  # Write TensorBoard logs to `./logs` directory
  tf.keras.callbacks.TensorBoard(log_dir='./logs')
]
model.fit(data, labels, batch_size=32, epochs=5, callbacks=callbacks,
          validation_data=(val_data, val_labels))
Train on 1000 samples, validate on 100 samples
Epoch 1/5
1000/1000 [==============================] - 0s 152us/step - loss: 11.4467 - acc: 0.1080 - val_loss: 11.5382 - val_acc: 0.1200
Epoch 2/5
1000/1000 [==============================] - 0s 80us/step - loss: 11.4459 - acc: 0.1130 - val_loss: 11.5373 - val_acc: 0.0900
Epoch 3/5
1000/1000 [==============================] - 0s 83us/step - loss: 11.4430 - acc: 0.1170 - val_loss: 11.5401 - val_acc: 0.1000
Epoch 4/5
1000/1000 [==============================] - 0s 85us/step - loss: 11.4416 - acc: 0.1310 - val_loss: 11.5363 - val_acc: 0.1000
Epoch 5/5
1000/1000 [==============================] - 0s 80us/step - loss: 11.4403 - acc: 0.1340 - val_loss: 11.5383 - val_acc: 0.0900

Save and restore

Weights only

Save and load the weights of a model using tf.keras.Model.save_weights:

model = tf.keras.Sequential([
layers.Dense(64, activation='relu'),
layers.Dense(10, activation='softmax')])

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
# Save weights to a TensorFlow Checkpoint file
model.save_weights('./weights/my_model')

# Restore the model's state,
# this requires a model with the same architecture.
model.load_weights('./weights/my_model')

By default, this saves the model's weights in the TensorFlow checkpoint file format. Weights can also be saved to the Keras HDF5 format (the default for the multi-backend implementation of Keras):

# Save weights to a HDF5 file
model.save_weights('my_model.h5', save_format='h5')

# Restore the model's state
model.load_weights('my_model.h5')

Configuration only

A model's configuration can be saved—this serializes the model architecture without any weights. A saved configuration can recreate and initialize the same model, even without the code that defined the original model. Keras supports JSON and YAML serialization formats:

# Serialize a model to JSON format
json_string = model.to_json()
json_string
'{"keras_version": "2.1.6-tf", "class_name": "Sequential", "config": {"name": "sequential_3", "layers": [{"class_name": "Dense", "config": {"name": "dense_17", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "dtype": null, "units": 64, "bias_initializer": {"class_name": "Zeros", "config": {"dtype": "float32"}}, "trainable": true, "activation": "relu", "kernel_regularizer": null, "activity_regularizer": null, "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null, "dtype": "float32"}}}}, {"class_name": "Dense", "config": {"name": "dense_18", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "dtype": null, "units": 10, "bias_initializer": {"class_name": "Zeros", "config": {"dtype": "float32"}}, "trainable": true, "activation": "softmax", "kernel_regularizer": null, "activity_regularizer": null, "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null, "dtype": "float32"}}}}]}, "backend": "tensorflow"}'
import json
import pprint
pprint.pprint(json.loads(json_string))
{'backend': 'tensorflow',
 'class_name': 'Sequential',
 'config': {'layers': [{'class_name': 'Dense',
                        'config': {'activation': 'relu',
                                   'activity_regularizer': None,
                                   'bias_constraint': None,
                                   'bias_initializer': {'class_name': 'Zeros',
                                                        'config': {'dtype': 'float32'}},
                                   'bias_regularizer': None,
                                   'dtype': None,
                                   'kernel_constraint': None,
                                   'kernel_initializer': {'class_name': 'GlorotUniform',
                                                          'config': {'dtype': 'float32',
                                                                     'seed': None}},
                                   'kernel_regularizer': None,
                                   'name': 'dense_17',
                                   'trainable': True,
                                   'units': 64,
                                   'use_bias': True}},
                       {'class_name': 'Dense',
                        'config': {'activation': 'softmax',
                                   'activity_regularizer': None,
                                   'bias_constraint': None,
                                   'bias_initializer': {'class_name': 'Zeros',
                                                        'config': {'dtype': 'float32'}},
                                   'bias_regularizer': None,
                                   'dtype': None,
                                   'kernel_constraint': None,
                                   'kernel_initializer': {'class_name': 'GlorotUniform',
                                                          'config': {'dtype': 'float32',
                                                                     'seed': None}},
                                   'kernel_regularizer': None,
                                   'name': 'dense_18',
                                   'trainable': True,
                                   'units': 10,
                                   'use_bias': True}}],
            'name': 'sequential_3'},
 'keras_version': '2.1.6-tf'}

Recreate the model (freshly initialized), from the json.

fresh_model = tf.keras.models.model_from_json(json_string)

Serializing a model to YAML format requires that you install pyyaml before you import TensorFlow:

yaml_string = model.to_yaml()
print(yaml_string)
backend: tensorflow
class_name: Sequential
config:
  layers:
  - class_name: Dense
    config:
      activation: relu
      activity_regularizer: null
      bias_constraint: null
      bias_initializer:
        class_name: Zeros
        config: {dtype: float32}
      bias_regularizer: null
      dtype: null
      kernel_constraint: null
      kernel_initializer:
        class_name: GlorotUniform
        config: {dtype: float32, seed: null}
      kernel_regularizer: null
      name: dense_17
      trainable: true
      units: 64
      use_bias: true
  - class_name: Dense
    config:
      activation: softmax
      activity_regularizer: null
      bias_constraint: null
      bias_initializer:
        class_name: Zeros
        config: {dtype: float32}
      bias_regularizer: null
      dtype: null
      kernel_constraint: null
      kernel_initializer:
        class_name: GlorotUniform
        config: {dtype: float32, seed: null}
      kernel_regularizer: null
      name: dense_18
      trainable: true
      units: 10
      use_bias: true
  name: sequential_3
keras_version: 2.1.6-tf

Recreate the model from the yaml


fresh_model = tf.keras.models.model_from_yaml(yaml_string)

Entire model

The entire model can be saved to a file that contains the weight values, the model's configuration, and even the optimizer's configuration. This allows you to checkpoint a model and resume training later—from the exact same state—without access to the original code.

# Create a trivial model
model = tf.keras.Sequential([
  layers.Dense(10, activation='softmax', input_shape=(32,)),
  layers.Dense(10, activation='softmax')
])
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.fit(data, labels, batch_size=32, epochs=5)


# Save entire model to a HDF5 file
model.save('my_model.h5')

# Recreate the exact same model, including weights and optimizer.
model = tf.keras.models.load_model('my_model.h5')
Epoch 1/5
1000/1000 [==============================] - 0s 258us/step - loss: 11.4740 - acc: 0.0830
Epoch 2/5
1000/1000 [==============================] - 0s 56us/step - loss: 11.4573 - acc: 0.0850
Epoch 3/5
1000/1000 [==============================] - 0s 64us/step - loss: 11.4524 - acc: 0.1030
Epoch 4/5
1000/1000 [==============================] - 0s 59us/step - loss: 11.4501 - acc: 0.1100
Epoch 5/5
1000/1000 [==============================] - 0s 67us/step - loss: 11.4495 - acc: 0.1020

Eager execution

Eager execution is an imperative programming environment that evaluates operations immediately. This is not required for Keras, but is supported by tf.keras and useful for inspecting your program and debugging.

All of the tf.keras model-building APIs are compatible with eager execution. And while the Sequential and functional APIs can be used, eager execution especially benefits model subclassing and building custom layers—the APIs that require you to write the forward pass as code (instead of the APIs that create models by assembling existing layers).

See the eager execution guide for examples of using Keras models with custom training loops and tf.GradientTape.

Distribution

Estimators

The Estimators API is used for training models for distributed environments. This targets industry use cases such as distributed training on large datasets that can export a model for production.

A tf.keras.Model can be trained with the tf.estimator API by converting the model to an tf.estimator.Estimator object with tf.keras.estimator.model_to_estimator. See Creating Estimators from Keras models.

model = tf.keras.Sequential([layers.Dense(10,activation='softmax'),
                          layers.Dense(10,activation='softmax')])

model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

estimator = tf.keras.estimator.model_to_estimator(model)
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpjbzrzzrl
INFO:tensorflow:Using the Keras model provided.
INFO:tensorflow:Using config: {'_train_distribute': None, '_master': '', '_evaluation_master': '', '_log_step_count_steps': 100, '_protocol': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7facfc300b00>, '_task_type': 'worker', '_keep_checkpoint_every_n_hours': 10000, '_device_fn': None, '_tf_random_seed': None, '_save_checkpoints_steps': None, '_global_id_in_cluster': 0, '_keep_checkpoint_max': 5, '_is_chief': True, '_num_worker_replicas': 1, '_model_dir': '/tmp/tmpjbzrzzrl', '_experimental_distribute': None, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_eval_distribute': None}

Multiple GPUs

tf.keras models can run on multiple GPUs using tf.contrib.distribute.DistributionStrategy. This API provides distributed training on multiple GPUs with almost no changes to existing code.

Currently, tf.contrib.distribute.MirroredStrategy is the only supported distribution strategy. MirroredStrategy does in-graph replication with synchronous training using all-reduce on a single machine. To use DistributionStrategy with Keras, convert the tf.keras.Model to a tf.estimator.Estimator with tf.keras.estimator.model_to_estimator, then train the estimator

The following example distributes a tf.keras.Model across multiple GPUs on a single machine.

First, define a simple model:

model = tf.keras.Sequential()
model.add(layers.Dense(16, activation='relu', input_shape=(10,)))
model.add(layers.Dense(1, activation='sigmoid'))

optimizer = tf.train.GradientDescentOptimizer(0.2)

model.compile(loss='binary_crossentropy', optimizer=optimizer)
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_23 (Dense)             (None, 16)                176       
_________________________________________________________________
dense_24 (Dense)             (None, 1)                 17        
=================================================================
Total params: 193
Trainable params: 193
Non-trainable params: 0
_________________________________________________________________

Define an input pipeline. The input_fn returns a tf.data.Dataset object used to distribute the data across multiple devices—with each device processing a slice of the input batch.

def input_fn():
  x = np.random.random((1024, 10))
  y = np.random.randint(2, size=(1024, 1))
  x = tf.cast(x, tf.float32)
  dataset = tf.data.Dataset.from_tensor_slices((x, y))
  dataset = dataset.repeat(10)
  dataset = dataset.batch(32)
  return dataset

Next, create a tf.estimator.RunConfig and set the train_distribute argument to the tf.contrib.distribute.MirroredStrategy instance. When creating MirroredStrategy, you can specify a list of devices or set the num_gpus argument. The default uses all available GPUs, like the following:

strategy = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(train_distribute=strategy)
INFO:tensorflow:Initializing RunConfig with distribution strategies.
INFO:tensorflow:Not using Distribute Coordinator.

Convert the Keras model to a tf.estimator.Estimator instance:

keras_estimator = tf.keras.estimator.model_to_estimator(
  keras_model=model,
  config=config,
  model_dir='/tmp/model_dir')
INFO:tensorflow:Using the Keras model provided.
INFO:tensorflow:Using config: {'_train_distribute': <tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7fad2a97ac50>, '_master': '', '_evaluation_master': '', '_log_step_count_steps': 100, '_protocol': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fad2a97acf8>, '_task_type': 'worker', '_keep_checkpoint_every_n_hours': 10000, '_device_fn': None, '_tf_random_seed': None, '_save_checkpoints_steps': None, '_global_id_in_cluster': 0, '_keep_checkpoint_max': 5, '_is_chief': True, '_num_worker_replicas': 1, '_model_dir': '/tmp/model_dir', '_experimental_distribute': None, '_distribute_coordinator_mode': None, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_eval_distribute': None}

Finally, train the Estimator instance by providing the input_fn and steps arguments:

keras_estimator.train(input_fn=input_fn, steps=10)
INFO:tensorflow:Device is available but not used by distribute strategy: /device:XLA_CPU:0
WARNING:tensorflow:Not all devices in DistributionStrategy are visible to TensorFlow session.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Warm-starting with WarmStartSettings: WarmStartSettings(ckpt_to_initialize_from='/tmp/model_dir/keras/keras_model.ckpt', vars_to_warm_start='.*', var_name_to_vocab_info={}, var_name_to_prev_var_name={})
INFO:tensorflow:Warm-starting from: ('/tmp/model_dir/keras/keras_model.ckpt',)
INFO:tensorflow:Warm-starting variable: dense_24/bias; prev_var_name: Unchanged
INFO:tensorflow:Warm-starting variable: dense_24/kernel; prev_var_name: Unchanged
INFO:tensorflow:Warm-starting variable: dense_23/kernel; prev_var_name: Unchanged
INFO:tensorflow:Warm-starting variable: dense_23/bias; prev_var_name: Unchanged
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/model_dir/model.ckpt.
INFO:tensorflow:loss = 0.69151586, step = 0
INFO:tensorflow:Saving checkpoints for 10 into /tmp/model_dir/model.ckpt.
INFO:tensorflow:Finalize system.
INFO:tensorflow:Loss for final step: 0.700209.