# tf.keras.callbacks.experimental.BackupAndRestore

<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/keras-team/keras/tree/v2.7.0/keras/callbacks.py#L1586-L1706) |

Callback to back up and restore the training state.

Inherits From: [`Callback`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/keras/callbacks/Callback)

    tf.keras.callbacks.experimental.BackupAndRestore(
        backup_dir
    )

`BackupAndRestore` callback is intended to recover training from an
interruption that has happened in the middle of a [`Model.fit`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/keras/Model#fit) execution, by
backing up the training states in a temporary checkpoint file (with the help
of a [`tf.train.CheckpointManager`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/train/CheckpointManager)), at the end of each epoch. Each backup
overwrites the previously written checkpoint file, so at any given time there
is at most one such checkpoint file for backup/restoring purpose.

If training restarts before completion, the training state (which includes the
`Model` weights and epoch number) is restored to the most recently saved state
at the beginning of a new [`Model.fit`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/keras/Model#fit) run. At the completion of a [`Model.fit`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/keras/Model#fit)
run, the temporary checkpoint file is deleted.

Note that the user is responsible to bring jobs back after the interruption.
This callback is important for the backup and restore mechanism for fault
tolerance purpose, and the model to be restored from an previous checkpoint is
expected to be the same as the one used to back up. If user changes arguments
passed to compile or fit, the checkpoint saved for fault tolerance can become
invalid.

#### Note:

1. This callback is not compatible with eager execution disabled.
2. A checkpoint is saved at the end of each epoch. After restoring, [`Model.fit`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/keras/Model#fit) redoes any partial work during the unfinished epoch in which the training got restarted (so the work done before the interruption doesn't affect the final model state).
3. This works for both single worker and multi-worker modes. When [`Model.fit`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/keras/Model#fit) is used with [`tf.distribute`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/distribute), it supports [`tf.distribute.MirroredStrategy`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/distribute/MirroredStrategy), [`tf.distribute.MultiWorkerMirroredStrategy`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/distribute/MultiWorkerMirroredStrategy), [`tf.distribute.TPUStrategy`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/distribute/TPUStrategy), and [`tf.distribute.experimental.ParameterServerStrategy`](https://www.tensorflow.org/versions/r2.7/api_docs/python/tf/distribute/experimental/ParameterServerStrategy).

#### Example:

    class InterruptingCallback(tf.keras.callbacks.Callback):
      def on_epoch_begin(self, epoch, logs=None):
        if epoch == 4:
          raise RuntimeError('Interrupting!')
    callback = tf.keras.callbacks.experimental.BackupAndRestore(
    backup_dir="/tmp/backup")
    model = tf.keras.models.Sequential([tf.keras.layers.Dense(10)])
    model.compile(tf.keras.optimizers.SGD(), loss='mse')
    try:
      model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10,
                batch_size=1, callbacks=[callback, InterruptingCallback()],
                verbose=0)
    except:
      pass
    history = model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10,
                batch_size=1, callbacks=[callback], verbose=0)
    # Only 6 more epochs are run, since first trainning got interrupted at
    # zero-indexed epoch 4, second training will continue from 4 to 9.
    len(history.history['loss'])
    6

<br />

<br />

<br />

| ## Args ||
|---|---|
| `backup_dir` | String, path to store the checkpoint. e.g. backup_dir = os.path.join(working_dir, 'backup') This is the directory in which the system stores temporary files to recover the model from jobs terminated unexpectedly. The directory cannot be reused elsewhere to store other files, e.g. by BackupAndRestore callback of another training, or by another callback (ModelCheckpoint) of the same training. |

<br />

## Methods

### `set_model`

[View source](https://github.com/keras-team/keras/tree/v2.7.0/keras/callbacks.py#L645-L646)

    set_model(
        model
    )

### `set_params`

[View source](https://github.com/keras-team/keras/tree/v2.7.0/keras/callbacks.py#L642-L643)

    set_params(
        params
    )