<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/backup_and_restore.py#L8-L210) |

Callback to back up and restore the training state.

Inherits From: [`Callback`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback)

    tf.keras.callbacks.BackupAndRestore(
        backup_dir,
        save_freq='epoch',
        double_checkpoint=False,
        delete_checkpoint=True
    )

### Used in the notebooks

| Used in the guide | Used in the tutorials |
|---|---|
| - [Migrate the fault tolerance mechanism](https://www.tensorflow.org/guide/migrate/fault_tolerance) | - [Multi-worker training with Keras](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras) - [Parameter server training with ParameterServerStrategy](https://www.tensorflow.org/tutorials/distribute/parameter_server_training) |

`BackupAndRestore` callback is intended to recover training from an
interruption that has happened in the middle of a [`Model.fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) execution, by
backing up the training states in a temporary checkpoint file, at the end of
each epoch. Each backup overwrites the previously written checkpoint file,
so at any given time there is at most one such checkpoint file for
backup/restoring purpose.

If training restarts before completion, the training state (which includes
the `Model` weights and epoch number) is restored to the most recently saved
state at the beginning of a new [`Model.fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) run. At the completion of a
[`Model.fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) run, the temporary checkpoint file is deleted.

Note that the user is responsible to bring jobs back after the interruption.
This callback is important for the backup and restore mechanism for fault
tolerance purpose, and the model to be restored from a previous checkpoint
is expected to be the same as the one used to back up. If user changes
arguments passed to compile or fit, the checkpoint saved for fault tolerance
can become invalid.

#### Example:

    class InterruptingCallback(keras.callbacks.Callback):
      def on_epoch_begin(self, epoch, logs=None):
        if epoch == 4:
          raise RuntimeError('Interrupting!')
    callback = keras.callbacks.BackupAndRestore(backup_dir="/tmp/backup")
    model = keras.models.Sequential([keras.layers.Dense(10)])
    model.compile(keras.optimizers.SGD(), loss='mse')
    model.build(input_shape=(None, 20))
    try:
      model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10,
                batch_size=1, callbacks=[callback, InterruptingCallback()],
                verbose=0)
    except:
      pass
    history = model.fit(np.arange(100).reshape(5, 20), np.zeros(5),
                        epochs=10, batch_size=1, callbacks=[callback],
                        verbose=0)
    # Only 6 more epochs are run, since first training got interrupted at
    # zero-indexed epoch 4, second training will continue from 4 to 9.
    len(history.history['loss'])
    6

<br />

<br />

<br />

| ## Args ||
|---|---|
| `backup_dir` | String, path of directory where to store the data needed to restore the model. The directory cannot be reused elsewhere to store other files, e.g. by the `BackupAndRestore` callback of another training run, or by another callback (e.g. `ModelCheckpoint`) of the same training run. |
| `save_freq` | `"epoch"`, integer, or `False`. When set to `"epoch"` the callback saves the checkpoint at the end of each epoch. When set to an integer, the callback saves the checkpoint every `save_freq` batches. Set `save_freq=False` only if using preemption checkpointing (i.e. with `save_before_preemption=True`). |
| `double_checkpoint` | Boolean. If enabled, `BackupAndRestore` callback will save 2 last training states (current and previous). After interruption if current state can't be loaded due to IO error (e.g. file corrupted) it will try to restore previous one. Such behaviour will consume twice more space on disk, but increase fault tolerance. Defaults to `False`. |
| `delete_checkpoint` | Boolean. This `BackupAndRestore` callback works by saving a checkpoint to back up the training state. If `delete_checkpoint=True`, the checkpoint will be deleted after training is finished. Use `False` if you'd like to keep the checkpoint for future usage. Defaults to `True`. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `model` | <br /> <br /> |

<br />

## Methods

### `on_batch_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L102-L104)

    on_batch_begin(
        batch, logs=None
    )

A backwards compatibility alias for `on_train_batch_begin`.

### `on_batch_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L106-L108)

    on_batch_end(
        batch, logs=None
    )

A backwards compatibility alias for `on_train_batch_end`.

### `on_epoch_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L110-L121)

    on_epoch_begin(
        epoch, logs=None
    )

Called at the start of an epoch.

Subclasses should override for any actions to run. This function should
only be called during TRAIN mode.

<br />

<br />

<br />

| Args ||
|---|---|
| `epoch` | Integer, index of epoch. |
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_epoch_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/backup_and_restore.py#L154-L158)

    on_epoch_end(
        epoch, logs=None
    )

Called at the end of an epoch.

Subclasses should override for any actions to run. This function should
only be called during TRAIN mode.

<br />

<br />

<br />

| Args ||
|---|---|
| `epoch` | Integer, index of epoch. |
| `logs` | Dict, metric results for this training epoch, and for the validation epoch if validation is performed. Validation result keys are prefixed with `val_`. For training epoch, the values of the `Model`'s metrics are returned. Example: `{'loss': 0.2, 'accuracy': 0.7}`. |

<br />

### `on_predict_batch_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L211-L225)

    on_predict_batch_begin(
        batch, logs=None
    )

Called at the beginning of a batch in `predict` methods.

Subclasses should override for any actions to run.

Note that if the `steps_per_execution` argument to `compile` in
`Model` is set to `N`, this method will only be called every
`N` batches.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch` | Integer, index of batch within the current epoch. |
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_predict_batch_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L227-L240)

    on_predict_batch_end(
        batch, logs=None
    )

Called at the end of a batch in `predict` methods.

Subclasses should override for any actions to run.

Note that if the `steps_per_execution` argument to `compile` in
`Model` is set to `N`, this method will only be called every
`N` batches.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch` | Integer, index of batch within the current epoch. |
| `logs` | Dict. Aggregated metric results up until this batch. |

<br />

### `on_predict_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L288-L297)

    on_predict_begin(
        logs=None
    )

Called at the beginning of prediction.

Subclasses should override for any actions to run.

<br />

<br />

<br />

| Args ||
|---|---|
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_predict_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L299-L308)

    on_predict_end(
        logs=None
    )

Called at the end of prediction.

Subclasses should override for any actions to run.

<br />

<br />

<br />

| Args ||
|---|---|
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_test_batch_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L174-L191)

    on_test_batch_begin(
        batch, logs=None
    )

Called at the beginning of a batch in `evaluate` methods.

Also called at the beginning of a validation batch in the `fit`
methods, if validation data is provided.

Subclasses should override for any actions to run.

Note that if the `steps_per_execution` argument to `compile` in
`Model` is set to `N`, this method will only be called every
`N` batches.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch` | Integer, index of batch within the current epoch. |
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_test_batch_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L193-L209)

    on_test_batch_end(
        batch, logs=None
    )

Called at the end of a batch in `evaluate` methods.

Also called at the end of a validation batch in the `fit`
methods, if validation data is provided.

Subclasses should override for any actions to run.

Note that if the `steps_per_execution` argument to `compile` in
`Model` is set to `N`, this method will only be called every
`N` batches.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch` | Integer, index of batch within the current epoch. |
| `logs` | Dict. Aggregated metric results up until this batch. |

<br />

### `on_test_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L265-L274)

    on_test_begin(
        logs=None
    )

Called at the beginning of evaluation or validation.

Subclasses should override for any actions to run.

<br />

<br />

<br />

| Args ||
|---|---|
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_test_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L276-L286)

    on_test_end(
        logs=None
    )

Called at the end of evaluation or validation.

Subclasses should override for any actions to run.

<br />

<br />

<br />

| Args ||
|---|---|
| `logs` | Dict. Currently the output of the last call to `on_test_batch_end()` is passed to this argument for this method but that may change in the future. |

<br />

### `on_train_batch_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L139-L155)

    on_train_batch_begin(
        batch, logs=None
    )

Called at the beginning of a training batch in `fit` methods.

Subclasses should override for any actions to run.

Note that if the `steps_per_execution` argument to `compile` in
`Model` is set to `N`, this method will only be called every
`N` batches.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch` | Integer, index of batch within the current epoch. |
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_train_batch_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/backup_and_restore.py#L160-L162)

    on_train_batch_end(
        batch, logs=None
    )

Called at the end of a training batch in `fit` methods.

Subclasses should override for any actions to run.

Note that if the `steps_per_execution` argument to `compile` in
`Model` is set to `N`, this method will only be called every
`N` batches.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch` | Integer, index of batch within the current epoch. |
| `logs` | Dict. Aggregated metric results up until this batch. |

<br />

### `on_train_begin`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/backup_and_restore.py#L113-L128)

    on_train_begin(
        logs=None
    )

Called at the beginning of training.

Subclasses should override for any actions to run.

<br />

<br />

<br />

| Args ||
|---|---|
| `logs` | Dict. Currently no data is passed to this argument for this method but that may change in the future. |

<br />

### `on_train_end`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/backup_and_restore.py#L208-L210)

    on_train_end(
        logs=None
    )

Called at the end of training.

Subclasses should override for any actions to run.

<br />

<br />

<br />

| Args ||
|---|---|
| `logs` | Dict. Currently the output of the last call to `on_epoch_end()` is passed to this argument for this method but that may change in the future. |

<br />

### `set_model`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L74-L75)

    set_model(
        model
    )

### `set_params`

[View source](https://github.com/keras-team/keras/tree/v3.15.1/keras/src/callbacks/callback.py#L71-L72)

    set_params(
        params
    )