tf.train.NanTensorHook

Class NanTensorHook

Inherits From: SessionRunHook

Defined in tensorflow/python/training/basic_session_run_hooks.py.

Monitors the loss tensor and stops training if loss is NaN.

Can either fail with exception or just stop training.

__init__

__init__(
    loss_tensor,
    fail_on_nan_loss=True
)

Initializes a NanTensorHook.

Args:

  • loss_tensor: Tensor, the loss tensor.
  • fail_on_nan_loss: bool, whether to raise exception when loss is NaN.

Methods

after_create_session

after_create_session(
    session,
    coord
)

Called when new TensorFlow session is created.

This is called to signal the hooks that a new session has been created. This has two essential differences with the situation in which begin is called:

  • When this is called, the graph is finalized and ops can no longer be added to the graph.
  • This method will also be called as a result of recovering a wrapped session, not only at the beginning of the overall session.

Args:

  • session: A TensorFlow Session that has been created.
  • coord: A Coordinator object which keeps track of all threads.

after_run

after_run(
    run_context,
    run_values
)

before_run

before_run(run_context)

begin

begin()

Called once before using the session.

When called, the default graph is the one that will be launched in the session. The hook can modify the graph by adding new operations to it. After the begin() call the graph will be finalized and the other callbacks can not modify the graph anymore. Second call of begin() on the same graph, should not change the graph.

end

end(session)

Called at the end of session.

The session argument can be used in case the hook wants to run final ops, such as saving a last checkpoint.

If session.run() raises exception other than OutOfRangeError or StopIteration then end() is not called. Note the difference between end() and after_run() behavior when session.run() raises OutOfRangeError or StopIteration. In that case end() is called but after_run() is not called.

Args:

  • session: A TensorFlow Session that will be soon closed.