Classes

A mutable, shareable, owning reference to a tensor.

Class wrapping a C pointer to a TensorHandle. This class owns the TensorHandle and is responsible for destroying it.

RMSProp optimizer.

It is recommended to leave the parameters of this optimizer at their default values (except for the learning rate, which can be freely tuned). This optimizer is usually a good choice for recurrent neural networks.

Reference: “rmsprop: Divide the gradient by a running average of its recent magnitude”

AdaGrad optimizer.

Individually adapts the learning rates of all model parameters by scaling them inversely proportional to the square root of the sum of all the historical squared values of the gradient.

Reference: “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization”

ADADELTA optimizer.

ADADELTA is a more robust extension of AdaGrad. ADADELTA adapts learning rates based on a moving window of gradient updates rather than by accumulating all past gradient norms. It can thus adapt faster to changing dynamics of the optimization problem space.

Reference: “ADADELTA: An Adaptive Learning Rate Method”

Adam optimizer.

Implements the Adam optimization algorithm. Adam is a stochastic gradient descent method that computes individual adaptive learning rates for different parameters from estimates of first- and second-order moments of the gradients.

Reference: “Adam: A Method for Stochastic Optimization” (Kingma and Ba, 2014).

Examples:

• Train a simple reinforcement learning agent:

...
// Instantiate an agent's policy - approximated by the neural network (`net`) after defining it 
in advance.
var net = Net(observationSize: Int(observationSize), hiddenSize: hiddenSize, actionCount: actionCount)
// Define the Adam optimizer for the network with a learning rate set to 0.01.
let optimizer = Adam(for: net, learningRate: 0.01)
...
// Begin training the agent (over a certain number of episodes).
while true {
...
    // Implementing the gradient descent with the Adam optimizer:
    // Define the gradients (use withLearningPhase to call a closure under a learning phase).
    let gradients = withLearningPhase(.training) {
        TensorFlow.gradient(at: net) { net -> Tensor<Float> in
            // Return a softmax (loss) function
            return loss = softmaxCrossEntropy(logits: net(input), probabilities: target)
        }
    }
    // Update the differentiable variables of the network (`net`) along the gradients with the Adam 
optimizer.
    optimizer.update(&net, along: gradients)
    ...
    }
}

• Train a generative adversarial network (GAN):

...
// Instantiate the generator and the discriminator networks after defining them.
var generator = Generator()
var discriminator = Discriminator()
// Define the Adam optimizers for each network with a learning rate set to 2e-4 and beta1 - to 0.5.
let adamOptimizerG = Adam(for: generator, learningRate: 2e-4, beta1: 0.5)
let adamOptimizerD = Adam(for: discriminator, learningRate: 2e-4, beta1: 0.5)
...
Start the training loop over a certain number of epochs (`epochCount`).
for epoch in 1...epochCount {
    // Start the training phase.
    ...
    for batch in trainingShuffled.batched(batchSize) {
        // Implementing the gradient descent with the Adam optimizer:
        // 1) Update the generator.
        ...
        let 𝛁generator = TensorFlow.gradient(at: generator) { generator -> Tensor<Float> in
            ...
            return loss
            }
        // Update the differentiable variables of the generator along the gradients (`𝛁generator`) 
        // with the Adam optimizer.
        adamOptimizerG.update(&generator, along: 𝛁generator)

        // 2) Update the discriminator.
        ...
        let 𝛁discriminator = TensorFlow.gradient(at: discriminator) { discriminator -> Tensor<Float> in
            ...
            return loss
        }
        // Update the differentiable variables of the discriminator along the gradients (`𝛁discriminator`) 
        // with the Adam optimizer.
        adamOptimizerD.update(&discriminator, along: 𝛁discriminator)
        }
}       

AdaMax optimizer.

A variant of Adam based on the infinity-norm.

Reference: Section 7 of “Adam - A Method for Stochastic Optimization”

AMSGrad optimizer.

This algorithm is a modification of Adam with better convergence properties when close to local optima.

Reference: “On the Convergence of Adam and Beyond”

RAdam optimizer.

Rectified Adam, a variant of Adam that introduces a term to rectify the adaptive learning rate variance.

Reference: [“On the Variance of the Adaptive Learning Rate and Beyond”] https://arxiv.org/pdf/1908.03265.pdf

Stochastic gradient descent (SGD) optimizer.

An optimizer that implements stochastic gradient descent, with support for momentum, learning rate decay, and Nesterov momentum.

A TensorFlow checkpoint file reader.