``````public class Adam<Model: Differentiable>: Optimizer
where
Model.TangentVector: VectorProtocol & PointwiseMultiplicative
& ElementaryFunctions & KeyPathIterable,
Model.TangentVector.VectorSpaceScalar == Float``````

Implements the Adam optimization algorithm. Adam is a stochastic gradient descent method that computes individual adaptive learning rates for different parameters from estimates of first- and second-order moments of the gradients.

Reference: “Adam: A Method for Stochastic Optimization” (Kingma and Ba, 2014).

### Examples:

• Train a simple reinforcement learning agent:
``````...
// Instantiate an agent's policy - approximated by the neural network (`net`) after defining it
var net = Net(observationSize: Int(observationSize), hiddenSize: hiddenSize, actionCount: actionCount)
// Define the Adam optimizer for the network with a learning rate set to 0.01.
let optimizer = Adam(for: net, learningRate: 0.01)
...
// Begin training the agent (over a certain number of episodes).
while true {
...
// Define the gradients (use withLearningPhase to call a closure under a learning phase).
TensorFlow.gradient(at: net) { net -> Tensor<Float> in
// Return a softmax (loss) function
return loss = softmaxCrossEntropy(logits: net(input), probabilities: target)
}
}
// Update the differentiable variables of the network (`net`) along the gradients with the Adam
optimizer.
...
}
}
``````
• Train a generative adversarial network (GAN):
``````...
// Instantiate the generator and the discriminator networks after defining them.
var generator = Generator()
var discriminator = Discriminator()
// Define the Adam optimizers for each network with a learning rate set to 2e-4 and beta1 - to 0.5.
...
Start the training loop over a certain number of epochs (`epochCount`).
for epoch in 1...epochCount {
// Start the training phase.
...
for batch in trainingShuffled.batched(batchSize) {
// 1) Update the generator.
...
let 𝛁generator = TensorFlow.gradient(at: generator) { generator -> Tensor<Float> in
...
return loss
}
// Update the differentiable variables of the generator along the gradients (`𝛁generator`)

// 2) Update the discriminator.
...
let 𝛁discriminator = TensorFlow.gradient(at: discriminator) { discriminator -> Tensor<Float> in
...
return loss
}
// Update the differentiable variables of the discriminator along the gradients (`𝛁discriminator`)
}
}
``````
• ``` Model ```

#### Declaration

``public typealias Model = Model``
• ``` learningRate ```

The learning rate.

#### Declaration

``public var learningRate: Float``
• ``` beta1 ```

A coefficient used to calculate the first moments of the gradients.

#### Declaration

``public var beta1: Float``
• ``` beta2 ```

A coefficient used to calculate the second moments of the gradients.

#### Declaration

``public var beta2: Float``
• ``` epsilon ```

A small scalar added to the denominator to improve numerical stability.

#### Declaration

``public var epsilon: Float``
• ``` decay ```

The learning rate decay.

#### Declaration

``public var decay: Float``
• ``` step ```

The current step.

#### Declaration

``public var step: Int``
• ``` firstMoments ```

The first moments of the weights.

#### Declaration

``public var firstMoments: Model.TangentVector``
• ``` secondMoments ```

The second moments of the weights.

#### Declaration

``public var secondMoments: Model.TangentVector``
• ``` init(for:learningRate:beta1:beta2:epsilon:decay:) ```

#### Declaration

``````public init(
for model: __shared Model,
learningRate: Float = 1e-3,
beta1: Float = 0.9,
beta2: Float = 0.999,
epsilon: Float = 1e-8,
decay: Float = 0
)``````

#### Parameters

 ``` learningRate ``` The learning rate. The default value is `1e-3`. ``` beta1 ``` The exponential decay rate for the 1st moment estimates. The default value is `0.9`. ``` beta2 ``` The exponential decay rate for the 2nd moment estimates. The default value is `0.999`. ``` epsilon ``` A small scalar added to the denominator to improve numerical stability. The default value is `1e-8`. ``` decay ``` The learning rate decay. The default value is `0`.
• ``` update(_:along:) ```

#### Declaration

``public func update(_ model: inout Model, along direction: Model.TangentVector)``
• ``` init(copying:to:) ```

#### Declaration

``public required init(copying other: Adam, to device: Device)``
[{ "type": "thumb-down", "id": "missingTheInformationINeed", "label":"Missing the information I need" },{ "type": "thumb-down", "id": "tooComplicatedTooManySteps", "label":"Too complicated / too many steps" },{ "type": "thumb-down", "id": "outOfDate", "label":"Out of date" },{ "type": "thumb-down", "id": "samplesCodeIssue", "label":"Samples / code issue" },{ "type": "thumb-down", "id": "otherDown", "label":"Other" }]
[{ "type": "thumb-up", "id": "easyToUnderstand", "label":"Easy to understand" },{ "type": "thumb-up", "id": "solvedMyProblem", "label":"Solved my problem" },{ "type": "thumb-up", "id": "otherUp", "label":"Other" }]