Optimizer that implements the FTRL algorithm.

Inherits From: Optimizer

"Follow The Regularized Leader" (FTRL) is an optimization algorithm developed at Google for click-through rate prediction in the early 2010s. It is most suitable for shallow models with large and sparse feature spaces. The algorithm is described in this paper. The Keras version has support for both online L2 regularization (the L2 regularization described in the paper above) and shrinkage-type L2 regularization (which is the addition of an L2 penalty to the loss function).


n = 0
sigma = 0
z = 0

Update rule for one variable w:

prev_n = n
n = n + g ** 2
sigma = (sqrt(n) - sqrt(prev_n)) / lr
z = z + g - sigma * w
if abs(z) < lambda_1:
  w = 0
  w = (sgn(z) * lambda_1 - z) / ((beta + sqrt(n)) / alpha + lambda_2)


  • lr is the learning rate
  • g is the gradient for the variable
  • lambda_1 is the L1 regularization strength
  • lambda_2 is the L2 regularization strength

Check the documentation for the l2_shrinkage_regularization_strength parameter for more details when shrinkage is enabled, in which case gradient is replaced with a gradient with shrinkage.