Regularizers

Regularization can help prevent overfitting. These have the signature fn(weights). The loss is typically added to tf.GraphKeys.REGULARIZATION_LOSS

tf.contrib.layers.apply_regularization(regularizer, weights_list=None)

Returns the summed penalty by applying regularizer to the weights_list.

Adding a regularization penalty over the layer weights and embedding weights can help prevent overfitting the training data. Regularization over layer biases is less common/useful, but assuming proper data preprocessing/mean subtraction, it usually shouldn't hurt much either.

Args:
  • regularizer: A function that takes a single Tensor argument and returns a scalar Tensor output.
  • weights_list: List of weights Tensors or Variables to apply regularizer over. Defaults to the GraphKeys.WEIGHTS collection if None.
Returns:

A scalar representing the overall regularization penalty.

Raises:
  • ValueError: If regularizer does not return a scalar output, or if we find no weights.

tf.contrib.layers.l1_regularizer(scale, scope=None)

Returns a function that can be used to apply L1 regularization to weights.

L1 regularization encourages sparsity.

Args:
  • scale: A scalar multiplier Tensor. 0.0 disables the regularizer.
  • scope: An optional scope name.
Returns:

A function with signature l1(weights) that apply L1 regularization.

Raises:
  • ValueError: If scale is negative or if scale is not a float.

tf.contrib.layers.l2_regularizer(scale, scope=None)

Returns a function that can be used to apply L2 regularization to weights.

Small values of L2 can help prevent overfitting the training data.

Args:
  • scale: A scalar multiplier Tensor. 0.0 disables the regularizer.
  • scope: An optional scope name.
Returns:

A function with signature l2(weights) that applies L2 regularization.

Raises:
  • ValueError: If scale is negative or if scale is not a float.

tf.contrib.layers.sum_regularizer(regularizer_list, scope=None)

Returns a function that applies the sum of multiple regularizers.

Args:
  • regularizer_list: A list of regularizers to apply.
  • scope: An optional scope name
Returns:

A function with signature sum_reg(weights) that applies the sum of all the input regularizers.