|View source on GitHub|
Tensor statistics and normalization.
class EMATensorNormalizer: TensorNormalizer with exponential moving avg. mean and var estimates.
class StreamingTensorNormalizer: Normalizes mean & variance based on full history of tensor values.
class TensorNormalizer: Encapsulates tensor normalization and owns normalization variables.
kahan_summation(...): Calculate stable acculated sum using compensation for low-bits.
parallel_variance_calculation(...): Calculate the sufficient statistics (average & second moment) of two sets.