Common Shannon entropy, the Evidence Lower BOund (ELBO), KL divergence, and more
all have information theoretic use and interpretations. They are also often
used in variational inference. This library brings together
estimating them, e.g. using Monte Carlo expectations.
Example of fitting a variational posterior with the ELBO.
# We start by assuming knowledge of the log of a joint density p(z, x) over # latent variable z and fixed measurement x. Since x is fixed, the Python # function does not take x as an argument. def log_joint(z): theta = tf.Variable(0.) # Trainable variable that helps define log_joint. ... # Next, define a Normal distribution with trainable parameters. q = distributions.Normal(mu=tf.Variable(0.), sigma=tf.Variable(1.)) # Now, define a loss function (negative ELBO) that, when minimized, will adjust # mu, sigma, and theta, increasing the ELBO, which we hope will both reduce the # KL divergence between q(z) and p(z | x), and increase p(x). Note that we # cannot guarantee both, but in general we expect both to happen. elbo = entropy.elbo_ratio(log_p, q, n=10) loss = -elbo # Minimize the loss train_op = tf.train.GradientDescentOptimizer(0.1).minimize(loss) tf.global_variables_initializer().run() for step in range(100): train_op.run()