Attend the Women in ML Symposium on December 7 Register now

tfp.substrates.numpy.distributions.GaussianProcessRegressionModel

Stay organized with collections Save and categorize content based on your preferences.

Posterior predictive distribution in a conjugate GP regression model.

Inherits From: GaussianProcess, Distribution

This class represents the distribution over function values at a set of points in some index set, conditioned on noisy observations at some other set of points. More specifically, we assume a Gaussian process prior, f ~ GP(m, k) with IID normal noise on observations of function values. In this model posterior inference can be done analytically. This Distribution is parameterized by

  • the mean and covariance functions of the GP prior,
  • the set of (noisy) observations and index points to which they correspond,
  • the set of index points at which the resulting posterior predictive distribution over function values is defined,
  • the observation noise variance,
  • jitter, to compensate for numerical instability of Cholesky decomposition,

in addition to the usual params like validate_args and allow_nan_stats.

Mathematical Details

Gaussian process regression (GPR) assumes a Gaussian process (GP) prior and a normal likelihood as a generative model for data. Given GP mean function m, covariance kernel k, and observation noise variance v, we have

  f ~ GP(m, k)

                   iid
  (y[i] | f, x[i])  ~  Normal(f(x[i]), v),   i = 1, ... , N

where y[i] are the noisy observations of function values at points x[i].

In practice, f is an infinite object (eg, a function over R^n) which can't be realized on a finite machine, but fortunately the marginal distribution over function values at a finite set of points is just a multivariate normal with mean and covariance given by the mean and covariance functions applied at our finite set of points (see [Rasmussen and Williams, 2006][1] for a more extensive discussion of these facts).

We spell out the generative model in detail below, but first, a digression on notation. In what follows we drop the indices on vectorial objects such as x[i], it being implied that we are generally considering finite collections of index points and corresponding function values and noisy observations thereof. Thus x should be considered to stand for a collection of index points (indeed, themselves often vectorial). Furthermore:

  • f(x) refers to the collection of function values at the index points in the collection x",
  • m(t) refers to the collection of values of the mean function at the index points in the collection t, and
  • k(x, t) refers to the matrix whose entries are values of the kernel function k at all pairs of index points from x and t.

With these conventions in place, we may write

  (f(x) | x) ~ MVN(m(x), k(x, x))

  (y | f(x), x) ~ Normal(f(x), v)

When we condition on observed data y at the points x, we can derive the posterior distribution over function values f(x) at those points. We can then compute the posterior predictive distribution over function values f(t) at a new set of points t, conditional on those observed data.

  (f(t) | t, x, y) ~ MVN(loc, cov)

  where

  loc = m(t) + k(t, x) @ inv(k(x, x) + v * I) @ (y - m(x))
  cov = k(t, t) - k(t, x) @ inv(k(x, x) + v * I) @ k(x, t)

where I is the identity matrix of appropriate dimension. Finally, the distribution over noisy observations at the new set of points t is obtained by adding IID noise from Normal(0., observation_noise_variance).

Examples

Draw joint samples from the posterior predictive distribution in a GP

regression model

import numpy as np
from tensorflow_probability.python.internal.backend.numpy.compat import v2 as tf
import tensorflow_probability as tfp; tfp = tfp.substrates.numpy

tfb = tfp.bijectors
tfd = tfp.distributions
psd_kernels = tfp.math.psd_kernels

# Generate noisy observations from a known function at some random points.
observation_noise_variance = .5
f = lambda x: np.sin(10*x[..., 0]) * np.exp(-x[..., 0]**2)
observation_index_points = np.random.uniform(-1., 1., 50)[..., np.newaxis]
observations = (f(observation_index_points) +
                np.random.normal(0., np.sqrt(observation_noise_variance)))

index_points = np.linspace(-1., 1., 100)[..., np.newaxis]

kernel = psd_kernels.MaternFiveHalves()

gprm = tfd.GaussianProcessRegressionModel(
    kernel=kernel,
    index_points=index_points,
    observation_index_points=observation_index_points,
    observations=observations,
    observation_noise_variance=observation_noise_variance)

samples = gprm.sample(10)
# ==> 10 independently drawn, joint samples at `index_points`.

Above, we have used the kernel with default parameters, which are unlikely to be good. Instead, we can train the kernel hyperparameters on the data, as in the next example.

Optimize model parameters via maximum marginal likelihood

Here we learn the kernel parameters as well as the observation noise variance using gradient descent on the maximum marginal likelihood.

# Suppose we have some data from a known function. Note the index points in
# general have shape `[b1, ..., bB, f1, ..., fF]` (here we assume `F == 1`),
# so we need to explicitly consume the feature dimensions (just the last one
# here).
f = lambda x: np.sin(10*x[..., 0]) * np.exp(-x[..., 0]**2)

observation_index_points = np.random.uniform(-1., 1., 50)[..., np.newaxis]
observations = f(observation_index_points) + np.random.normal(0., .05, 50)

# Define a kernel with trainable parameters. Note we use TransformedVariable
# to apply a positivity constraint.
amplitude = tfp.util.TransformedVariable(
  1., tfb.Exp(), dtype=tf.float64, name='amplitude')
length_scale = tfp.util.TransformedVariable(
  1., tfb.Exp(), dtype=tf.float64, name='length_scale')
kernel = psd_kernels.ExponentiatedQuadratic(amplitude, length_scale)

observation_noise_variance = tfp.util.TransformedVariable(
    np.exp(-5), tfb.Exp(), name='observation_noise_variance')

# We'll use an unconditioned GP to train the kernel parameters.
gp = tfd.GaussianProcess(
    kernel=kernel,
    index_points=observation_index_points,
    observation_noise_variance=observation_noise_variance)

optimizer = tf.optimizers.Adam(learning_rate=.05, beta_1=.5, beta_2=.99)

@tf.function
def optimize():
  with tf.GradientTape() as tape:
    loss = -gp.log_prob(observations)
  grads = tape.gradient(loss, gp.trainable_variables)
  optimizer.apply_gradients(zip(grads, gp.trainable_variables))
  return loss

# We can construct the posterior at a new set of `index_points` using the same
# kernel (with the same parameters, which we'll optimize below).
index_points = np.linspace(-1., 1., 100)[..., np.newaxis]
gprm = tfd.GaussianProcessRegressionModel(
    kernel=kernel,
    index_points=index_points,
    observation_index_points=observation_index_points,
    observations=observations,
    observation_noise_variance=observation_noise_variance)

# First train the model, then draw and plot posterior samples.
for i in range(1000):
  neg_log_likelihood_ = optimize()
  if i % 100 == 0:
    print("Step {}: NLL = {}".format(i, neg_log_likelihood_))

print("Final NLL = {}".format(neg_log_likelihood_))

samples = gprm.sample(10).numpy()
# ==> 10 independently drawn, joint samples at `index_points`.

import matplotlib.pyplot as plt
plt.scatter(np.squeeze(observation_index_points), observations)
plt.plot(np.stack([index_points[:, 0]]*10).T, samples.T, c='r', alpha=.2)
Marginalization of model hyperparameters

Here we use TensorFlow Probability's MCMC functionality to perform marginalization of the model hyperparameters: kernel params as well as observation noise variance.

f = lambda x: np.sin(10*x[..., 0]) * np.exp(-x[..., 0]**2)
observation_index_points = np.random.uniform(-1., 1., 25)[..., np.newaxis]
observations = np.random.normal(f(observation_index_points), .05)

gaussian_process_model = tfd.JointDistributionSequential([
  tfd.LogNormal(np.float64(0.), np.float64(1.)),
  tfd.LogNormal(np.float64(0.), np.float64(1.)),
  tfd.LogNormal(np.float64(0.), np.float64(1.)),
  lambda noise_variance, length_scale, amplitude: tfd.GaussianProcess(
      kernel=psd_kernels.ExponentiatedQuadratic(amplitude, length_scale),
      index_points=observation_index_points,
      observation_noise_variance=noise_variance)
])

initial_chain_states = [
    1e-1 * tf.ones([], dtype=np.float64, name='init_amplitude'),
    1e-1 * tf.ones([], dtype=np.float64, name='init_length_scale'),
    1e-1 * tf.ones([], dtype=np.float64, name='init_obs_noise_variance')
]

# Since HMC operates over unconstrained space, we need to transform the
# samples so they live in real-space.
unconstraining_bijectors = [
    tfp.bijectors.Softplus(),
    tfp.bijectors.Softplus(),
    tfp.bijectors.Softplus(),
]

def unnormalized_log_posterior(*args):
  return gaussian_process_model.log_prob(*args, x=observations)

num_results = 200
@tf.function
def run_mcmc():
  return tfp.mcmc.sample_chain(
      num_results=num_results,
      num_burnin_steps=500,
      num_steps_between_results=3,
      current_state=initial_chain_states,
      kernel=tfp.mcmc.TransformedTransitionKernel(
          inner_kernel = tfp.mcmc.HamiltonianMonteCarlo(
              target_log_prob_fn=unnormalized_log_posterior,
              step_size=[np.float64(.15)],
              num_leapfrog_steps=3),
          bijector=unconstraining_bijectors),
      trace_fn=lambda _, pkr: pkr.inner_results.is_accepted)
[
      amplitudes,
      length_scales,
      observation_noise_variances
], is_accepted = run_mcmc()

print("Acceptance rate: {}".format(np.mean(is_accepted)))

# Now we can sample from the posterior predictive distribution at a new set
# of index points.
index_points = np.linspace(-1., 1., 200)[..., np.newaxis]
gprm = tfd.GaussianProcessRegressionModel(
    # Batch of `num_results` kernels parameterized by the MCMC samples.
    kernel=psd_kernels.ExponentiatedQuadratic(amplitudes, length_scales),
    index_points=index_points,
    observation_index_points=observation_index_points,
    observations=observations,
    observation_noise_variance=observation_noise_variances)
samples = gprm.sample()

# Plot posterior samples and their mean, target function, and observations.
plt.plot(np.stack([index_points[:, 0]]*num_results).T,
        samples.numpy().T,
        c='r',
        alpha=.01)
plt.plot(index_points[:, 0], np.mean(samples, axis=0), c='k')
plt.plot(index_points[:, 0], f(index_points))
plt.scatter(observation_index_points[:, 0], observations)

References

[1]: Carl Rasmussen, Chris Williams. Gaussian Processes For Machine Learning, 2006.

kernel PositiveSemidefiniteKernel-like instance representing the GP's covariance function.
index_points float Tensor representing finite collection, or batch of collections, of points in the index set over which the GP is defined. Shape has the form [b1, ..., bB, e, f1, ..., fF] where F is the number of feature dimensions and must equal kernel.feature_ndims and e is the number (size) of index points in each batch. Ultimately this distribution corresponds to an e-dimensional multivariate normal. The batch shape must be broadcastable with kernel.batch_shape and any batch dims yielded by mean_fn.
observation_index_points float Tensor representing finite collection, or batch of collections, of points in the index set for which some data has been observed. Shape has the form [b1, ..., bB, e, f1, ..., fF] where F is the number of feature dimensions and must equal kernel.feature_ndims, and e is the number (size) of index points in each batch. [b1, ..., bB, e] must be broadcastable with the shape of observations, and [b1, ..., bB] must be broadcastable with the shapes of all other batched parameters (kernel.batch_shape, index_points, etc). The default value is None, which corresponds to the empty set of observations, and simply results in the prior predictive model (a GP with noise of variance predictive_noise_variance).
observations float Tensor representing collection, or batch of collections, of observations corresponding to observation_index_points. Shape has the form [b1, ..., bB, e], which must be brodcastable with the batch and example shapes of observation_index_points. The batch shape [b1, ..., bB] must be broadcastable with the shapes of all other batched parameters (kernel.batch_shape, index_points, etc.). The default value is None, which corresponds to the empty set of observations, and simply results in the prior predictive model (a GP with noise of variance predictive_noise_variance).
observation_noise_variance float Tensor representing the variance of the noise in the Normal likelihood distribution of the model. May be batched, in which case the batch shape must be broadcastable with the shapes of all other batched parameters (kernel.batch_shape, index_points, etc.). Default value: 0.
predictive_noise_variance float Tensor representing the variance in the posterior predictive model. If None, we simply re-use observation_noise_variance for the posterior predictive noise. If set explicitly, however, we use this value. This allows us, for example, to omit predictive noise variance (by setting this to zero) to obtain noiseless posterior predictions of function values, conditioned on noisy observations.
mean_fn Python callable that acts on index_points to produce a collection, or batch of collections, of mean values at index_points. Takes a Tensor of shape [b1, ..., bB, f1, ..., fF] and returns a Tensor whose shape is broadcastable with [b1, ..., bB]. Default value: None implies the constant zero function.
cholesky_fn<