Attend the Women in ML Symposium on December 7

# tfp.substrates.numpy.distributions.GaussianProcessRegressionModel

Stay organized with collections Save and categorize content based on your preferences.

Posterior predictive distribution in a conjugate GP regression model.

Inherits From: `GaussianProcess`, `Distribution`

This class represents the distribution over function values at a set of points in some index set, conditioned on noisy observations at some other set of points. More specifically, we assume a Gaussian process prior, `f ~ GP(m, k)` with IID normal noise on observations of function values. In this model posterior inference can be done analytically. This `Distribution` is parameterized by

• the mean and covariance functions of the GP prior,
• the set of (noisy) observations and index points to which they correspond,
• the set of index points at which the resulting posterior predictive distribution over function values is defined,
• the observation noise variance,
• jitter, to compensate for numerical instability of Cholesky decomposition,

in addition to the usual params like `validate_args` and `allow_nan_stats`.

#### Mathematical Details

Gaussian process regression (GPR) assumes a Gaussian process (GP) prior and a normal likelihood as a generative model for data. Given GP mean function `m`, covariance kernel `k`, and observation noise variance `v`, we have

``````  f ~ GP(m, k)

iid
(y[i] | f, x[i])  ~  Normal(f(x[i]), v),   i = 1, ... , N
``````

where `y[i]` are the noisy observations of function values at points `x[i]`.

In practice, `f` is an infinite object (eg, a function over `R^n`) which can't be realized on a finite machine, but fortunately the marginal distribution over function values at a finite set of points is just a multivariate normal with mean and covariance given by the mean and covariance functions applied at our finite set of points (see [Rasmussen and Williams, 2006][1] for a more extensive discussion of these facts).

We spell out the generative model in detail below, but first, a digression on notation. In what follows we drop the indices on vectorial objects such as `x[i]`, it being implied that we are generally considering finite collections of index points and corresponding function values and noisy observations thereof. Thus `x` should be considered to stand for a collection of index points (indeed, themselves often vectorial). Furthermore:

• `f(x)` refers to the collection of function values at the index points in the collection `x`",
• `m(t)` refers to the collection of values of the mean function at the index points in the collection `t`, and
• `k(x, t)` refers to the matrix whose entries are values of the kernel function `k` at all pairs of index points from `x` and `t`.

With these conventions in place, we may write

``````  (f(x) | x) ~ MVN(m(x), k(x, x))

(y | f(x), x) ~ Normal(f(x), v)
``````

When we condition on observed data `y` at the points `x`, we can derive the posterior distribution over function values `f(x)` at those points. We can then compute the posterior predictive distribution over function values `f(t)` at a new set of points `t`, conditional on those observed data.

``````  (f(t) | t, x, y) ~ MVN(loc, cov)

where

loc = m(t) + k(t, x) @ inv(k(x, x) + v * I) @ (y - m(x))
cov = k(t, t) - k(t, x) @ inv(k(x, x) + v * I) @ k(x, t)
``````

where `I` is the identity matrix of appropriate dimension. Finally, the distribution over noisy observations at the new set of points `t` is obtained by adding IID noise from `Normal(0., observation_noise_variance)`.

#### Examples

##### Draw joint samples from the posterior predictive distribution in a GP

regression model

``````import numpy as np
from tensorflow_probability.python.internal.backend.numpy.compat import v2 as tf
import tensorflow_probability as tfp; tfp = tfp.substrates.numpy

tfb = tfp.bijectors
tfd = tfp.distributions
psd_kernels = tfp.math.psd_kernels

# Generate noisy observations from a known function at some random points.
observation_noise_variance = .5
f = lambda x: np.sin(10*x[..., 0]) * np.exp(-x[..., 0]**2)
observation_index_points = np.random.uniform(-1., 1., 50)[..., np.newaxis]
observations = (f(observation_index_points) +
np.random.normal(0., np.sqrt(observation_noise_variance)))

index_points = np.linspace(-1., 1., 100)[..., np.newaxis]

kernel = psd_kernels.MaternFiveHalves()

gprm = tfd.GaussianProcessRegressionModel(
kernel=kernel,
index_points=index_points,
observation_index_points=observation_index_points,
observations=observations,
observation_noise_variance=observation_noise_variance)

samples = gprm.sample(10)
# ==> 10 independently drawn, joint samples at `index_points`.
``````

Above, we have used the kernel with default parameters, which are unlikely to be good. Instead, we can train the kernel hyperparameters on the data, as in the next example.

##### Optimize model parameters via maximum marginal likelihood

Here we learn the kernel parameters as well as the observation noise variance using gradient descent on the maximum marginal likelihood.

``````# Suppose we have some data from a known function. Note the index points in
# general have shape `[b1, ..., bB, f1, ..., fF]` (here we assume `F == 1`),
# so we need to explicitly consume the feature dimensions (just the last one
# here).
f = lambda x: np.sin(10*x[..., 0]) * np.exp(-x[..., 0]**2)

observation_index_points = np.random.uniform(-1., 1., 50)[..., np.newaxis]
observations = f(observation_index_points) + np.random.normal(0., .05, 50)

# Define a kernel with trainable parameters. Note we use TransformedVariable
# to apply a positivity constraint.
amplitude = tfp.util.TransformedVariable(
1., tfb.Exp(), dtype=tf.float64, name='amplitude')
length_scale = tfp.util.TransformedVariable(
1., tfb.Exp(), dtype=tf.float64, name='length_scale')

observation_noise_variance = tfp.util.TransformedVariable(
np.exp(-5), tfb.Exp(), name='observation_noise_variance')

# We'll use an unconditioned GP to train the kernel parameters.
gp = tfd.GaussianProcess(
kernel=kernel,
index_points=observation_index_points,
observation_noise_variance=observation_noise_variance)

@tf.function
def optimize():
loss = -gp.log_prob(observations)
return loss

# We can construct the posterior at a new set of `index_points` using the same
# kernel (with the same parameters, which we'll optimize below).
index_points = np.linspace(-1., 1., 100)[..., np.newaxis]
gprm = tfd.GaussianProcessRegressionModel(
kernel=kernel,
index_points=index_points,
observation_index_points=observation_index_points,
observations=observations,
observation_noise_variance=observation_noise_variance)

# First train the model, then draw and plot posterior samples.
for i in range(1000):
neg_log_likelihood_ = optimize()
if i % 100 == 0:
print("Step {}: NLL = {}".format(i, neg_log_likelihood_))

print("Final NLL = {}".format(neg_log_likelihood_))

samples = gprm.sample(10).numpy()
# ==> 10 independently drawn, joint samples at `index_points`.

import matplotlib.pyplot as plt
plt.scatter(np.squeeze(observation_index_points), observations)
plt.plot(np.stack([index_points[:, 0]]*10).T, samples.T, c='r', alpha=.2)
``````
##### Marginalization of model hyperparameters

Here we use TensorFlow Probability's MCMC functionality to perform marginalization of the model hyperparameters: kernel params as well as observation noise variance.

``````f = lambda x: np.sin(10*x[..., 0]) * np.exp(-x[..., 0]**2)
observation_index_points = np.random.uniform(-1., 1., 25)[..., np.newaxis]
observations = np.random.normal(f(observation_index_points), .05)

gaussian_process_model = tfd.JointDistributionSequential([
tfd.LogNormal(np.float64(0.), np.float64(1.)),
tfd.LogNormal(np.float64(0.), np.float64(1.)),
tfd.LogNormal(np.float64(0.), np.float64(1.)),
lambda noise_variance, length_scale, amplitude: tfd.GaussianProcess(
index_points=observation_index_points,
observation_noise_variance=noise_variance)
])

initial_chain_states = [
1e-1 * tf.ones([], dtype=np.float64, name='init_amplitude'),
1e-1 * tf.ones([], dtype=np.float64, name='init_length_scale'),
1e-1 * tf.ones([], dtype=np.float64, name='init_obs_noise_variance')
]

# Since HMC operates over unconstrained space, we need to transform the
# samples so they live in real-space.
unconstraining_bijectors = [
tfp.bijectors.Softplus(),
tfp.bijectors.Softplus(),
tfp.bijectors.Softplus(),
]

def unnormalized_log_posterior(*args):
return gaussian_process_model.log_prob(*args, x=observations)

num_results = 200
@tf.function
def run_mcmc():
return tfp.mcmc.sample_chain(
num_results=num_results,
num_burnin_steps=500,
num_steps_between_results=3,
current_state=initial_chain_states,
kernel=tfp.mcmc.TransformedTransitionKernel(
inner_kernel = tfp.mcmc.HamiltonianMonteCarlo(
target_log_prob_fn=unnormalized_log_posterior,
step_size=[np.float64(.15)],
num_leapfrog_steps=3),
bijector=unconstraining_bijectors),
trace_fn=lambda _, pkr: pkr.inner_results.is_accepted)
[
amplitudes,
length_scales,
observation_noise_variances
], is_accepted = run_mcmc()

print("Acceptance rate: {}".format(np.mean(is_accepted)))

# Now we can sample from the posterior predictive distribution at a new set
# of index points.
index_points = np.linspace(-1., 1., 200)[..., np.newaxis]
gprm = tfd.GaussianProcessRegressionModel(
# Batch of `num_results` kernels parameterized by the MCMC samples.
index_points=index_points,
observation_index_points=observation_index_points,
observations=observations,
observation_noise_variance=observation_noise_variances)
samples = gprm.sample()

# Plot posterior samples and their mean, target function, and observations.
plt.plot(np.stack([index_points[:, 0]]*num_results).T,
samples.numpy().T,
c='r',
alpha=.01)
plt.plot(index_points[:, 0], np.mean(samples, axis=0), c='k')
plt.plot(index_points[:, 0], f(index_points))
plt.scatter(observation_index_points[:, 0], observations)
``````

#### References

[1]: Carl Rasmussen, Chris Williams. Gaussian Processes For Machine Learning, 2006.

`kernel` `PositiveSemidefiniteKernel`-like instance representing the GP's covariance function.
`index_points` `float` `Tensor` representing finite collection, or batch of collections, of points in the index set over which the GP is defined. Shape has the form `[b1, ..., bB, e, f1, ..., fF]` where `F` is the number of feature dimensions and must equal `kernel.feature_ndims` and `e` is the number (size) of index points in each batch. Ultimately this distribution corresponds to an `e`-dimensional multivariate normal. The batch shape must be broadcastable with `kernel.batch_shape` and any batch dims yielded by `mean_fn`.
`observation_index_points` `float` `Tensor` representing finite collection, or batch of collections, of points in the index set for which some data has been observed. Shape has the form `[b1, ..., bB, e, f1, ..., fF]` where `F` is the number of feature dimensions and must equal `kernel.feature_ndims`, and `e` is the number (size) of index points in each batch. `[b1, ..., bB, e]` must be broadcastable with the shape of `observations`, and `[b1, ..., bB]` must be broadcastable with the shapes of all other batched parameters (`kernel.batch_shape`, `index_points`, etc). The default value is `None`, which corresponds to the empty set of observations, and simply results in the prior predictive model (a GP with noise of variance `predictive_noise_variance`).
`observations` `float` `Tensor` representing collection, or batch of collections, of observations corresponding to `observation_index_points`. Shape has the form `[b1, ..., bB, e]`, which must be brodcastable with the batch and example shapes of `observation_index_points`. The batch shape `[b1, ..., bB]` must be broadcastable with the shapes of all other batched parameters (`kernel.batch_shape`, `index_points`, etc.). The default value is `None`, which corresponds to the empty set of observations, and simply results in the prior predictive model (a GP with noise of variance `predictive_noise_variance`).
`observation_noise_variance` `float` `Tensor` representing the variance of the noise in the Normal likelihood distribution of the model. May be batched, in which case the batch shape must be broadcastable with the shapes of all other batched parameters (`kernel.batch_shape`, `index_points`, etc.). Default value: `0.`
`predictive_noise_variance` `float` `Tensor` representing the variance in the posterior predictive model. If `None`, we simply re-use `observation_noise_variance` for the posterior predictive noise. If set explicitly, however, we use this value. This allows us, for example, to omit predictive noise variance (by setting this to zero) to obtain noiseless posterior predictions of function values, conditioned on noisy observations.
`mean_fn` Python `callable` that acts on `index_points` to produce a collection, or batch of collections, of mean values at `index_points`. Takes a `Tensor` of shape `[b1, ..., bB, f1, ..., fF]` and returns a `Tensor` whose shape is broadcastable with `[b1, ..., bB]`. Default value: `None` implies the constant zero function.
`cholesky_fn`<