Ignore observed data shape when sampling posterior predictive

[twiecki]

When making predictions we always need to make sure that the data into the likelihood has the right shape, even though we don't even need it because we resample new data.

We could just remove the observed data for PP sampling so that that isn't required anymore.

[ricardoV94]

I often miss the ability to specify a shape different than the observed data for PP sampling. I think this Discourse issue is about the same: https://discourse.pymc.io/t/posterior-predictive-check-limited-to-certain-sample-sizes/7120/3

Is this what you meant with this Issue? If not, could you give a minimal example?

[mjhajharia]

aren't we using dims and coords instead of shape now(or atleast encouraging that)?

[twiecki]

For a lot of models like linear regression there are two pm.Data() for the X and Y (this one goes into the observed).

Then when doing post pred sampling we need to update X and Y, but Y can just be dummy placeholder values. I think setting Y in this case is unnecessary as the shape of X just translates through to the output.

I'm not quite sure how to handle the post pred shape in the case where the likelihood shape is not specified externally (via an X). Maybe as a kwarg to sample_posterior_predictive?

[mjhajharia]

maybe in the second occurence of pm.data, we can just do pm.set_data({'data': data_vals})
in between final sampling and initial input there's one data container / placeholder to train part right?

think setting Y in this case is unnecessary as the shape of X just translates through to the output.

I might be missing something but we need some sort of placeholders for X right, which we train during sample_pp. I'm not sure if I make sense, I'm somewhat speculating I need to understand the whole coords dims thing properly again.

[twiecki]

@almostmeenal We still need to set X, no way around that, as we still need the features/predictors.

There's analog to scikit-learn. For us, specifying and sampling the model is basically sklearnmodel.fit(X_train, y). Their sklearnmodel.predict(X_test) is for us:

pm.set_data({"X": X_test, "Y": np.empty(X_test.shape[0])})
pm.sample_posterior_predictive()

I would like us to get rid of having to set Y above.

[mjhajharia]

pm.set_data({"X": X_test, "Y": np.empty(X_test.shape[0])})
pm.sample_posterior_predictive()

got it, thanks for clarifying. So is the likely solution then only removing observed data?

[twiecki]

So that would work for the case where there's an X to specify the shape. However, what if that's not the case? (When writing this issue I hadn't considered this complexity, but maybe we can think it through here).

An example would be the coin-flip model where I infer a p of a Bernoulli with n coin-flips (specified by the length of the data I pass in). In that case if I just call pm.sample_posterior_predictive() I would get pp samples of length n which is what I want. In that case we couldn't just remove the data. If we want to get more or less length, we would need to use pm.Data for the observed data and update with a place-holder. There we also couldn't replace.

So the key question is if the shape of the observed is specified by its inputs (like when an X is present) or not. And I'm not sure we can automatically infer that.

One solution might be to rely on the dims kwarg if it's present to specify the shape. Although I'm not sure we have a method of updating that. Perhaps this needs to wait for something like pymc-devs/pytensor#352.

Anyone got ideas?

[brandonwillard]

If I'm understanding this correctly, all this is already covered in v4.

[twiecki]

@brandonwillard Neat! How do I set the target shape in v4?

[brandonwillard]

Here's a small example in v4:

import numpy as np

import aesara
import aesara.tensor as at

import pymc3 as pm


observed = aesara.shared(np.random.normal(10, 1, size=200), borrow=True)

with pm.Model() as m:
    mu = pm.Normal("mu", 0, 100)
    Y = pm.Normal("Y", mu, 1, observed=observed)

with m:
    trace = pm.sample()

with m:
    pp_trace = pm.sample_posterior_predictive(trace)

The original shape of the observations is 200:

>>> pp_trace["Y"].shape
(4000, 200)

Change the shape of the observations to 100:

observed.set_value(np.random.normal(10, 1, size=100))

with m:
    pp_trace = pm.sample_posterior_predictive(trace)

>>> pp_trace["Y"].shape
(4000, 100)

Change the shape to (2, 3, 100) (i.e. sample a size of (2, 3)):

with m:
    pp_trace = pm.sample_posterior_predictive(trace, size=(2, 3))

>>> pp_trace["Y"].shape
(4000, 2, 3, 100)

The same can be done for any other shared variables (i.e. not just observed ones) or variables that get their shapes from the shape of a shared variable (e.g. regressions parameters).

[ricardoV94]

This issue needs a bit more discussion to understand what is the behavior we want. I don't think we want to always ignore the shape of the observed variables.