How sure are you? Confidence v. Prediction intervals

srh@haptonstahl.org (Stephen Haptonstahl, Ph.D.) — Thu, 03 Jul 2025 00:00:00 -0400

Two common uses for statistical (or machine learning) models are:

Something happened. Why?
Something will happen. What?

Spoiler Alert
#

When reporting results, tell your audience how certain you are. This is essential context.
“Why?”: When presenting parameters (drivers, slopes, causal effects, and the like) use something like a confidence interval, Bayesian credible interval, or bootstrapping the data.
“What?”: When presenting predictions, use a conformal prediction interval, Bayesian posterior predictive interval, or the like. Forecasts must use these.

“Why?” = inference about drivers
#

When we ask “why?” we are trying to make an inference about how the data was generated. Specifically, inferences about the parameters of the model, which can be thought of as measures of the relationships between features and the target.

One number doesn't do it: Context matters

srh@haptonstahl.org (Stephen Haptonstahl, Ph.D.) — Sun, 22 Jun 2025 00:00:00 -0400

“Our net promoter score for the new version is 42.”

Okay. So what? Is 42 good or bad? Should we continue rolling out the new version, or should we roll back to the previous version?

Uncertainty on Polimath

How sure are you? Confidence v. Prediction intervals

Spoiler Alert #

“Why?” = inference about drivers #

One number doesn't do it: Context matters

Spoiler Alert
#

“Why?” = inference about drivers
#