Continuously ranked probability score

The crps() and scrps() functions and their loo_*() counterparts can be used to compute the continuously ranked probability score (CRPS) and scaled CRPS (SCRPS) (as defined by Bolin and Wallin, 2023). CRPS is a proper scoring rule, and strictly proper when the first moment of the predictive distribution is finite. Both can be expressed in terms of samples form the predictive distribution. See, for example, a paper by Gneiting and Raftery (2007) for a comprehensive discussion on CRPS.

Usage

crps(x, ...)

scrps(x, ...)

loo_crps(x, ...)

loo_scrps(x, ...)

# S3 method for class 'matrix'
crps(x, x2, y, ..., permutations = 1)

# S3 method for class 'numeric'
crps(x, x2, y, ..., permutations = 1)

# S3 method for class 'matrix'
loo_crps(
  x,
  x2,
  y,
  log_lik,
  ...,
  permutations = 1,
  r_eff = 1,
  cores = getOption("mc.cores", 1)
)

# S3 method for class 'matrix'
scrps(x, x2, y, ..., permutations = 1)

# S3 method for class 'numeric'
scrps(x, x2, y, ..., permutations = 1)

# S3 method for class 'matrix'
loo_scrps(
  x,
  x2,
  y,
  log_lik,
  ...,
  permutations = 1,
  r_eff = 1,
  cores = getOption("mc.cores", 1)
)

Arguments

x

A S by N matrix (draws by observations), or a vector of length S when only single observation is provided in y.

...

Passed on to E_loo() in the loo_*() version of these functions.

x2

Independent draws from the same distribution as draws in x. Should be of the identical dimension.

y

A vector of observations or a single value.

permutations

An integer, with default value of 1, specifying how many times the expected value of |X - X'| (|x - x2|) is computed. The row order of x2 is shuffled as elements x and x2 are typically drawn given the same values of parameters. This happens, e.g., when one calls posterior_predict() twice for a fitted rstanarm or brms model. Generating more permutations is expected to decrease the variance of the computed expected value.

log_lik

A log-likelihood matrix the same size as x.

r_eff

An optional vector of relative effective sample size estimates containing one element per observation. See psis() for details.

cores

The number of cores to use for parallelization of [psis()]. See psis() for details.

Value

A list containing two elements: estimates and pointwise. The former reports estimator and standard error and latter the pointwise values. Following Bolin & Wallin (2023), a larger value is better.

Details

To compute (S)CRPS, the user needs to provide two sets of draws, x and x2, from the predictive distribution. This is due to the fact that formulas used to compute CRPS involve an expectation of the absolute difference of x and x2, both having the same distribution. See the permutations argument, as well as Gneiting and Raftery (2007) for details.

References

Bolin, D., & Wallin, J. (2023). Local scale invariance and robustness of proper scoring rules. Statistical Science, 38(1):140-159.

Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378.

Examples

if (FALSE) { # \dontrun{
# An example using rstanarm
library(rstanarm)
data("kidiq")
fit <- stan_glm(kid_score ~ mom_hs + mom_iq, data = kidiq)
ypred1 <- posterior_predict(fit)
ypred2 <- posterior_predict(fit)
crps(ypred1, ypred2, y = fit$y)
loo_crps(ypred1, ypred2, y = fit$y, log_lik = log_lik(fit))
} # }