Performance of Bayesian Models

Compute indices of model performance for (general) linear models.

Usage

# S3 method for class 'stanreg'
model_performance(model, metrics = "all", verbose = TRUE, ...)

# S3 method for class 'BFBayesFactor'
model_performance(
  model,
  metrics = "all",
  verbose = TRUE,
  average = FALSE,
  prior_odds = NULL,
  ...
)

Arguments

model: Object of class stanreg or brmsfit.
metrics: Can be "all", "common" or a character vector of metrics to be computed (some of c("LOOIC", "WAIC", "R2", "R2_adj", "RMSE", "SIGMA", "LOGLOSS", "SCORE")). "common" will compute LOOIC, WAIC, R2 and RMSE.
verbose: Toggle off warnings.
...: Arguments passed to or from other methods.
average: Compute model-averaged index? See bayestestR::weighted_posteriors().
prior_odds: Optional vector of prior odds for the models compared to the first model (or the denominator, for BFBayesFactor objects). For data.frames, this will be used as the basis of weighting.

Value

A data frame (with one row) and one column per "index" (see metrics).

Details

Depending on model, the following indices are computed:

ELPD: expected log predictive density. Larger ELPD values mean better fit. See looic().
LOOIC: leave-one-out cross-validation (LOO) information criterion. Lower LOOIC values mean better fit. See looic().
WAIC: widely applicable information criterion. Lower WAIC values mean better fit. See ?loo::waic.
R2: r-squared value, see r2_bayes().
R2_adjusted: LOO-adjusted r-squared, see r2_loo().
RMSE: root mean squared error, see performance_rmse().
SIGMA: residual standard deviation, see insight::get_sigma().
LOGLOSS: Log-loss, see performance_logloss().
SCORE_LOG: score of logarithmic proper scoring rule, see performance_score().
SCORE_SPHERICAL: score of spherical proper scoring rule, see performance_score().
PCP: percentage of correct predictions, see performance_pcp().

References

Gelman, A., Goodrich, B., Gabry, J., and Vehtari, A. (2018). R-squared for Bayesian regression models. The American Statistician, The American Statistician, 1-6.

Examples

# \donttest{
model <- suppressWarnings(rstanarm::stan_glm(
  mpg ~ wt + cyl,
  data = mtcars,
  chains = 1,
  iter = 500,
  refresh = 0
))
model_performance(model)
#> # Indices of model performance
#> 
#> ELPD    | ELPD_SE |   LOOIC | LOOIC_SE |    WAIC |    R2 | R2 (adj.) |  RMSE | Sigma
#> ------------------------------------------------------------------------------------
#> -78.243 |   4.270 | 156.486 |    8.539 | 156.468 | 0.814 |     0.798 | 2.445 | 2.660

model <- suppressWarnings(rstanarm::stan_glmer(
  mpg ~ wt + cyl + (1 | gear),
  data = mtcars,
  chains = 1,
  iter = 500,
  refresh = 0
))
model_performance(model)
#> # Indices of model performance
#> 
#> ELPD    | ELPD_SE |   LOOIC | LOOIC_SE |    WAIC |    R2 | R2 (marg.)
#> ---------------------------------------------------------------------
#> -79.362 |   4.741 | 158.723 |    9.482 | 158.664 | 0.820 |      0.823
#> 
#> ELPD    | R2 (adj.) | R2_adjusted_marginal |   ICC |  RMSE | Sigma
#> ------------------------------------------------------------------
#> -79.362 |     0.788 |                0.788 | 0.184 | 2.442 | 2.594
# }