Proper Scoring Rules

Calculates the logarithmic, quadratic/Brier and spherical score from a model with binary or count outcome.

Usage

performance_score(model, verbose = TRUE, ...)

Arguments

model: Model with binary or count outcome.
verbose: Toggle off warnings.
...: Arguments from other functions, usually only used internally.

Value

A list with three elements, the logarithmic, quadratic/Brier and spherical score.

Details

Proper scoring rules can be used to evaluate the quality of model predictions and model fit. performance_score() calculates the logarithmic, quadratic/Brier and spherical scoring rules. The spherical rule takes values in the interval [0, 1], with values closer to 1 indicating a more accurate model, and the logarithmic rule in the interval [-Inf, 0], with values closer to 0 indicating a more accurate model.

For stan_lmer() and stan_glmer() models, the predicted values are based on posterior_predict(), instead of predict(). Thus, results may differ more than expected from their non-Bayesian counterparts in lme4.

Note

Code is partially based on GLMMadaptive::scoring_rules().

References

Carvalho, A. (2016). An overview of applications of proper scoring rules. Decision Analysis 13, 223–242. doi:10.1287/deca.2016.0337

Examples

## Dobson (1990) Page 93: Randomized Controlled Trial :
counts <- c(18, 17, 15, 20, 10, 20, 25, 13, 12)
outcome <- gl(3, 1, 9)
treatment <- gl(3, 3)
model <- glm(counts ~ outcome + treatment, family = poisson())

performance_score(model)
#> # Proper Scoring Rules
#> 
#> logarithmic: -2.5979
#>   quadratic:  0.2095
#>   spherical:  0.3238
# \donttest{
data(Salamanders, package = "glmmTMB")
model <- glmmTMB::glmmTMB(
  count ~ spp + mined + (1 | site),
  zi = ~ spp + mined,
  family = nbinom2(),
  data = Salamanders
)

performance_score(model)
#> # Proper Scoring Rules
#> 
#> logarithmic:  -1.3275
#>   quadratic: 262.1651
#>   spherical:   0.0316
# }