Compute the (conditional) equivalence test for frequentist models.
# S3 method for lm
equivalence_test(
x,
range = "default",
ci = 0.95,
rule = "classic",
p_values = FALSE,
verbose = TRUE,
...
)
# S3 method for merMod
equivalence_test(
x,
range = "default",
ci = 0.95,
rule = "classic",
effects = c("fixed", "random"),
p_values = FALSE,
verbose = TRUE,
...
)
x  A statistical model. 

range  The range of practical equivalence of an effect. May be

ci  Confidence Interval (CI) level. Default to 
rule  Character, indicating the rules when testing for practical
equivalence. Can be 
p_values  Logical, if 
verbose  Toggle warnings and messages. 
...  Arguments passed to or from other methods. 
effects  Should parameters for fixed effects ( 
A data frame.
In classical null hypothesis significance testing (NHST) within a frequentist
framework, it is not possible to accept the null hypothesis, H0  unlike
in Bayesian statistics, where such probability statements are possible.
“... one can only reject the null hypothesis if the test
statistics falls into the critical region(s), or fail to reject this
hypothesis. In the latter case, all we can say is that no significant effect
was observed, but one cannot conclude that the null hypothesis is true.”
(Pernet 2017). One way to address this issues without Bayesian methods
is Equivalence Testing, as implemented in equivalence_test()
.
While you either can reject the null hypothesis or claim an inconclusive result
in NHST, the equivalence test adds a third category, "accept". Roughly
speaking, the idea behind equivalence testing in a frequentist framework is
to check whether an estimate and its uncertainty (i.e. confidence interval)
falls within a region of "practical equivalence". Depending on the rule for
this test (see below), statistical significance does not necessarily indicate
whether the null hypothesis can be rejected or not, i.e. the classical
interpretation of the pvalue may differ from the results returned from
the equivalence test.
This rule follows the “HDI+ROPE decision rule” (Kruschke,
2014, 2018) used for the
Bayesian counterpart()
. This
means, if the confidence intervals are completely outside the ROPE, the
"null hypothesis" for this parameter is "rejected". If the ROPE
completely covers the CI, the null hypothesis is accepted. Else, it's
undecided whether to accept or reject the null hypothesis. Desirable
results are low proportions inside the ROPE (the closer to zero the
better).
This rule follows the “TOST rule”, i.e. a two onesided test
procedure (Lakens 2017). Following this rule, practical
equivalence of an effect (i.e. H0) is rejected, when the
coefficient is statistically significant and the narrow
confidence intervals (i.e. 12*alpha
) include or
exceed the ROPE boundaries. Practical equivalence is assumed
(i.e. H0 accepted) when the narrow confidence intervals are completely
inside the ROPE, no matter if the effect is statistically significant
or not. Else, the decision whether to accept or reject H0 is undecided.
The Conditional Equivalence Testing as described by Campbell and Gustafson 2018. According to this rule, practical equivalence is rejected when the coefficient is statistically significant. When the effect is not significant and the narrow confidence intervals are completely inside the ROPE, we accept H0, else it is undecided.
For rule = "classic"
, "narrow" confidence intervals are used for
equivalence testing. "Narrow" means, the the intervals is not 1  alpha,
but 1  2 * alpha. Thus, if ci = .95
, alpha is assumed to be 0.05
and internally a cilevel of 0.90 is used. rule = "cet"
uses
both regular and narrow confidence intervals, while rule = "bayes"
only uses the regular intervals.
Second generation pvalues (SGPV) were proposed as a statistic
that represents “the proportion of datasupported hypotheses
that are also null hypotheses” (Blume et al. 2018). This statistic
is actually computed in the same way as the percentage inside the ROPE as
returned by equivalence_test()
(see Lakens and Delacre 2020
for details on computation of the SGPV). Thus, the "inside ROPE"
column reflects the SGPV.
The calculation of pvalues is somewhat "experimental". For parameters, where H0...
... is rejected, the pvalue equals a NHST as if the upper / lower
boundary of the ROPE (see range
) would be the pointnull to test
against.
... is accepted, the pvalue is set to 1.
... is undecided, the pvalue equals a NHST against the pointnull, however, the "uncertainty" (i.e. ROPE range) is added to the confidence intervals (so the upper confidence interval limit equals the regular upper confidence interval limit + half the ROPE range).
All pvalues are then adjusted for multiple testing (using
stats::p.adjust()
with method = "fdr"
).
Some attention is required for finding suitable values for the ROPE limits
(argument range
). See 'Details' in bayestestR::rope_range()
for further information.
There is also a plot()
method implemented in the seepackage.
Blume, J. D., D'Agostino McGowan, L., Dupont, W. D., & Greevy, R. A. (2018). Secondgeneration pvalues: Improved rigor, reproducibility, & transparency in statistical analyses. PLOS ONE, 13(3), e0188299. https://doi.org/10.1371/journal.pone.0188299
Campbell, H., & Gustafson, P. (2018). Conditional equivalence testing: An alternative remedy for publication bias. PLOS ONE, 13(4), e0195145. doi: 10.1371/journal.pone.0195145
Kruschke, J. K. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan. Academic Press
Kruschke, J. K. (2018). Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science, 1(2), 270280. doi: 10.1177/2515245918771304
Lakens, D. (2017). Equivalence Tests: A Practical Primer for t Tests, Correlations, and MetaAnalyses. Social Psychological and Personality Science, 8(4), 355–362. doi: 10.1177/1948550617697177
Lakens, D., & Delacre, M. (2020). Equivalence Testing and the Second Generation PValue. MetaPsychology, 4. https://doi.org/10.15626/MP.2018.933
Pernet, C. (2017). Null hypothesis significance testing: A guide to commonly misunderstood concepts and recommendations for good practice. F1000Research, 4, 621. doi: 10.12688/f1000research.6963.5
For more details, see bayestestR::equivalence_test()
.
Further readings can be found in the references.
data(qol_cancer)
model < lm(QoL ~ time + age + education, data = qol_cancer)
# default rule
equivalence_test(model)
#> # TOSTtest for Practical Equivalence
#>
#> ROPE: [1.99 1.99]
#>
#> Parameter H0 inside ROPE 90% CI
#> (Intercept) Rejected 0.00 % [59.33 68.41]
#> time Undecided 83.52 % [0.76 2.53]
#> age Accepted 100.00 % [0.26 0.32]
#> education [mid] Rejected 0.00 % [ 5.13 12.39]
#> education [high] Rejected 0.00 % [10.14 18.57]
#>
# conditional equivalence test
equivalence_test(model, rule = "cet")
#> # Conditional Equivalence Testing
#>
#> ROPE: [1.99 1.99]
#>
#> Parameter H0 inside ROPE 90% CI
#> (Intercept) Rejected 0.00 % [59.33 68.41]
#> time Undecided 83.52 % [0.76 2.53]
#> age Accepted 100.00 % [0.26 0.32]
#> education [mid] Rejected 0.00 % [ 5.13 12.39]
#> education [high] Rejected 0.00 % [10.14 18.57]
#>
# plot method
if (require("see", quietly = TRUE)) {
result < equivalence_test(model)
plot(result)
}
#> Error: package or namespace load failed for ‘see’:
#> object ‘check_heterogeneity’ is not exported by 'namespace:parameters'