Effect size for differences

Compute effect size indices for standardized differences: Cohen's d, Hedges' g and Glass’s delta (\(\Delta\)). (This function returns the population estimate.)

Both Cohen's d and Hedges' g are the estimated the standardized difference between the means of two populations. Hedges' g provides a bias correction (using the exact method) to Cohen's d for small sample sizes. For sample sizes > 20, the results for both statistics are roughly equivalent. Glass’s delta is appropriate when the standard deviations are significantly different between the populations, as it uses only the second group's standard deviation.

Usage

cohens_d(
  x,
  y = NULL,
  data = NULL,
  pooled_sd = TRUE,
  mu = 0,
  paired = FALSE,
  ci = 0.95,
  alternative = "two.sided",
  verbose = TRUE,
  ...
)

hedges_g(
  x,
  y = NULL,
  data = NULL,
  pooled_sd = TRUE,
  mu = 0,
  paired = FALSE,
  ci = 0.95,
  alternative = "two.sided",
  verbose = TRUE,
  ...
)

glass_delta(
  x,
  y = NULL,
  data = NULL,
  mu = 0,
  ci = 0.95,
  alternative = "two.sided",
  verbose = TRUE,
  ...
)

Arguments

x: A formula, a numeric vector, or a character name of one in data.
y: A numeric vector, a grouping (character / factor) vector, a or a character name of one in data. Ignored if x is a formula.
data: An optional data frame containing the variables.
pooled_sd: If TRUE (default), a sd_pooled() is used (assuming equal variance). Else the mean SD from both groups is used instead.
mu: a number indicating the true value of the mean (or difference in means if you are performing a two sample test).
paired: If TRUE, the values of x and y are considered as paired. This produces an effect size that is equivalent to the one-sample effect size on x - y.
ci: Confidence Interval (CI) level
alternative: a character string specifying the alternative hypothesis; Controls the type of CI returned: "two.sided" (default, two-sided CI), "greater" or "less" (one-sided CI). Partial matching is allowed (e.g., "g", "l", "two"...). See One-Sided CIs in effectsize_CIs.
verbose: Toggle warnings and messages on or off.
...: Arguments passed to or from other methods. When x is a formula, these can be subset and na.action.

Value

A data frame with the effect size ( Cohens_d, Hedges_g, Glass_delta) and their CIs (CI_low and CI_high).

Details

Set pooled_sd = FALSE for effect sizes that are to accompany a Welch's t-test (Delacre et al, 2021).

Note

The indices here give the population estimated standardized difference. Some statistical packages give the sample estimate instead (without applying Bessel's correction).

Confidence (Compatibility) Intervals (CIs)

Unless stated otherwise, confidence (compatibility) intervals (CIs) are estimated using the noncentrality parameter method (also called the "pivot method"). This method finds the noncentrality parameter ("ncp") of a noncentral t, F, or \(\chi^2\) distribution that places the observed t, F, or \(\chi^2\) test statistic at the desired probability point of the distribution. For example, if the observed t statistic is 2.0, with 50 degrees of freedom, for which cumulative noncentral t distribution is t = 2.0 the .025 quantile (answer: the noncentral t distribution with ncp = .04)? After estimating these confidence bounds on the ncp, they are converted into the effect size metric to obtain a confidence interval for the effect size (Steiger, 2004).

For additional details on estimation and troubleshooting, see effectsize_CIs.

CIs and Significance Tests

"Confidence intervals on measures of effect size convey all the information in a hypothesis test, and more." (Steiger, 2004). Confidence (compatibility) intervals and p values are complementary summaries of parameter uncertainty given the observed data. A dichotomous hypothesis test could be performed with either a CI or a p value. The 100 (1 - \(\alpha\))% confidence interval contains all of the parameter values for which p > \(\alpha\) for the current data and model. For example, a 95% confidence interval contains all of the values for which p > .05.

Note that a confidence interval including 0 does not indicate that the null (no effect) is true. Rather, it suggests that the observed data together with the model and its assumptions combined do not provided clear evidence against a parameter value of 0 (same as with any other value in the interval), with the level of this evidence defined by the chosen \(\alpha\) level (Rafi & Greenland, 2020; Schweder & Hjort, 2016; Xie & Singh, 2013). To infer no effect, additional judgments about what parameter values are "close enough" to 0 to be negligible are needed ("equivalence testing"; Bauer & Kiesser, 1996).

References

Algina, J., Keselman, H. J., & Penfield, R. D. (2006). Confidence intervals for an effect size when variances are not equal. Journal of Modern Applied Statistical Methods, 5(1), 2.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Ed.). New York: Routledge.
Delacre, M., Lakens, D., Ley, C., Liu, L., & Leys, C. (2021, May 7). Why Hedges’ g*s based on the non-pooled standard deviation should be reported with Welch's t-test. https://doi.org/10.31234/osf.io/tu6mp
Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Sage.

Examples

# \donttest{
data(mtcars)
mtcars$am <- factor(mtcars$am)

# Two Independent Samples ----------

(d <- cohens_d(mpg ~ am, data = mtcars))
#> Cohen's d |         95% CI
#> --------------------------
#> -1.48     | [-2.27, -0.67]
#> 
#> - Estimated using pooled SD.
# Same as:
# cohens_d("mpg", "am", data = mtcars)
# cohens_d(mtcars$mpg[mtcars$am=="0"], mtcars$mpg[mtcars$am=="1"])

# More options:
cohens_d(mpg ~ am, data = mtcars, pooled_sd = FALSE)
#> Cohen's d |         95% CI
#> --------------------------
#> -1.41     | [-2.26, -0.53]
#> 
#> - Estimated using un-pooled SD.
cohens_d(mpg ~ am, data = mtcars, mu = -5)
#> Cohen's d |        95% CI
#> -------------------------
#> -0.46     | [-1.17, 0.26]
#> 
#> - Deviation from a difference of -5.
#> - Estimated using pooled SD.
cohens_d(mpg ~ am, data = mtcars, alternative = "less")
#> Cohen's d |        95% CI
#> -------------------------
#> -1.48     | [-Inf, -0.80]
#> 
#> - Estimated using pooled SD.
#> - One-sided CIs: lower bound fixed at [-Inf].
hedges_g(mpg ~ am, data = mtcars)
#> Hedges' g |         95% CI
#> --------------------------
#> -1.44     | [-2.21, -0.65]
#> 
#> - Estimated using pooled SD.
glass_delta(mpg ~ am, data = mtcars)
#> Glass' delta |         95% CI
#> -----------------------------
#> -1.17        | [-1.93, -0.39]


# One Sample ----------

cohens_d(wt ~ 1, data = mtcars)
#> Cohen's d |       95% CI
#> ------------------------
#> 3.29      | [2.44, 4.24]

# same as:
# cohens_d("wt", data = mtcars)
# cohens_d(mtcars$wt)

# More options:
cohens_d(wt ~ 1, data = mtcars, mu = 3)
#> Cohen's d |        95% CI
#> -------------------------
#> 0.22      | [-0.13, 0.58]
#> 
#> - Deviation from a difference of 3.
hedges_g(wt ~ 1, data = mtcars, mu = 3)
#> Hedges' g |        95% CI
#> -------------------------
#> 0.22      | [-0.13, 0.57]
#> 
#> - Deviation from a difference of 3.


# Paired Samples ----------

data(sleep)

cohens_d(Pair(extra[group == 1], extra[group == 2]) ~ 1, data = sleep)
#> Cohen's d |         95% CI
#> --------------------------
#> -1.28     | [-2.23, -0.44]

# same as:
# cohens_d(sleep$extra[sleep$group == 1], sleep$extra[sleep$group == 2], paired = TRUE)

# More options:
cohens_d(Pair(extra[group == 1], extra[group == 2]) ~ 1, data = sleep, mu = -1)
#> Cohen's d |        95% CI
#> -------------------------
#> -0.47     | [-1.18, 0.21]
#> 
#> - Deviation from a difference of -1.
hedges_g(Pair(extra[group == 1], extra[group == 2]) ~ 1, data = sleep)
#> Hedges' g |         95% CI
#> --------------------------
#> -1.17     | [-2.04, -0.40]


# Interpretation -----------------------
interpret_cohens_d(-1.48, rules = "cohen1988")
#> [1] "large"
#> (Rules: cohen1988)
#> 
interpret_hedges_g(-1.48, rules = "sawilowsky2009")
#> [1] "very large"
#> (Rules: sawilowsky2009)
#> 
interpret_glass_delta(-1.48, rules = "gignac2016")
#> [1] "large"
#> (Rules: gignac2016)
#> 
# Or:
interpret(d, rules = "sawilowsky2009")
#> Cohen's d |         95% CI | Interpretation
#> -------------------------------------------
#> -1.48     | [-2.27, -0.67] |     very large
#> 
#> - Estimated using pooled SD.
#> - Interpretation rule: sawilowsky2009

# Common Language Effect Sizes
d_to_cles(1.48)
#> $`Pr(superiority)`
#> [1] 0.852339
#> 
#> $`Cohen's U3`
#> [1] 0.9305634
#> 
#> $Overlap
#> [1] 0.4593
#> 
# Or:
print(d, append_CLES = TRUE)
#> Cohen's d |         95% CI
#> --------------------------
#> -1.48     | [-2.27, -0.67]
#> 
#> - Estimated using pooled SD.
#> 
#> ## Common Language Effect Sizes:
#> Parameter       | Coefficient |       95% CI
#> --------------------------------------------
#> Pr(superiority) |        0.15 | [0.05, 0.32]
#> Cohen's U3      |        0.07 | [0.01, 0.25]
#> Overlap         |        0.46 | [0.26, 0.74]
# }