In scientific research, the p-value and statistical significance have been used as the primary criteria for evaluating results for many years. However, statistical significance only indicates whether the observed difference occurred by chance; it does not provide information about how large or how important that difference is. This is where the concept of effect size comes into play. Effect size is a quantitative measure that expresses the magnitude of the effect of one variable on another or the size of the difference between two groups in a standardized form.
The Distinction Between Statistical Significance and Effect Size
Statistical significance tests are highly sensitive to sample size (n). When working with a very large sample, even very small differences that have no practical equivalent or meaning can result in a statistically significant p < 0.05 (explained with scenarios later in the text). This allows the researcher to answer the question: “How important is the difference I found for the real world?” Today, many prestigious medical journals require the reporting of not only the p-value but also effect sizes and confidence intervals.
Types of Effect Size
Effect size calculations vary according to the type of statistical analysis performed. They are basically divided into three main families:
- Measures of Difference (d Family): Expresses the difference between the means of two groups in terms of standard deviation. The most common is the Cohen’s d coefficient. If the Cohen’s d value is 0.5, it means the mean of the experimental group is half a standard deviation above the control group.
- Measures of Association (r Family): Indicates the strength of the relationship between variables. The Pearson correlation coefficient (r) and r-squared (coefficient of determination), which is the proportion of explained variance, are in this group.
- Ratio Measures: These are values such as Odds Ratio (OR) and Risk Ratio (RR), used especially in medicine and epidemiology.
While these values are not rigid rules, the concepts of “small” or “large” can change according to the field of research (e.g., education, psychology, or surgery). To evaluate the relevant effect size, the primary hypothesis should be determined in the relevant field, and the effect size should be decided according to that predicted analysis.
Application Examples
Scenario 1: Consider a pharmaceutical company’s study on a new painkiller. In a sample of 10,000 people, the new drug reduced pain by 2 more points out of 100, and p < 0.001 was found. The result is statistically significant. However, if Cohen’s d is calculated and a very low value like 0.05 is found, it is understood that this difference has no clinical significance and the drug is not practically superior to old drugs.
Scenario 2: When examining the effect of an educational method on student achievement, the p-value may have come out as 0.08 (insignificant) because the number of groups is small (n=30). However, if the effect size is d=0.70 (large effect), the researcher realizes that this method is potentially very effective but failed to reach the significance threshold due to insufficient sample size. This indicates that the study should be repeated with a larger group.
Scenario 3: Consider a study of 10,000 people conducted by a pharmaceutical company in patients using an anti-hypertensive drug. Suppose the difference between the post-treatment systolic blood pressure values of the samples using drug A (routine) and drug B (new) is only 2 mmHg (e.g., 132 mmHg vs. 130 mmHg) and a statistically significant difference is noted (p = 0.03). Although the results are statistically significant, can it be concluded that the new drug has meaningful effects in real life and that it makes sense to market it? The answer is no. In conclusion, while even minimal differences can yield statistically significant results in studies with very high sample sizes, the adaptability of these differences to daily life is directly related to the concept of “effect size.” In this example, there is likely a low effect size (d).
Relationship Between Power Analysis and Effect Size
Effect size is of vital importance in the research planning stage (prospective power analysis). Before starting a study, a researcher must determine how much of an effect they want to capture and calculate how many subjects they need to recruit to detect this effect with 80% or 90% power. As the expected effect size decreases, the required sample size increases geometrically.
Conclusion
Effect size is the bridge that allows us to understand the practical value of scientific findings. Focusing only on the p-value can lead to a “significance illusion” in the scientific literature. Reporting standardized effect sizes by researchers both provides data for meta-analysis studies and allows for the correct evaluation of knowledge in a cumulative manner.
References
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Routledge. https://doi.org/10.4324/9780203771587
- Sullivan GM, Feinn R. Using Effect Size-or Why the P Value Is Not Enough. J Grad Med Educ. 2012 Sep;4(3):279-82. https://doi.org/10.4300/jgme-d-12-00156.1 PMID: 23997866; PMCID: PMC3444174.
- Sawilowsky, S. S. (2009). New effect size rules of thumb. Journal of Modern Applied Statistical Methods, 8(2), 597 – 599.
