Does Increasing Sample Size Inevitably Lead to Statistical Significance? Insights From a Monte Carlo Simulation Study for Medical Research Design.

Authors

Mengjie Li, Shuwen Yang, Yixuan Wang, Zhenglin Xin, Ying Sun, Yuanyuan Li, Lei Gao, Zhe Xu, Jingfeng Bi

Published in

Journal of evaluation in clinical practice. Volume 32. Issue 5. Pages e70520.

Abstract

This study addresses the common misconception that larger sample sizes inevitably produce smaller P-values and higher statistical power, using Monte Carlo simulations to examine the effect of sample size under scenarios with and without a true effect.
Monte Carlo simulations were performed with 1000 iterations for each of 11 sample sizes per group (n = 5-800). Two scenarios were simulated: (1) both groups drawn from the same normal distribution (μ = 4.5, σ = 0.5; null hypothesis H₀ true), and (2) groups drawn from different normal distributions (μ₁= 4.5, σ = 0.5; μ₂= 4.4, σ = 0.5; alternative hypothesis H₁ true). For each iteration, the absolute mean difference (|X̅₁ - X̅₂|), standard error (S_e), t-test p-value and the proportion of significant results (p < 0.05) were calculated. This proportion was interpreted as the Type I error rate or statistical power, depending on the scenario.
When a true effect existed, increasing sample size led to a marked rise in statistical power, with the proportion of significant results increasing from 5.4% to 98.1%. In contrast, when no true effect existed, p-values remained uniformly distributed across all sample sizes, and the frequency of p < 0.05 consistently approximated the nominal 5% significance level. Importantly, increasing sample size under the null hypothesis did not increase the likelihood of detecting significance.
Larger sample sizes do not create statistical significance in the absence of a true effect; they only enhance the detection of effects that genuinely exist. Study design and statistical interpretation should therefore be guided by effect size and clinical relevance rather than by sample size alone.

PMID:
42472374
Bibliographic data and abstract were imported from PubMed on 19 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.