p-value distribution for different effect-sizes

during a long experiment, there can be different sources of variability. for example, the participant can get sleepy, which influences measures like reaction times on a slower timescale. these slow drifts can then mask differences between experimental conditions. pic.twitter.com/kGiK78Yf02

"but the correlation is so large, it must be a robust effect" 🤔 for small samples, large values for the computed correlation coefficient are more likely to appear, even in the absence of any relationship. see here for 2 independently generated variables & different sample sizes: pic.twitter.com/bkUfOypGz2