Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

>In actuality you will find that many null hypotheses hold true

I'm assuming you mean within the confines of the experiment, correct? I agree. The tweet author was eluding to the fact that "IRL" the null hypothesis is almost never true at the population level. Meaning if you grab a large enough sample you will detect very, very small differences. (This was her Lucky Charms ~ blood type example in the tweet). I also agree with that. I don't think the two claims are mutually exclusive and the fact they can coexist is (I believe) precisely her point about sample size.



I mean there are many real world examples where the impact of A has no effect on B, obviously. The number I'm thinking of, has, truly, no effect on the time since you last blinked. No sample size will change that.

Lucky Charms does probably relate to blood type in some impossibly small way. It makes sense that a biological trait has some relation to dietary consumption. I don't think any sort of non-garbage tier journal with peer review would publish it, but good on p-values for helping us detect them though. Not bad on sample size for making it possible to discern this effect size with a high degree of confidence.

It's worth noting that we also have many other tools to help us. For example you can test, given an expected effect size and a sample size, what the probability is of getting a statistically significant result, or a non-significant result, or a significant result that erroneously goes in the wrong direction. Or what the range of likely true mean effect size is given a significant sample difference.

We want large samples. They enhance confidence in findings. The author's premise seems to be it's better not to know that small rocks exist if we're only looking for big rocks. But fails to mention that the tools to find small rocks also help us identify big rocks with more clarity.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: