For my demonstration, let’s assume we got 2 groups, both consisting of 250 people. Those 2 groups consist of one intervention group and one control group. The intervention group received a special training to improve their reaction times while the control group did not. We don’t quite know whether or not the training actually might have deteriorated the performance of the people because it wore them out too much, so we are testing two-tailed. This simply means that we can’t be quite sure which direction the results will be in.
As you can see, it took us times to create significance with absolutely random data. If you click a few times more you will see that sometimes you have significant results on the first try – and you could publish a study about that. But what if it’s not on the first try?
But you were sure
There is also another possibility: Suppose you have a study that you invested a lot of time into. Now you find out that your study was actually useless – no difference is found and no journal will publish it. But you were sure there had to be an effect. You experience cognitive dissonance. So you can either (1) admit you were wrong or (2) try to get more respondents. Since it is hard to admit you were wrong, you conclude it must have been the sample because you see a “tendency” in the data and you just need a bit more respondents.
So you set out to get some more respondents, and re-analyse the data. You see what you just did? You gave coincidence another chance to hit at your sample. And if that fails – maybe another try, just a few more?
Next time you see a great study where you think “this effect can’t be”, maybe think about this example – if you re-collect data ten times, will you once be able to achieve significance? Of course, it should also be noted that by far not all studies are flawed and tinkered with – but yes, a lot of them are.