October 16, 2018 at 4:12 pm #23535
Can somebody confirm for me exactly what the 95% CI represents for group comparison results based on permutation testing in Braph? My assumption is that 95% of the randomly generated group differences from the permutation tests fall between the upper and lower CI values.
For example: the actual difference in the strength of a given node between two groups is 2.771 and the 95% CI output by Braph is -1.626 to 1.608. Does this indicate that for 95% of 1000 permutations of the data, the difference between groups was somewhere between -1.626 and 1.608? And since the actual group difference is outside that range, it is significant?
Apologies if an explanation for this is provided on somewhere on the site–I searched but couldn’t find it. Thanks for your help!!
JeffNovember 2, 2018 at 7:56 am #23867
Your assumption is one way of looking at the problem. In particular, the confidence intervals are considered to consist of all set of parameters (in our case difference between the two group measures) for which the null hypothesis cannot be rejected. The null hypothesis is that there is no association between the two groups and that the observed difference is due to randomness. As a result, if the actual difference between the groups’ means fall within the confidence interval it is considered non-significant.
The whole process can be summarized as follows: the permutation test results in a sampling population of random values under the assumption that the null hypothesis is true. Then, based on your actual difference and this sampling population, a p-value at confidence level α can be computed. Finally, this can be used to compute a nominal confidence interval of 1 – α.
Therefore, in your example, since the actual difference is outside of the confidence interval it follows that the null hypothesis can be rejected, in other words this difference can be considered significant.
Hope this helps. If you have any more questions, please do not hesitate to ask.