Permutation test: p, CI, CI of P 置换检验相关统计量的计算

For research purpose, I've read a lot materials on permutation test issue. Here is a summary. Should be useful. Still, thanks for contributors online.

P value calculation

Because the actual value is one of those permutations, I would like to change the way I calculate p-value from: n/m to (n+1)/(m+1), then the p-value will not be 0. Besides, the latter way could be more powerful.

See:

P-values equal to 0 in permutation test (second answer)

Phipson, B., and Smyth, G. K. (2010). Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Stat. Appl. Genet. Molec. Biol. Volume 9, Issue 1, Article 39.

CI for P value

Because the p-value we obtained are actually an approximately to the actual p-value, thus usually a confidence interval could be calculated for p-value, using binomial proportion confidence interval as Vinh mentioned.

See:

Coull B A. Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions[J]. American Statistician, 1998, 52(2):119-126.

Ross T D. Accurate confidence intervals for binomial proportion and Poisson rate estimation[J]. Computers in Biology & Medicine, 2003, 33(6):509-531.

Confidence Interval ON a P-value

P-values equal to 0 in permutation test (first answer)

In a permutation test, how to get a confidence interval for the estimated p-value?

Understanding Binomial Confidence Intervals 二项分布的置信区间

CI for permutation test: inverted from p value

There is some people and literatures supporting the calculating CI by directly using the value in distribution located at p = 0.05, might be referred as 'consonance interval' or 'randomization confidence intervals' in some literature.

See:

Lamotte L R, Volaufova J. Prediction Intervals Via Consonance Intervals[J]. Journal of the Royal Statistical Society, 1999, 48(3):419-424.

Ernst M D. Permutation Methods: A Basis for Exact Inference[J]. Statistical Science, 2004, 19(4):676-685.

How do we create a confidence interval for the parameter of a permutation test? (first answer)

Confidence Interval and P Value uncertainty for Permutation Test

And it is somehow controversial, see:

Why it doesn’t make sense in general to form confidence intervals by inverting hypothesis tests