Law of large numbers and Central limit theorem

大数定律 Law of large numbers (LLN)

虽然名字是 Law，但其实是严格证明过的 Theorem

• weak law of large number (Khinchin's law)

The weak law of large numbers: the sample average converges in probability to the expected value

$\bar{X_n}=\frac{1}{n}(X_1+ \cdots +X_n) \overset{p}{\to} E\{X\} $

• strong law of large number (proved by Kolmogorov in 1930)

The strong law of large numbers: the sample average converges almost surely to the expected value

$\bar{X_n}=\frac{1}{n}(X_1+ \cdots +X_n) \overset{a.s.}{\to} E\{X\} $

https://en.wikipedia.org/wiki/Law_of_large_numbers

https://terrytao.wordpress.com/2008/06/18/the-strong-law-of-large-numbers/

中心极限定理 Central Limit Theorem (CLT)

https://en.wikipedia.org/wiki/Central_limit_theorem

切比雪夫不等式 (Chebyshev's Inequality)

Let $X$ be a random variable with finite expected value $\mu$ and finit non-zero variance $\sigma^2$, then for any real number $k>0$,

$ \mathrm{Pr} \left( \left|X-\mu\right| \geq k \right) \leq \frac{\sigma^2}{k^2}$

马尔科夫不等式 (Markov's inequality)

If X is a nonnegative random variable and a > 0, then the probability that X is at least a is at most the expectation of X divided by a

$ \mathrm{Pr} \left( X \geq a \right) \leq \frac{\mu}{a} $

切尔诺夫限 (Chernoff bound)

The generic Chernoff bound for a random variable X is attained by applying Markov's inequality to e^tX. For every t > 0:

$ \mathrm{Pr} \left( X \geq a \right)=\mathrm{Pr} \left( e^{tX} \geq e^{ta} \right) \leq \frac{E[e^{tX}]}{e^{ta}} $