Sufficient Statistic (充分统计量)

定义
充分统计量的判定
最小统计量
例子
- Poisson
- Normal
- 指数分布
- Gamma

Sufficient statistic - Wikipedia

Sufficient statistic - arizona

定义

统计量是一些随机样本$X_1, X_2, \cdots, X_n$的函数

\[T = r(X_1, X_2, \cdots, X_n).
\]

样本$X$的分布$f_{\theta}(X)=f(X;\theta)$由位置参数$\theta$决定, 通常我们通过极大似然估计

\[\max_{\theta} \quad P(X_1,X_2,\cdots, X_n ;\theta) = \prod_{i=1}^n P(X_i;\theta) = \prod_{i=1}^n f_{\theta}(X_i).
\]

而充分统计量是指这样的统计量:

\[P(\{X_i\}|T=t;\theta) = P(\{X_i\}|T=t),
\]

即在给定$T(X)=t$的情况下, $\{X_i\}$的条件联合分布与未知参数$\theta$无关.

Example: 考虑伯努利分布, 成功的概率为$p$, 失败的概率为$1-p$, 有$n$个独立同分布的样本$X_1, X_2,\cdots, X_n$, 则:

\[P(\{X_i\};p) = p^{\sum_i X_i}(1-p)^{n-\sum_i X_i},
\]

实际上(后面会讲到)$T=\sum_i^n X_i$为其一充分统计量. 实际上,

\[P(\{X_i\}|T=t;p) = \frac{P(\{X_i\}, T=t; p)}{P(T=t;p)} = \frac{\mathbb{I}[{\sum_{i}^nX_i=t]}\cdot p^t (1-p)^{n-t}}{C_n^t p^t (1-p)^{n-t}}=\frac{\mathbb{I}[\sum_i^n X_i = t]}{C_n^t}.
\]

显然与位置参数$p$无关.

充分统计量特别的意义, 比如上面提到的极大似然估计, 由于

\[P(\{X_i\};\theta) = P(\{X_i\}, T;\theta) = P(\{X_i\}|T;\theta) \:P(T;\theta) = P(\{X_i\}|T) \:P(T;\theta),
\]

由于$P(\{X_i\}|T)$与$\theta$无关, 所以最大化上式等价于

\[\max_{\theta} \quad P(T;\theta) = P(r(X_1, X_2,\cdots, X_n); \theta).
\]

特别地, 有时候标量$T$并不充分, 需要$T=(T_1, T_2,\cdots, T_k)$ 整体作为充分统计量, 比如当正态分布地$\mu, \sigma$均为未知参数的时候, $T=(\frac{1}{n}\sum_i X_i, \frac{1}{n-1}\sum_i (X_i - \bar{X})^2)$. 性质和上面的别无二致, 所以下面也不特别说明了.

当置于贝叶斯框架下时, 可以发现:

\[P(\theta|\{X_i\}) = \frac{P(\{X_i\}, \theta)}{P(\{X_i\})}
= \frac{P(\{X_i\}, T, \theta)}{P(\{X_i\}, T)}
= \frac{P(\{X_i\}| T, \theta) P(T|\theta)}{P(\{X_i\}, T)}
= \frac{P(\{X_i\}| T) P(T|\theta)}{P(\{X_i\}, T)}
= P(\theta|T).
\]

即给定$\{X_i\}$或者$T$, $\theta$的条件(后验)分布是一致的.

特别地, 我们可以用互信息来定义充分统计量, $T$为充分统计量, 当且仅当

\[I(\theta;X) = I(\theta;T(X)).
\]

注: 一般情况下$I(\theta;X) \ge I(\theta;T(X))$.

充分统计量的判定

用上面的标准来判断充分统计量是非常困难的一件事, 好在有Fisher-Neyman分离定理:

Factorization Theorem: $\{X_i\}$的联合密度函数为$f_{\theta}(X)$, 则$T$是关于$\theta$的充分统计量当且仅当存在非负函数$g, h$满足

\[f(X_1, X_2,\cdots, X_n; \theta) = h(X_1, X_2,\cdots, X_n) g(T; \theta).
\]

注: $T$可以是$T=(T_1, T_2,\cdots, T_k)$.

proof:

$\Rightarrow$

\[p(X_1,X_2,\cdots, X_n;\theta) = p(\{X_i\}|T;\theta) = p(\{X_i\}|T;\theta)p(T;\theta) = p(\{X_i\}|T)p(T;\theta)
\]

此时

\[g(T;\theta) = p(T;\theta), \\
h(X_1, X_2,\cdots, X_n) = p(\{X_i\}|T).
\]

$\Leftarrow$

为了符号简便, 令$X = \{X_1, X_2,\cdots, X_n\}$.

\[\begin{array}{ll}
p(T=t;\theta)
&= \int_{T(X)=t} p(X,T=t;\theta) \mathrm{d}X \\
&= \int_{T(X)=t} f(X;\theta) \mathrm{d}X \\
&= \int_{T(X)=t} h(X) g(T=t;\theta) \mathrm{d}X \\
&= \int_{T(X)=t} h(X) \mathrm{d}X \cdot g(T=t;\theta) \\
\end{array}.
\]

则

\[\begin{array}{ll}
p(X | T=t;\theta)
&= \frac{p(X,T=t;\theta)}{p(T=t;\theta)} \\
&= \frac{p(X;\theta)}{p(T=t;\theta)} \\
&= \frac{h(X)g(T=t;\theta)}{\int_{T(X)=t}h(X)\mathrm{d} X \cdot g(T=t;\theta)} \\
&= \frac{h(X)}{\int_{T(X)=t}h(X)}. \\
\end{array}
\]

与$\theta$无关.

注: 上述的证明存疑.

最小统计量

最小统计量S, 即

S是充分统计量;
充分统计量$T$, 存在$f$, 使得$S=f(T)$.

注: 若$T$是充分统计量, 则任意的可逆函数$f$得到的$f(T)$也是充分统计量.

例子

$U[0, \theta]$

均匀分布, 此时

\[p(X_1, X_2,\cdots, X_n;\theta) = \frac{1}{\theta^n} \mathbb{I}[0\le \min \{X_i\}] \cdot \mathbb{I}[\max \{X_i\} \le \theta],
\]

故

\[T = \max \{X_i\}, \: g(T;\theta) = \mathbb{I}[\max \{X_i\} \cdot \frac{1}{\theta^n}, \: h(X) = \mathbb{I}[0\le \min \{X_i\}].
\]

$U[\alpha, \beta]$

\[p(X_1, X_2,\cdots, X_n;\alpha,\beta) = \frac{1}{(\beta - \alpha)^n} \mathbb{I}[\alpha\le \min \{X_i\}] \cdot \mathbb{I}[\max \{X_i\} \le \theta],
\]

\[T = (\min \{X_i\}, \max \{X_i\}), \\
g(T;\alpha, \beta) = \frac{1}{(\beta - \alpha)^n} \mathbb{I}[\alpha\le \min \{X_i\}] \cdot \mathbb{I}[\max \{X_i\} \le \theta], \\
h(X) = 1.
\]

Poisson

\[P(X;\lambda) = \frac{\lambda^X e^{-\lambda}}{X!}.
\]

\[p(X_1, X_2,\cdots, X_n;\lambda) = e^{-n\lambda} \lambda^{\sum_{i}X_i} \cdot \frac{1}{\prod_i X_i!}.
\]

\[T = \sum_iX_i, \\
g(T;\theta) = e^{-n\lambda} \cdot \lambda^T, \\
h(X) = \frac{1}{\prod_{i} X_i!}.
\]

Normal

\[P(X;\mu,\sigma) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp(-\frac{(X-\mu)^2}{2\sigma^2}).
\]

\[p(X_1, X_2,\cdots, X_n;\mu, \sigma) = (2\pi\sigma^2)^{-\frac{n}{2}} \exp (-\frac{1}{2\sigma^2}\sum_{i=1}^n (X_i - \bar{X})^2) \exp(-\frac{n}{2\sigma^2})(\mu-\bar{X})^2.
\]

若$\sigma$已知:

\[T=\frac{1}{n}\sum X_i = \bar{X} , \\
g(T;\mu) = (2\pi\sigma^2)^{-\frac{n}{2}} \exp(-\frac{n}{2\sigma^2})(\mu-T)^2, \\
h(X) = \exp (-\frac{1}{2\sigma^2}\sum_{i=1}^n (X_i - \bar{X})^2).
\]

若$\sigma$未知:

\[T = (\bar{X}, s^2), s^2 = \frac{\sum_{i=1}^n(X_i-\bar{X})^2}{n-1}, \\
g(T;\mu,\sigma) = (2\pi\sigma^2)^{-\frac{n}{2}}\exp(-\frac{n-1}{2\sigma^2}s^2) \exp(-\frac{n}{2\sigma^2})(\mu-\bar{X})^2, \\
h(X) = 1.
\]

指数分布

\[p(X) = \frac{1}{\lambda} e^{-\frac{X}{\lambda}}, \quad X \ge 0.
\]

\[p(X_1, X_2,\cdots, X_n;\lambda) = \frac{1}{\lambda^n} e^{-\frac{\sum_{i=1}^n X_i}{\lambda}}.
\]

\[T = \sum_{i=1}^n X_i, \\
g(T;\lambda) = \frac{1}{\lambda^n} e^{-\frac{T}{\lambda}}, \\
h(X) = 1.
\]

Gamma

\[\Gamma(\alpha, \beta) = \frac{1}{\Gamma(\alpha) \beta^{\alpha}}X^{\alpha-1} e^{-\frac{X}{\beta}}.
\]

\[p(X_1, X_2,\cdots, X_n;\alpha, \beta) = \frac{1}{(\Gamma(\alpha) \beta^{\alpha})^n}(\prod_{i} X_i)^{\alpha-1} e^{-\frac{\sum_iX_i}{\beta}}.
\]

\[T = (\prod_i X_i, \sum_i X_i), \\
g(T;\theta) = \frac{1}{(\Gamma(\alpha) \beta^{\alpha})^n}(\prod_{i} X_i)^{\alpha-1} e^{-\frac{\sum_iX_i}{\beta}}, \\
h(X) = 1.
\]