Statistical Models¶

Assume that the data \(x_1,...,x_n\) are outcomes of r.v. \(X_1,...,X_n \sim F\), which assumes to be unknown.

A statistical model is a family \(\mathcal F\) of probability distributions of \((X_1,...,X_n)\).

Theoretically, \(F\in \mathcal F\) but in practice this is not always true, we are to find some \(F_0 \in \mathcal F\) close enough to \(F\) so that \(\mathcal F\) is useful.

Parametric models¶

For a given \(\mathcal F\), we can parametrize as \(\mathcal F = \{F_\theta:\theta \in \Theta\}\)

If \(\Theta \subset \mathbb R^p\) then \(\mathcal F\) is a parametric model and \(\theta \in \mathbb R^p = (\theta_1,...,\theta_p)\)

Non-parametric models¶

If \(\Theta\) is not finite dimensional then the model is said to be non-parametric (in this case, \(\in\mathbb R^\infty\))

Example \(g(x)\approx \sum^p \beta_k \phi_k(x)\) for some functions \(\phi_1,...,\phi_p\) and unknown parameters \(\beta_1,...,\beta_p\)

Semi-parametric models¶

Non-parametric models often have a finite dimensional parametric component.

Example \(Y_i = g(x_i) + \epsilon_i\) with \(\{\epsilon_i\}\) iid. \(N(0,\sigma^2)\) and \(g,\sigma^2\) are unknown

Example¶

Consider the linear regression \(Y_i = \beta_0 + \beta_1x_i + \epsilon_i\) for observations \((x_1,Y_1), ..., (x_n, Y_n)\) where \(\epsilon_i \sim N(0,\sigma^2)\) iid. Such model is parametric model.

However, if relax the assumption to \(E(\epsilon_i) = 0, E(\epsilon_i^2) = \sigma^2\), then this will be semi-parametric model.

Example¶

Let \(X_1,...,X_n\) be iid. Exponential r.v. representing survival times.

\[f(x;\lambda) = \lambda e^{-\lambda x}\mathbb I(x\geq 0)\]

\(\lambda >0\) is unknown

Let \(C_1,...,C_n\) be independent with unknown cdf \(G\) (or cdfs \(G_i\))

Observe \(Z_i = \min(X_i, C_i), \delta_i = \mathbb I(X_i\leq C_i)\)

parameters \(\lambda, G\) so that semi-parametric model.

Bayesian models¶

Assume a parametric model with \(\Theta \subset \mathbb R^p\), for each \(\theta \in \Theta\), think of the join cdf \(F_\theta\) as the conditional distribution of \(\mathcal X\) given \(\theta\).

Bayesian inference put a probability distribution on \(\Theta\), i.e. a prior.

After observing \(x_1,...,x_n\), we can use Bayes Theorem to obtain a posterior distribution of \(\theta\) given \(X_1 = x_1,...,X_n = x_n\)

Statistical Functionals¶

To estimate the characteristics of a model \(F\), we often consider \(\theta(F)\), i.e. a mapping \(\theta: \mathcal F\rightarrow \mathbb R\)

Examples¶

\(\theta(F) = \mathbb E_F(X_i)= \mathbb E_F(h(X_i))\)
\(\theta(F) = F^{-1}(\tau)\) quantiles
\(\theta(F) = \mathbb E_F\big[\frac{X_i}{\mu(F)}\ln(\frac{X_i}{\mu(F)})\big], P(X_i > 0 ) = 1, \mu(F) = \mathbb E_F(X_i)\) Theil index

Substitution principle¶

First estimate \(F\rightarrow \hat F\) and substitute \(\hat F\) into \(\theta (\hat F)\)
If \(\theta\) is continuous, Using continuous mapping theorem, \(\theta(\hat F) \approx \theta(F)\)

Example empirical distribution function (edf)

\[\hat F(x) = \frac{1}{n} \sum^n \mathbb I(X_i \leq x) = \text{proportion of observations } \leq x\]

Note that the edf is just a sample mean and WLLN, CLT holds, for each \(\mathbb I(X_i \leq x)\) is iid. Bernoulli

Therefore,

\(E(\hat F(x)) = F(x), var(\hat F(x)) = \frac{F(x)(1-F(x))}{n}\)
WLLN \(\hat F(x) = \hat F_n(x) \rightarrow^p F(x), \forall x\)
CLT \(\sqrt n(\hat F_n(x)-F(x))\rightarrow^f N(0, F(x)(1-F(x)))\)