Autoregressive (AR) and Moving Average (MA) model

Autoregressive(AR) and moving average(MA) model

A process $\{X_t\}$ is said to be an ARMA(p,q) process if

• $\{X_t\}$ is stationary
• $\forall t. X_t - \phi_1X_{t-1}-...-\phi_qX_{t-p} = a_t + \theta_1a_{t-1}+...+\theta_qa_{t-q}$
  using backward shift operation notation $B^h=x_{t-h}$:
  $\Phi(B)x_t = (1-\phi_1B - ... - \phi_p B^p)x_t = (1+\theta_1B + ...+\theta_qB^q)a_t = \Theta(B)a_t$ where $a_t \sim NID(0, \sigma^2)$

$\{X_t\}$ is an ARMA(p,q) process with mean $\mu$ if $\{X_t-\mu\}$ is an ARMA(p,q) process.

Moving average model(MA(q))

MA($\infty$) If $\{a_t\}\sim NID(0, \sigma^2)$ then we say that $\{X_t\}$ is a MA($\infty$) process of $\{a_t\}$ if $\exists\{\psi_n\}, \sum^\infty |\psi_j|<\infty$ and $X_t = \sum^\infty \psi_j a_{t-j}$ where $t\in\mathbb{Z}$.

We can calculate ACF of a stochastic process $\{X_t\}$ a.l.s. $\{X_t\}$ can be writtin in the form of a MA($\infty$) process

Also, MA($\infty$) is a required condition for $\{X_t\}$ to be stationary.

Theorem The MA($\infty$) process is stationary with 0 mean and autocovariance function $\gamma(k) = \sigma^2 \sum^\infty \psi_j\psi_{j+|k|}$

MA(q) $X_t = \sum_{i=0}^q \theta_i a_{t-i} = \Theta(B)a_t$ $\theta_0 = 1, B$ is the backward shift operator, $B^hX_t = X_{t-h}$ and $a_t\sim NID(0, \sigma^2)$

Under MA(q) model

$$\begin{align*} \gamma(1) = cov(X_t, X_{t+1})&=cov(\sum_{i=0}^q\theta_i a_{t-i}, \sum_{i=0}^q \theta_ia_{t+1-i})\\ &=E(\sum_{i=0}^{q-1}\theta_i\theta_{i+1}a_{t-i}a_{t-i}) &a\sim NID, cov(a_i,a_j) =0\\ &=\sigma^2(\sum_{i=0}^{q-1}\theta_i\theta_{i+1}) \end{align*}$$

Similarly,

$$\begin{align*} \gamma(k)=cov(X_t, X_{t+k}) &=cov(\sum_{i=0}^q \theta_t a_{t-i}, \sum_{i=0}^q \theta_i a_{t+k - i})\\ &=\sigma^2 \sum_{i=0}^{q-k}\theta_i\theta_{i+k}\mathbb{I}(|k|\leq q) \end{align*}$$

Then, the autocorrelation function (ACF) will be

$$\begin{align*} \rho_k &= \gamma_k/\sqrt{var(X_t)var(X_{t+k})}\\ & = \gamma_k / \sigma^2\sum_{i=0}^{q} \theta_i^2\\ & =\sigma^2 \sum_{i=0}^{q-k}\theta_i\theta_{i+k}\mathbb{I}(|k|\leq q) / \sigma^2\sum_{i=0}^{q} \theta_i^2\\ & = \sum_{i=0}^{q-k}\theta_i\theta_{i+k}\mathbb{I}(k\leq q) / \sum_{i=0}^{q} \theta_i^2 \end{align*}$$

Autoregressive model of order p (AR(p))

$X_t - \phi_1X_{t-1}-...-\phi_pX_{t-p} = \Phi(B)X_t = a_t$
where $a_t\sim NID(0, \sigma^2), B^hX_t = X_{t-h}, h\in\mathbb{Z}, \Phi(B)=(1-\phi_1B-...-\phi_p B^p)$

AR(1)

Notice that for a $AR(1)$ process, $a\sim NID(0, \sigma^2)$ and $a_t$ is uncorrelated with all previous $X_s, s<t$

$$\begin{align*} X_t &= \phi X_{t-1} + a_t\\ &=\phi(\phi X_{t-2}+a_{t-1})+a_t&\text{ replace }X_{t-1}\\ &...&\text{repeated replacing}\\ &=\sum_0^\infty \phi^i a_{t-i} \end{align*}$$

is a $MA(\infty)$ process

$$\begin{align*} \gamma(k) &= cov(X_t, X_{t+k})\\ &=cov(\sum_0^\infty \phi^i a_{t-i}, \sum_0^\infty \phi^i a_{t+k-i})\\ &=cov(\sum_0^\infty \phi^i a_{t-i}, \sum_0^\infty \phi^{i+k} a_{t-i} + \sum_0^{k-1} \phi^i a_{t+k-i})\\ &= \phi^k\sum_0^\infty (\phi^ia_{t-i})^2\\ &=\phi^k \gamma(0)=\phi^k var(X_t) \end{align*}$$

$$\begin{align*} \gamma(0) &=var(X_t)\\ &=\sum^\infty \phi^{2i}a^2_{t-i}\\ &=\sigma^2(\sum^\infty (\phi^2)^i) &a\sim NID(0,\sigma^2)\\ &=\sigma^2(1-\phi^2)^{-1} &\text{when }\phi^2<1 \text{, by Maclaurin's series} \end{align*}$$

Causal or future independent AR process when $|\phi|< 1$ for an $AR(1)$

Checking stationarity of AR(p)

$\Phi(B) = 1-\phi_1B-...-\phi_pB^p=0$ must have all the roots line outside the unit circle.

ACF

AR(1) Case

$$X_t = \phi X_{t-1} + a_t, a_t\sim NID(0,\sigma^2)$$

For $k\in\mathbb{Z}^+$, multiply $X_{t-k}$ on both sides

$$X_t X_{t-k} = \phi X_{t-1}X_{t-k} + a_t X_{t-k}$$

Taking expectation, consider $E(a_tX_{t-k})$

\begin{align} cov(a_t, X_{t-k}) &= E(a_t X_{t-k})-E(a_t)E(X_{t-k})\ &= E(a_t X_{t-k}) - 0\ &= cov(a_t, \sum_0^\infty \phi^i a_{t-k-i}) = 0 \end{align}

$a_t$ is uncorrelated with previous $a$'s.

$$E(X_t X_{t-k}) = \phi E(X_{t-1}X_{t-k})$$

since $cov(X_t,X_{t-k}) = E(X_tX_{t-k})-0$

$$\gamma(k)=\phi\gamma(k-1)$$

By induction, $\gamma(k)=\phi^k\gamma(0)$

AR(2) Case

$$X_t = \phi_1 X_{t-1} + \phi_2 X_{t-2} + a_t$$

Multiple both sides by $X_t$

$$X_t^2 = \phi_1 X_{t-1}X_t + \phi_2 X_{t-2}X_t + X_t a_t$$

Taking expectation, note that $X_t$ is a lin.comb of $a$.

$$\gamma(0) = \gamma(1) + \gamma(2) + \sigma^2$$

$$\gamma(0)(1-\phi_1\rho(1)-\phi_2\rho(2)) = \sigma^2 \text{ since }\rho(k)=\gamma(k)/\gamma(0)$$

Multiple both sides by $X_{t-1}$ and take expectations

$$E(X_tX_{t-1}) = \phi_1 E(X_{t-1}X_{t-1}) + \phi_2E(X_{t-2}X_{t-1}) + E(a_t X_{t-1})$$

$$\gamma(1) = \phi_1\gamma(0) + \phi_2\gamma(1)$$

$$\rho(1) = \phi_1 + \phi_2\rho(1)$$

$$\rho(1) = \frac{\phi_1}{1-\phi_2}$$

Multiple both sides by $X_{t-2}$ and take expectations

$$E(X_tX_{t-2}) = \phi_1 E(X_{t-1}X_{t-2}) + \phi_2E(X_{t-2}X_{t-2}) + E(a_t X_{t-2})$$

$$\gamma(2) = \phi_1\gamma(1) + \phi_2\gamma(0)$$

$$\rho(2) = \phi_1\rho(1) + \phi_2$$

... Using this pattern

$$\rho(h) = \phi_1\rho(h-1)+\phi_2\rho(h-2)$$

with base case

$$\rho(0)=1, \rho(1) = \frac{\phi_1}{1-\phi_2}$$

AR(p) case

Given $X_t = (\sum_1^p \phi_iX_{t-i}) + a_t$, is stationary is all $p$ roots lie outside of the unit circle

Yule-Walker equations
For the first $p$ autocorrelations:

$$\rho(k) = \sum_1^p \phi_i\rho_{|k-i|}$$

Partial Autocorrelation Function (PACF)

$\phi_{kk} = corr(X_t, X_{t+k}\mid X_{t+1},...,X_{t+k-1})$
the correlation between $X_t, X_{t+k}$ after their mutual linear dependency on the intervening variables has been removed.

For a given lag $k$, $\forall j \in \{1,2,...,k\}$.

$$\rho_i = \sum_1^k\phi_{ki}\rho_{j-i}$$

We regard the ACFs are given, take regression parameters $\phi_{ki}$ and wish to solve for $\phi_{kk}$.
which all together forms the Yule-Walker equations.

Example
For lag 1, $\rho_1 = \phi_{11},\rho_0\Rightarrow \rho_1=\phi_{11}$

For lag 2,

$$\rho_1 = \phi_{21} + \phi_{22}\rho_1$$

$$\rho_2 = \phi_{21}\rho_1 + \phi_{22}$$

$$\Rightarrow \phi_{22} = \frac{\rho_2 - \rho_1^2}{1-\rho_1^2}$$

Causal and invertible

Causal/stationary if $X_t$ can be expressed as an MA($\infty$) process

Invertible if $X_t$ can be expressed as an AR($\infty$) process.

Duality between AR amd MA processes

A finite-order stationary AR(p) process corresponds to a MA($\infty$) process, and a finite-order invertible MA(q) corresponds to an AR($\infty$) process.

Example

Given model $X_t - \phi_1 X_{t-1} - \phi_2 X_{t-2} = a_t = \theta a_{t-1}$

Assume the process is causal, then $X_t = \sum_0^\infty \psi_i a_{t-i} = a_t\sum_0^\infty \psi_i B^i = \psi(B)a_t$ by causal process
$\phi(B)X_t = \theta(B) a_t \Rightarrow X_t = \frac{\theta(B)a_t}{\phi(B)}$ by ARMA model
$\Rightarrow \Theta(B)/\Phi(B)=\Psi(B)$

Replace back into the model $1+\theta B = (\sum_0^\infty \psi_iB^i)(1-\phi_1B - \phi_2B^2)$

Consider $B$, $\theta B = \psi_1B -\phi_1B\Rightarrow \psi_1 = \phi_1 + \theta$

Consider $B^2$, $0 = -\theta_2B^2-\psi_1\theta_1B +\psi_2B^2\Rightarrow \psi_2 = \phi_2 + \phi_1(\phi_1+ \theta)$

Assume the process is invertible, then
$a_t = \sum_0^\infty \pi_i X_{t-i} = X_t\sum_0^\infty \pi_i B^i$,
similarly we get $\Phi(B)=\Theta(B)\Pi(B)$

Wold Decomposition

Any zero-mean process $\{X_t\}$ wgucg us bit deterministic can be expressed as a sum of $X_t = U_t + V_t$ where $\{U_t\}$ denotes an MA($\infty$) process and $\{V_t\}$ is a deterministic process which is uncorrelated with $\{U_t\}$ - deterministic if the values $X_{n+j}, j\geq 1$ of the process $\{X_t\}$ were perfectly predicatable in term of $\mu_n=sp\{X_t\}$ - If $X_n$ comes from a deterministic process, it can be predicted (or determined) by its past observations of the process

Model identification

process	ACF	PACF
AR(p)	tails off	cuts off after lag p
MA(q)	cuts off after lag q	tails off
ARMA(p,q)	tails off after (q-p)	tails off after (p-q)

Model Adequacy

The overall tests that check an entire group of residual autocorrelation functions are called portmanteau tests.

Box and Pierce $Q = n \sum_1^m \hat\rho_k^2 \sim \chi^2_{m-(p+q)}$
Ljung and Box $Q=\sum_1^m \frac{n(n+2)\hat\rho_k^2}{n-k}\sim \chi^2_{m-(p+q)}$
$n$ is the number of observations
$m$ is the max lag
$p,q$ are fitted model

Model selection

$$AIC = -2\log ML + 2k$$

$$BIC = -2 \log ML + k \log n$$

BIC puts more penalties on the number of parameters

Example: Application of ARMA in Investment

Alternative assets modeling

$y_t$ and $r_t$ denote observable appraisal and latent economic returns.

Goal to infer unobservable economic returns using appraisal returns

Geltner method commercial real state

$$y_t = \phi y_{t-1} + (1-\phi)r_t = \sum^\infty \phi^j(1-\phi)r_{t-j} = \sum^\infty w_jr_{t-j}$$

(by substitute $y_{t-1}$) where $\phi\in (0,1), w_j := \phi^j (1-\phi)$ is the weight

$$y_t = \hat\phi y_{t-1}+\hat a_t , \hat r_t = \frac{\hat a_t}{1-\hat\phi}$$

$$var(\hat r_t)=\frac{\sigma^2}{(1-\hat\phi)^2}$$

Gertmansky, Low, & Markorov

$$y_t = \sum^q w_i r_{t-i}$$

where $w_i\in(0,1), \sum w_i = 1$
Since $y_t$ is a linear combination of white noise

$$y_t = \sum^q \theta_i a_{t-i} = \sum_i^q \left(\frac{\theta_i}{\sum_j^q \theta_j}\sum_j^q \theta_j a_{t-i}\right) = \sum^q w_i r_{t-i}$$

Factor Modeling
The economic returns can be regressed by the market returns

$$r_t = \alpha + \beta r_{Mt} + e_t$$

$$y_t = \sum^q w_i (\alpha + \beta r_{M_,t-i} + e_{t-i})$$

$$= \sum^q w_i a + \beta \sum^q w_i r_{M,t-i} + \sum^q w_i e_{t-i} =\alpha + \beta \sum^q w_i r_{M,t-i} + \sum^q w_i e_{t-i}$$