Functions of Random Variable

Change of Variable

Theorem Let $X$ be a r.v. and $Y=g(X)$ where $g:\mathbb R\rightarrow\mathbb R$ is a funciton. Then

• $X$ is discrete,

$$\text{pmf}_Y(y) = \sum_{x: g(x) = y} \text{pmf}_X(x)$$

• $X$ is continuous and $g$ is an appropriate transformation,

$$\text{cdf}_Y(y) = \int_{x: g(x)\leq y}\text{pdf}_X(x)dx = P(\{x: g(x) \leq y\})$$

$$\text{pdf}_Y(y) = \frac{d}{dy} \text{cdf}_Y(y)$$

Theorem Let $F(x) = \text{cdf}_X(x)$, then $F(X)\sim uniform(0, 1)$

proof. Let $Y=F(X)$,

$$\text{cdf}_Y(y) = P(\{x:F(x) < y\}) = P(X \leq x) = F(x) = y$$

Change of Single Variable

Theorem (change of variable) Let $X$ be continuous r.v. and function $g$ be differentiable and injective. Then

$$\text{pdf}_Y(y) = \text{pdf}_X(g^{-1}(y))|\frac{d}{dy}g^{-1}(y)|$$

proof. wlog assume $g$ is increasing (as a appropriate transformation function), then \begin{align} \text{pdf}Y(y) &= \frac{d}{dy}\text{cdf}_Y(y)\ &= \frac{d}{dy}\int_X(x)dx\ &= \frac{d}{dy} \text{cdf}_X(g^{-1}(y)) \ &= \text{pdf}_X(g^{-1}(y))|\frac{d}{dy}g^{-1}(y)| \end{align}^{g^{-1}(y)} \text{pdf}

Change of Variables for Multivariate Functions

Theorem For discrete random variables $\mathbf X = (X_1,...,X_n)$, Let $\mathbf G:\mathbb R^n\rightarrow\mathbb R^m$ be the transformation s.t. $\mathbf Y = (Y_1,...,Y_m), \mathbf Y = \mathbf G(\mathbf X), Y_i = g_i(X_1, ..., X_n)$. Then

$$\text{pmf}_{\mathbf Y} = \sum_{\mathbf x: \mathbf G(\mathbf x) = \mathbf y}\text{pmf}_{\mathbf X}(\mathbf x)$$

random variables $X_1,...,X_n$ are said to be independent and identically distributed (iid.) if $X_i$'s are independent and have the same distribution.

Example

the sum of independent Bernoulli trails follows binomial distribution.

proof. Let $X_i \sim \text{Bern.}(p)$. $Y_n = \sum^n X_i$. We will prove by induction.

Obviously $Y_1 = X_1 \sim \text{Bern.}(p)\equiv \text{binomial}(1, p)$

Assume $Y_k \sim \text{binomial}(k, p)$. Then

$$\begin{align*} P(Y_{k+1} = 0) &= P(Y_k=0, X_{k+1} - 0) \\ &= P(Y_k = 0)P(X_{k+1}=0)\\ &= {k\choose 0}(1-p)^k (1-p) \\ &= {k+1\choose 0}(1-p)^{k+1}\\ P(Y_{k+1}=j) &= P((Y_k = j, X_{k+1} = 0)\cup (Y_k = j-1, X_{k+1} = 1))\\ &= P(Y_k = j)P(X_{k+1} = 0) + P(Y_k = j-1)P(X_{k+1} = 1)\\ &= {k\choose j}p^j(1-p)^{k-j}(1-p) + {k\choose j-1}p^{j-1}(1-p)^{k-(j-1)}p\\ &= {k+1\choose j}p^j(1-p)^{k+1-j}\\ Y_{k+1}&\sim \text{binomial} (k+1,p) \end{align*}$$

Theorem

If $X,Y$ are independent continuous r.v. then

$$\text{pdf}_{X+Y}(z) = \int \text{pdf}_X(x)\text{pdf}_Y(z-x)dx$$

proof. Let $Z=X+Y$

$$\begin{align*} \text{cdf}_Z(z) &= P(X+Y \leq z)\\ &= P(X \leq x, Y \leq z-x)\\ &=P(X \leq x)P(Y \leq z-x)\\ &= \int_{-\infty}^{\infty} \int_{-\infty}^{z-x}\text{pdf}_X(x)\text{pdf}_Y(y)dydx\\ &= \int_{-\infty}^{\infty} \text{pdf}_X(x) \text{cdf}_Y(z-x)dx\\ \text{pdf}_Z(z) &= \frac{d}{dz} \int_{-\infty}^{\infty} \text{pdf}_X(x) \text{cdf}_Y(z-x)dx\\ &= \int_{-\infty}^{\infty} \text{pdf}_X(x) \text{pdf}_Y(z-x)dx\\ \end{align*}$$

Example

For $X_1,...,X_n$ iid, $Y_n = \max(X_1,...,X_n), Y_1 = \min(X_1,...,X_n)$.

$$\begin{align*} \text{cdf}_{Y_n}(y) &= P(\max(X_1,...,X_n) \leq y) \\ &= P(X_1 \leq y,...,X_n\leq y)\\ &= \prod^n P(X_i \leq y) \\ &= \text{cdf}_X(y)^n\\ \text{cdf}_{Y_1}(y) &= 1 - P(Y_1 > y)\\ &= 1 - P(\min(X_1,...,X_n) > y) \\ &= 1 - P(X_1 > y,...,X_n > y)\\ &= 1 - \prod^n P(X_i > y) \\ &= 1 - (1-\text{cdf}_X(y))^n\\ \text{cdf}_{Y_1,Y_n}(y_1,y_2)&= P(Y_1\leq y_1, Y_2\leq y2)\\ &= P(y_2\leq y_2) - P(Y_1 > y_1, Y_2\leq y_2)\\ &= \text{cdf}_X(y_2)^n - \prod^n P(y_1 < X_i \leq y_2)\\ &= \text{cdf}_X(y_2)^n - (\text{cdf}_X(y_2) - \text{cdf}_X(y_1))^n \end{align*}$$

Theorem (change of variables)

For random variables $\mathbf X = (X_1,...,X_n)$, Let $\mathbf G:\mathbb R^n\rightarrow\mathbb R^m$ be the transformation s.t. $\mathbf Y = (Y_1,...,Y_m), \mathbf Y = \mathbf G(\mathbf X), Y_i = g_i(X_1, ..., X_n)$. IF $\mathbf G$ is injective and differentiable, then

$$\text{pdf}_{\mathbf Y}(\mathbf y) = \text{pdf}_{\mathbf X}(\mathbf G^{-1}(\mathbf y))|\det(\frac{D}{D\mathbf y}\mathbf G^{-1}(\mathbf y))|$$