Useful Probability facts

An event $A$ is a subset of the sample space $\Omega$, the set of all possible outcomes. We assign probabilities $P(A) \in [0,1]$ to events.

A random variable $X : \Omega \to \mathbb{R}$ is a function that gives a number to each outcome. It is discrete when it takes values in a countable set $\mathcal{X}$, and continuous when it takes values in $\mathbb{R}^d$. For a die, $\Omega = \{1,2,3,4,5,6\}$ and $X(\omega) = 1$ if $\omega$ is even, $0$ otherwise, maps outcomes to $\{0,1\}$.

The distribution of $X$ is described by one of three objects:

• PMF (probability mass function), discrete: $\;p_X(x) = \Pr(X = x)$.
• PDF (probability density function), continuous: $\;p_X(x)$ such that $\Pr(X \in A) = \int_A p_X(x)\, dx$.
• CDF (cumulative distribution function), both: $\;F_X(x) = \Pr(X \le x)$.

Either way the mass has to sum to one,

$$\sum_x p(x) = 1 \quad \text{(discrete)}, \qquad \int_{-\infty}^{\infty} p(x)\, dx = 1 \quad \text{(continuous)}.$$

With two variables the joint distribution $p(x,y)$ holds everything. From it we recover one variable by summing (or integrating) out the other, which is the marginal,

$$p(x) = \sum_y p(x,y) \qquad \text{or} \qquad p(x) = \int p(x,y)\, dy,$$

and we condition on a known value with the conditional,

$$p(x \mid y) = \frac{p(x,y)}{p(y)}, \qquad p(y) > 0.$$

Dividing by $p(y)$ makes $p(x \mid y)$ a full distribution in $x$ that integrates to 1:

$$\int p(x \mid y)\, dx = \frac{1}{p(y)} \int p(x,y)\, dx = \frac{p(y)}{p(y)} = 1.$$

Rearranging the conditional gives the product rule $p(x,y) = p(x \mid y)\,p(y) = p(y \mid x)\,p(x)$, and equating the two factorings gives Bayes' rule, which flips the conditioning direction:

$$p(x \mid y) = \frac{p(y \mid x)\,p(x)}{p(y)}, \qquad p(y) = \sum_x p(y \mid x)\,p(x).$$

Two variables are independent, written $X \perp Y$, when the joint factors into the marginals, which is the same as saying conditioning on $Y$ tells us nothing about $X$:

$$p(x,y) = p(x)\,p(y) \quad\Longleftrightarrow\quad p(x \mid y) = p(x).$$

They are conditionally independent given $Z$, written $X \perp Y \mid Z$, when the same factoring holds once $Z$ is fixed:

$$p(x,y \mid z) = p(x \mid z)\,p(y \mid z) \quad\Longleftrightarrow\quad p(x \mid y, z) = p(x \mid z).$$

The expectation of $X$ is its probability-weighted average,

$$\mathbb{E}[X] = \sum_x x\, p(x) \qquad \text{or} \qquad \mathbb{E}[X] = \int x\, p(x)\, dx,$$

and for any function $f$ we average $f$ against the same distribution,

$$\mathbb{E}[f(X)] = \sum_x f(x)\, p(x).$$

A subscript on the expectation says which distribution we are averaging over, so $\mathbb{E}_{X \sim p}[\,\cdot\,]$ or just $\mathbb{E}_X[\,\cdot\,]$ when $p$ is clear. This matters once several distributions are floating around. Expectation is linear, for constants $a, b$,

$$\mathbb{E}[a X + b Y] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y],$$

and this holds whether or not $X$ and $Y$ are independent.

The variance measures how far $X$ spreads around its mean $\mu = \mathbb{E}[X]$, as an average squared deviation,

$$\operatorname{Var}(X) = \mathbb{E}\!\big[(X - \mu)^2\big].$$

Expanding the square and using linearity gives the form we actually compute with,

$$\operatorname{Var}(X) = \mathbb{E}[X^2 - 2\mu X + \mu^2] = \mathbb{E}[X^2] - 2\mu\,\mathbb{E}[X] + \mu^2 = \mathbb{E}[X^2] - \mathbb{E}[X]^2.$$

A constant shift moves the mean along with $X$ and leaves the spread alone, while a scale factor pulls out squared, so

$$\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X).$$

The covariance is the same idea for two variables, the average product of their deviations,

$$\operatorname{Cov}(X, Y) = \mathbb{E}\!\big[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])\big] = \mathbb{E}[XY] - \mathbb{E}[X]\,\mathbb{E}[Y],$$

so $\operatorname{Var}(X) = \operatorname{Cov}(X, X)$. Expanding $\mathbb{E}[((X - \mathbb{E}X) + (Y - \mathbb{E}Y))^2]$ the same way gives the variance of a sum,

$$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y).$$

When $X$ and $Y$ are independent the cross term $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$ cancels, so $\operatorname{Cov}(X, Y) = 0$ and variances simply add. The converse is not true, zero covariance does not imply independence. Dividing the covariance by the two standard deviations rescales it into the correlation $\rho = \operatorname{Cov}(X,Y) / (\sigma_X \sigma_Y) \in [-1, 1]$.

The CLT is a powerful result in statistics: the average of a variable drawn from any distribution is itself Gaussian. Put simply, take many independent draws, average them, and treat that average as a variable: that variable is Gaussian no matter what you drew from. For $X_1, \dots, X_n$ independent with mean $\mu$ and variance $\sigma^2$, the sample mean $\bar{X} = \frac{1}{n}\sum_{i} X_i$ is the variable, and

$$\frac{\bar{X} - \mu}{\sigma / \sqrt{n}} \;\xrightarrow{\; n \to \infty \;}\; \mathcal{N}(0, 1).$$

The CLT is one of the main reasons Gaussians are so ubiquitous in nature.

For a convex function $\phi$ and any random variable $X$, the average of the function is at least the function of the average,

$$\phi\big(\mathbb{E}[X]\big) \;\le\; \mathbb{E}\big[\phi(X)\big].$$

If $\phi$ is concave the inequality goes the other way. It is an equality when $\phi$ is linear over the range of $X$, or when $X$ is constant.

Here is why it holds. A convex function lies above each of its tangent lines, so at any point $m$ there is a slope $g$ (a subgradient, the ordinary derivative $\phi'(m)$ when it exists) with

$$\phi(x) \;\ge\; \phi(m) + g\,(x - m) \qquad \text{for all } x.$$

Take $m = \mathbb{E}[X]$ and put $x = X$, so the bound holds for the random variable itself,

$$\phi(X) \;\ge\; \phi\big(\mathbb{E}[X]\big) + g\,\big(X - \mathbb{E}[X]\big).$$

Now take expectations of both sides. The right-hand side is linear in $X$, and $\mathbb{E}\big[X - \mathbb{E}[X]\big] = 0$, so the slope term vanishes and what remains is

$$\mathbb{E}\big[\phi(X)\big] \;\ge\; \phi\big(\mathbb{E}[X]\big).$$

This is used pretty much everywhere and follows directly from the definition of expectation. Start from the average of $f$ over the joint distribution, and split the joint with the product rule $p(x,y) = p(x \mid y)\,p(y)$:

$$\mathbb{E}_{X,Y}\!\big[f(X,Y)\big] = \sum_{x,y} f(x,y)\, p(x,y) = \sum_{y} p(y) \sum_{x} f(x,y)\, p(x \mid y).$$

The inner sum $\sum_x f(x,y)\, p(x \mid y)$ is by definition the conditional expectation $\mathbb{E}_{X \mid Y}[f(X,y) \mid Y = y]$, a function of $y$ alone. The outer sum then averages that over $p(y)$, which is an expectation over $Y$. Reassembled,

$$\mathbb{E}_{X,Y}\!\big[f(X,Y)\big] = \mathbb{E}_{Y}\!\left[\, \mathbb{E}_{X \mid Y}\!\big[\, f(X,Y) \mid Y\,\big] \,\right].$$

Reading the result inside-out:

• The inner expectation $\mathbb{E}_{X \mid Y}[\,\cdot \mid Y]$ averages over the conditional distribution of $X$ given $Y$, producing a function of $Y$ alone.
• The outer expectation $\mathbb{E}_{Y}[\,\cdot\,]$ averages that result over the marginal distribution of $Y$.

Written out, the outer average is just a sum (or integral) over $y$ weighted by how likely each $y$ is:

$$\mathbb{E}_{X,Y}\!\big[f(X,Y)\big] = \sum_y \mathbb{E}_{X \mid Y}\!\big[f(X,y) \mid Y = y\big]\, P_Y(Y = y),$$ $$\mathbb{E}_{X,Y}\!\big[f(X,Y)\big] = \int_{-\infty}^{\infty} \mathbb{E}_{X \mid Y}\!\big[f(X,y) \mid Y = y\big]\, p_Y(y)\, dy.$$

Suppose $\hat{\theta}$ is an estimate of some fixed quantity $\theta$, computed from random data so $\hat{\theta}$ is itself random. Two things can make it wrong. It can be off on average, and it can be noisy. The bias is the average error $\operatorname{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$, and the variance is the noise around that average.

Write the mean $m = \mathbb{E}[\hat{\theta}]$ and split the error into a noise part $\hat{\theta} - m$ and a bias part $m - \theta$:

$$\hat{\theta} - \theta = (\hat{\theta} - m) + (m - \theta).$$

Squaring gives three terms,

$$(\hat{\theta} - \theta)^2 = (\hat{\theta} - m)^2 + 2(\hat{\theta} - m)(m - \theta) + (m - \theta)^2.$$

Let's take the expectation of each. The first term is $\operatorname{Var}(\hat{\theta})$ by definition. In the middle term $m - \theta$ is a constant, so we can pull it out by linearity, leaving $2(m - \theta)\,\mathbb{E}[\hat{\theta} - m]$. That last factor is zero, since $\mathbb{E}[\hat{\theta} - m] = \mathbb{E}[\hat{\theta}] - m = m - m = 0$, so the cross term vanishes. The last term is already constant. What is left is

$$\mathbb{E}\!\big[(\hat{\theta} - \theta)^2\big] = \underbrace{\operatorname{Var}(\hat{\theta})}_{\text{noise}} + \underbrace{\big(\mathbb{E}[\hat{\theta}] - \theta\big)^2}_{\text{bias}^2}.$$

So the expected squared error is variance plus bias squared, which is a natural tradeoff. It works a bit like a waterbed: pushing one down tends to push the other up. You can think of it as a small Pareto problem, where whoever designs the estimator has to decide how much bias and how much variance they can tolerate.

For a squared-error prediction of the form

$$\mathcal{R}(f) = \mathbb{E}_{X,Y}\!\big[\, (Y - f(X))^2 \,\big],$$

the conditional mean $\mathbb{E}[Y \mid X = x]$ is the best predictor.

To see this, look at one input $x$ at a time. There the prediction is just a number, $c = f(x)$, and the question becomes: which number $c$ has the smallest expected squared distance to $Y$? Call $\mu = \mathbb{E}[Y \mid x]$ and expand the error around it:

$$(Y - c)^2 = (Y - \mu)^2 + 2(Y - \mu)(\mu - c) + (\mu - c)^2.$$

Take the expectation over $Y$ with $x$ held fixed. Then $\mu$ and $c$ are both just numbers, so the factor $\mu - c$ pulls out of the middle term, which vanishes because $\mathbb{E}[Y - \mu \mid x] = 0$, leaving

$$\mathbb{E}\!\big[(Y - c)^2 \mid x\big] = \mathbb{E}\!\big[(Y - \mu)^2 \mid x\big] + (\mu - c)^2.$$

Only the last term involves $c$, and it is smallest at zero, so the best number is $c = \mu$. The first term is the spread of $Y$ around its mean at $x$, the noise floor no prediction can remove. And since the full risk is just these per-$x$ risks averaged over $x$,

$$\mathcal{R}(f) = \mathbb{E}_X\!\Big[\, \mathbb{E}\!\big[(Y - f(X))^2 \mid X\big] \,\Big],$$

making the best choice at every $x$ minimizes the whole thing. The minimizer is the regression function

$$f^\star(x) = \mathbb{E}[Y \mid X = x].$$

A lot of the time we work with the log of a density, $\log p(x)$, instead of the density itself. It is the log of the PDF evaluated at a point $x$, so it is a number attached to that point, not the log of a sample. A few reasons it shows up everywhere:

• Products become sums. Independent variables have $p(x, y) = p(x)\,p(y)$, so $\log p(x,y) = \log p(x) + \log p(y)$. A sum is easier to differentiate, and it does not underflow the way a long product of small probabilities does.
• Many objectives are written with it: maximum likelihood maximizes $\sum \log p$, entropy is $\mathcal{H}(p) = -\mathbb{E}[\log p]$, the score function is $\nabla_\theta \log p_\theta$, and KL divergences are expectations of log-ratios.
• $\log$ is monotonic, so maximizing $\log p$ maximizes $p$. We get the easier algebra without moving the optimum.

For a scalar Gaussian the density is $\mathcal{N}(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\big(-(x-\mu)^2 / 2\sigma^2\big)$, so its log is a quadratic in $x$ plus a constant:

$$\log \mathcal{N}(x \mid \mu, \sigma^2) = -\frac{(x-\mu)^2}{2\sigma^2} - \log \sigma - \tfrac{1}{2}\log(2\pi).$$

It is an upside-down parabola: largest at $x = \mu$ and falling off quadratically away from it, with the $-\log\sigma - \tfrac{1}{2}\log(2\pi)$ term shifting the curve so the density integrates to one. For a diagonal Gaussian over $\mathbb{R}^d$ (independent coordinates) the log-density is just the sum over dimensions,

$$\log \mathcal{N}(x \mid \mu, \sigma^2) = -\sum_{i=1}^{d} \Big[ \frac{(x_i - \mu_i)^2}{2\sigma_i^2} + \log \sigma_i \Big] - \frac{d}{2}\log(2\pi).$$

We often need the gradient of an expectation whose distribution itself depends on the parameter,

$$\nabla_\theta\, \mathbb{E}_{x \sim p_\theta}\big[\, g(x) \,\big].$$

We cannot push the gradient inside, because the distribution $p_\theta$ we average over moves with $\theta$. Writing the expectation as an integral and differentiating, only $p_\theta$ depends on $\theta$:

$$\nabla_\theta\, \mathbb{E}_{x \sim p_\theta}\big[g(x)\big] = \nabla_\theta \int p_\theta(x)\, g(x)\, dx = \int \nabla_\theta p_\theta(x)\, g(x)\, dx.$$

The log-derivative trick $\nabla_\theta \log p_\theta(x) = \nabla_\theta p_\theta(x) / p_\theta(x)$ lets us put a $p_\theta(x)$ back in front, which turns the integral into an expectation again:

$$\nabla_\theta p_\theta(x) = p_\theta(x)\, \nabla_\theta \log p_\theta(x),$$ $$\nabla_\theta\, \mathbb{E}_{x \sim p_\theta}\big[g(x)\big] = \int p_\theta(x)\, \nabla_\theta \log p_\theta(x)\, g(x)\, dx = \mathbb{E}_{x \sim p_\theta}\big[\, g(x)\, \nabla_\theta \log p_\theta(x) \,\big].$$

This is the score function estimator, also called the likelihood-ratio estimator, and $\nabla_\theta \log p_\theta(x)$ is the score. It only needs to sample $x$ and to differentiate $\log p_\theta$, so it works even when $g$ is not differentiable in $x$, discrete $x$ included. The price is variance: it sees the value of $g$ at sampled points and the score, not the shape of $g$.

The baseline leaves the mean alone but does change the variance. Write the score as $s(x) = \nabla_\theta \log p_\theta(x)$. Since $\mathbb{E}[s] = 0$, the variance of a single term is

$$\mathrm{Var}\big[(g - b)\,s\big] = \mathbb{E}[g^2 s^2] - 2b\,\mathbb{E}[g\,s^2] + b^2\,\mathbb{E}[s^2] - \big(\mathbb{E}[g\,s]\big)^2,$$

a parabola in $b$ opening upward, since the leading coefficient $\mathbb{E}[s^2]$ is nonnegative (it is the Fisher information). Setting the derivative to zero, the variance-minimizing baseline is

$$b^\star = \frac{\mathbb{E}[g\,s^2]}{\mathbb{E}[s^2]},$$

a score-squared-weighted average of $g$, not the plain mean $\mathbb{E}[g]$. The two agree only when $g$ is uncorrelated with $s^2$. In practice people still use the mean of $g$ (or of the returns) as the baseline, since it is cheap and usually close enough, but it is not the true minimizer.

Here the goal is to differentiate through a random sample. The expectation $\mathbb{E}_{x \sim p_\theta}[g(x)]$ averages over a stochastic $x$ whose distribution depends on $\theta$, and we want its gradient in $\theta$. As with the score function, we cannot push $\nabla_\theta$ straight inside, because the randomness we average over moves with $\theta$. The reparametrization trick gets around this by moving the $\theta$-dependence out of the sampling step and into a deterministic function. For a Gaussian $p_\theta = \mathcal{N}(\mu_\theta, \sigma_\theta^2)$, write

$$x = \mu_\theta + \sigma_\theta\, \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I).$$

Now the randomness lives entirely in $\epsilon$, drawn from a fixed distribution with no $\theta$ in it, and $\theta$ only enters through the deterministic map $x = \mu_\theta + \sigma_\theta\, \epsilon$. Since the distribution we average over ($\epsilon$'s) no longer depends on $\theta$, the gradient passes straight through the expectation and onto the sample:

$$\nabla_\theta\, \mathbb{E}_{x \sim p_\theta}\big[\, g(x) \,\big] = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)}\big[\, \nabla_\theta\, g(\mu_\theta + \sigma_\theta\, \epsilon) \,\big].$$

With one noise sample per term this is a plain average of gradients. By the chain rule each term is $\nabla_x g \cdot \nabla_\theta(\mu_\theta + \sigma_\theta\, \epsilon)$, so it backpropagates through the sampled $x$. The one requirement is that $g$ be differentiable in $x$, so this works for continuous variables, not discrete ones.

If $u$ has density $p_u$ and we pass it through a smooth, monotone map $a = f(u)$, what is the density of $a$? The rule comes from conserving probability mass: the chance that $u$ falls in a small interval must equal the chance that $a$ falls in the image of that interval. Writing those two small intervals as $|du|$ and $|da|$,

$$p_a(a)\, |da| = p_u(u)\, |du|.$$

Dividing by $|da|$ gives the density of $a$,

$$p_a(a) = p_u(u) \left| \frac{du}{da} \right| = \frac{p_u(u)}{|f'(u)|}, \qquad u = f^{-1}(a),$$

using $da/du = f'(u)$ for the last step. The $|f'(u)|$ factor accounts for how much $f$ stretches or compresses the line near $u$: where $f$ is steep, the same mass is spread over a wider range of $a$, so the density there drops. In several dimensions $f'(u)$ becomes the Jacobian of $f$ and $|f'(u)|$ its absolute determinant.

Sometimes we want a distribution over a bounded interval, say $(-1, 1)$, while keeping the Gaussian's easy reparametrization. The squashed Gaussian samples a Gaussian and then pushes the sample through $\tanh$:

$$u \sim \mathcal{N}(\mu_\theta, \sigma_\theta^2), \qquad a = \tanh(u).$$

Since $\tanh$ maps $\mathbb{R} \to (-1, 1)$, every sample lands in range, and the map is smooth so the reparametrization trick still passes gradients through it. Clipping a Gaussian to the interval would instead give zero gradient wherever it was clipped, which is why the smooth squash is preferred.

Passing $u$ through the nonlinear $\tanh$ stretches the density by different amounts at different points, so we correct for it with the change of variables above. The derivative of $\tanh$ is $f'(u) = 1 - \tanh^2(u)$, which gives

$$p(a) = \frac{p(u)}{|f'(u)|} = \frac{\mathcal{N}(u \mid \mu_\theta, \sigma_\theta^2)}{1 - \tanh^2(u)},$$

where $1 - \tanh^2 u \ge 0$ so the absolute value drops. Taking logs turns the division into a subtraction, and for a multi-dimensional action with independent coordinates the per-coordinate corrections add up:

$$\log p(a) = \log \mathcal{N}(u \mid \mu_\theta, \sigma_\theta^2) - \sum_i \log\big(1 - \tanh^2(u_i)\big),$$

where implementations add a small constant inside the log for numerical stability. This is the density used whenever an entropy or log-probability term is needed.