Discrete Random Variables II
This page is required preparation for Class 3. It covers the material in lecture videos M1 to M4: expected value, variance, conditional expectation, covariance and correlation, and sums of random variables. It works through each idea step by step, with calculations of the kind you will need for problem sets and exams. It is a lightly edited version of my lectures for Econ 103, the introductory statistics course I taught at the University of Pennsylvania. Where a recording exists, each section links to the point in the video where it begins, so you can read, watch, or both.
This page builds on Discrete Random Variables I. That page covers the probability mass function, the Bernoulli and binomial random variables, and joint, marginal, and conditional distributions. Its sections on joint, marginal, and conditional distributions were required for Class 2, and you will need them here. In particular, the sections on conditional expectation and covariance use the joint pmf from its section on joint versus marginal pmfs and the conditional pmfs from its section on conditional pmfs.
Notation
My lectures and the slides in the course lecture videos use different symbols for the same objects. I write \(p(x)\) for the probability mass function (pmf) of a discrete random variable \(X\), \(E\) for expected value, and \(p_{XY}(x,y)\), \(p_X(x)\), and \(p_{Y|X}(y|x)\) for joint, marginal, and conditional pmfs. The course lecture slides write \(f(x)\), \(f(x,y)\), \(f(x)\) and \(f(y)\), and \(f(y|x)\) for these, and \(\mathbb{E}\) for expected value.
Expected value
There is no video for this section. It follows the slides of my Econ 103 Lecture 8.
If the realizations of the coin flip random variable were payoffs, \(\$0\) for tails and \(\$1\) for heads, how much would you expect to win per play, on average, in a long sequence of plays? The expected value of a discrete random variable \(X\) is given by \[ E[X] = \sum_{\text{all } x} x \cdot p(x) \] In other words, the expected value of a discrete random variable is the probability-weighted average of its realizations. We sometimes write \(\mu\) as shorthand for \(E[X]\).
For a Bernoulli random variable, which takes the value 0 with probability \(1 - p\) and the value 1 with probability \(p\), \[ E[X] = \sum_{\text{all } x} x \cdot p(x) = 0 \cdot (1-p) + 1 \cdot p = p \] As another example, let \(X\) be a random variable with support set \(\{1, 2, 3\}\), where \(p(1) = p(2) = p(3) = 1/3\). Then \[ E[X] = \sum_{\text{all } x} x \cdot p(x) = 1 \times 1/3 + 2 \times 1/3 + 3 \times 1/3 = 2 \]
There is a crucial distinction between random variables and constants that students sometimes miss. If \(X\) is a random variable, the values it takes on are random, and a function \(g(X)\) of a random variable is itself a random variable. On the other hand, \(E[X]\) is a constant (you should convince yourself of this). Realizations \(x\) are constants; what is random is which realization the random variable takes on. Parameters, such as \(p\) for a Bernoulli random variable, are constants, and so is the sample size \(n\).
How much would you be willing to pay for the right to play the following game? Imagine a fair coin. The coin is tossed once. If it falls heads, you receive a prize of $2 and the game stops. If not, it is tossed again. If it falls heads on the second toss, you get $4 and the game stops. If not, it is tossed again. If it falls heads on the third toss, you get $8 and the game stops, and so on. The game stops after the first head is thrown. If the first head is thrown on the \(x\)th toss, the prize is $\(2^x\).
Let \(X\) be the trial number of the first head, so that the prize is \(Y = 2^X\):
| \(x\) | \(2^x\) | \(p(x)\) | \(2^x \cdot p(x)\) |
|---|---|---|---|
| 1 | 2 | 1/2 | 1 |
| 2 | 4 | 1/4 | 1 |
| 3 | 8 | 1/8 | 1 |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
| \(n\) | \(2^n\) | \(1/2^n\) | 1 |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
\[ E[Y] = \sum_{\text{all } x} 2^x \cdot p(x) = 1 + 1 + 1 + \cdots = \infty \]
Functions of random variables are themselves random variables. For example, let \(Y = e^X\), where \(X \sim \text{Bernoulli}(p)\). The support of \(Y\) is \(\{e^0, e^1\} = \{1, e\}\), and its pmf is \[ p_Y(y) = \left\{\begin{array}{ll} p & y = e\\ 1 - p & y = 1\\ 0 & \text{otherwise} \end{array}\right. \] So the expectation of \(Y = e^X\) is \[ \sum_{y \in \{1, e\}} y \cdot p_Y(y) = (1-p) \cdot 1 + p \cdot e = 1 + p(e - 1) \] while the function of the expectation is \(e^{E[X]} = e^p\). These are not the same. In general, \[ E[g(X)] \neq g(E[X]) \] The expected value of a function is not the function of the expected value. To calculate the expected value of a function \(g\) of a discrete random variable \(X\), we use \[ E[g(X)] = \sum_{\text{all } x} g(x)\, p(x) \] This is how we proceeded in the St. Petersburg game example. For example, if \(X\) has support \(\{-1, 0, 1\}\) with \(p(-1) = p(0) = p(1) = 1/3\), then \[ E[X^2] = \sum_{x \in \{-1, 0, 1\}} x^2 p(x) = (-1)^2 \cdot (1/3) + (0)^2 \cdot (1/3) + (1)^2 \cdot (1/3) = 2/3 \]
Linearity of expectation
Watch: Lecture 9 from 1:00.
We’ve seen that it’s not true that the expected value of a function equals the function of the expected value: you can’t interchange the \(E\) for expected value and a function. Well, it turns out there’s one very special case where you can interchange those two things. This case is so special that we have a name for it. It is a property of expected value, and we call it the linearity of expectation. It holds for all random variables, continuous as well as discrete. The proof is a little different for the continuous case, but the basic logic is the same.
Suppose we’ve got a random variable \(X\) and two constants \(a\) and \(b\). Then the expected value of the function \(a + bX\) is equal to the function of the expected value: it equals \(a\) plus \(b\) times the expected value of \(X\): \[ E[a + bX] = a + bE[X] \] So while it’s not true in general that you can interchange expected value and a function like this, it is true in this very special case, where the function is linear. If you remember one thing about random variables, remember this: expected value is linear. It’s a linear operator. This is going to turn out to be crucial in pretty much all the formulas that follow. For example, if \(X \sim \text{Bernoulli}(1/3)\) and \(Y = 3X + 2\), then \(E[X] = 0 \times 2/3 + 1 \times 1/3 = 1/3\), and \(E[Y] = E[3X + 2] = 3E[X] + 2 = 3\).
So let’s take a closer look at the proof. It’s not that hard, and understanding how it works will give you a sense of why this linearity of expectation is something we study at all. You’re going to see exactly why it’s true; it’s not going to be a mysterious thing. What is the expected value of \(a + bX\)? Well, \(a + bX\) is a function: \(g(x) = a + bx\). And we know how to calculate the expected value of a function of a random variable: we sum over all \(x\) in the support, the pmf multiplied not by \(x\), but by the function \(g(x)\): \[ \begin{aligned} E[a + bX] &= \sum_{\text{all } x} (a + bx)\, p(x)\\ &= \sum_{\text{all } x} p(x) \cdot a + \sum_{\text{all } x} p(x) \cdot bx\\ &= a \sum_{\text{all } x} p(x) + b \sum_{\text{all } x} x \cdot p(x)\\ &= a + bE[X] \end{aligned} \] So what have I done here? I’m just using our properties of summation notation to split things up. I’ve taken \(a + bx\) multiplied by \(p(x)\), distributed the \(p(x)\) through to the \(a\) and the \(bx\), and split the sum into two pieces. Now, what’s our second favorite trick besides splitting up sums? We like to find things that do not vary over the index of summation and pull them out of the sum. \(a\) does not depend on \(x\). That’s the crucial point here. \(a\) is a constant. The linearity of expectation refers to the special function that takes the form \(a + bx\), where \(a\) and \(b\) are constants that do not depend on \(x\). They’re not random; they’re just some numbers. I haven’t specified what they are, but whatever \(a\) is, it doesn’t depend on \(x\), and whatever \(b\) is, it doesn’t depend on \(x\). So I can pull them in front.
Now we’re pretty much done. What happens if I sum \(p(x)\) over all \(x\) in the support? \(p(x)\) is the probability mass function, so I’m summing it at every single realization in the support of \(X\). These realizations are all mutually exclusive, and they’re collectively exhaustive: one of them has to happen. So if you sum the probabilities of all of them, you get 1. What about the second piece? If you sum this, you’re not going to get 1, because it isn’t the plain pmf by itself; it’s multiplied by something else. But what is this object? If we have a discrete random variable, and we sum over all \(x\) in the support \(x\) times \(p(x)\), that’s just the definition of expected value. So the first sum is 1, and 1 times \(a\) is \(a\), and the second sum is just the expected value of \(X\). And so we have exactly what we wanted: \(a\) plus \(b\) times the expected value of \(X\). A very simple proof. All it uses is the formula for the expected value of a function, a couple of basic operations with sums, the properties of pmfs, and the definition of the expected value of a discrete random variable.
Variance and standard deviation
Watch: Lecture 9 from 7:00.
Okay, so expected value is linear. That’s fantastic. It’s going to allow us to derive all kinds of cool stuff. That’s one class of functions of a random variable \(X\), the class \(a + bX\). But there are lots of other functions we could think about. In particular, we’re often going to be very interested in the variance. If we want to talk about the variance of a random variable \(X\), which we will often denote \(\sigma^2\), it is defined as the expected value of the following function of \(X\): you take \(X\) and subtract \(\mu\), where \(\mu\) is shorthand for the expected value of \(X\). Remember, the expected value of \(X\) is a constant, so this is just a number. We have to work out what it is, but it’s a number. So we define variance in terms of the function where you take a random variable, subtract its mean, and square it. This is a function \(g(x) = (x - \mu)^2\), and the variance of \(X\) is simply the expected value of that particular function of \(X\). And, unsurprisingly, the standard deviation is just the square root of the variance: \[ \sigma^2 = Var(X) = E\left[(X - \mu)^2\right] = E\left[(X - E[X])^2\right], \qquad \sigma = \sqrt{\sigma^2} = SD(X) \] So you don’t have to memorize something separate for the standard deviation. Just remember that the standard deviation is always the square root of the variance.
So variance and standard deviation are both expectations of functions of a random variable. It follows that variance and standard deviation are constants; they are not random variables. Here is the point. When you take an expected value of something, be it a plain random variable or some function of that random variable, you’re multiplying probabilities by either realizations or some function of the realizations. Realizations are just numbers: they’re all the possible values a random variable could take on. And the probability mass function spits out a bunch of numbers, these probabilities. So when you actually carry out the summation, you get a number. Nothing random about it. It also follows that to understand exactly what these things do and how they behave, we can use the facts we’ve already learned about expected value. Variance is not some weird new concept; it’s just the expected value of a particular function.
In particular, it follows immediately that we know how to calculate the variance of a discrete random variable. Recall that \(\mu\), the expected value of \(X\), is the sum over all \(x\) in the support of \(x\) times \(p(x)\): the pmf times the corresponding realization. And to find the expected value of the function \(g(x) = (x - \mu)^2\), we sum over all \(x\) in the support \(p(x)\) times \(g(x)\): \[ \mu = E[X] = \sum_{\text{all } x} x\, p(x), \qquad Var(X) = E\left[(X - \mu)^2\right] = \sum_{\text{all } x} (x - \mu)^2 p(x) \] So to evaluate the variance of a discrete random variable, you first need to know the mean, \(\mu\). Then you plug that number into this formula. Once you know \(\mu\), everything else is just the realizations and the pmf, and you will get a number out.
Watch: Lecture 9 from 11:40.
As it turns out, the formula above is not, in general, the fastest way to calculate a variance. In fact, I can’t think of a single example where it’s the fastest way. There is an equivalent way to do the calculation, which will always give you exactly the same answer but is much faster. I call this the shortcut formula for variance. Let me be clear: this is not the definition of variance; it is just a shortcut for doing calculations. The definition of variance is the expected value of the quantity \(X\) minus \(\mu\), squared. The shortcut formula says that the answer you get from this is identical to the following: take the expected value of \(X^2\) by itself, and then subtract the expected value of \(X\), quantity squared: \[ Var(X) = E\left[(X - \mu)^2\right] = E[X^2] - \left(E[X]\right)^2 \] Notice, and this is crucial, that the square in the first term is inside the expectation, whereas the square in the second term is outside. So the second term is \(\mu^2\), but the first term is not, not, not, not equal to \(\mu^2\). The expected value of \(X^2\) is not, in general, equal to the expected value of \(X\), squared. We learned that you can’t switch the order of expectation and function, and that the one exception is linear functions. This is not a linear function; it’s quadratic. So these two terms are going to be different, which is good, because if they weren’t, then our definition of variance would be totally vacuous: it would always be 0. We’ll prove the shortcut formula below, in the section on sums of random variables. For the moment, let’s just try computing with it.
Watch: Lecture 9 from 13:39.
Suppose we want to calculate the variance of a Bernoulli random variable. It has two possible realizations, 0 and 1. The probability of 1 is \(p\), and the probability of 0 is \(1 - p\). We usually call 1 a success and 0 a failure.
The first step is to calculate the expected value of \(X\). We sum over the possible realizations, 0 and 1, the pmf times \(x\): \(1 - p\) times 0, which is 0, plus \(p\) times 1, which is \(p\). So the expected value of a Bernoulli random variable is just \(p\), the probability of success. The second step is more interesting. Now we calculate the expected value of the quantity \(X^2\). This is the expected value of a function \(g(x)\), where \(g\) is the quadratic function \(x^2\). Remember that the way we calculate the expected value of a function is to sum over all realizations in the support the pmf multiplied by that function of \(x\). So we take each of the realizations, square it, and multiply by the probability of getting that realization. Again, the function is being applied to \(x\), not to \(p(x)\). Do not try to square \(p(x)\); you’ll get crazy answers. It’s always the realization to which we apply the function. So we have \(0^2\) times \((1 - p)\) plus \(1^2\) times \(p\): \[ \begin{aligned} \mu = E[X] &= \sum_{x \in \{0,1\}} p(x) \cdot x = (1-p) \cdot 0 + p \cdot 1 = p\\ E[X^2] &= \sum_{x \in \{0,1\}} x^2 p(x) = 0^2 (1-p) + 1^2 p = p\\ \sigma^2 = Var(X) &= E[X^2] - \left(E[X]\right)^2 = p - p^2 = p(1-p) \end{aligned} \] Now, that’s interesting, because this does not normally happen. It’s quite a surprise to see that the expected value of the Bernoulli, \(p\), is exactly equal to the expected value of its square. And we can see exactly why that happened: \(0^2\) and \(1^2\) are the same as 0 and 1. This is a special feature of the Bernoulli random variable that you’re not going to see in other examples. So don’t make the mistake of thinking that, in general, the expected value of \(X^2\) equals the expected value of \(X\), because it doesn’t. Finally, we use the shortcut formula to combine these two pieces. To calculate the variance of \(X\), we don’t need to proceed by the definition; we can instead take the expected value of \(X^2\), which is \(p\), minus the expected value of \(X\), quantity squared, which is \(p^2\). And we’re done. This is usually written \(p(1 - p)\). So the variance of a Bernoulli random variable is equal to the probability of success multiplied by the probability of failure, a very simple and easily computed quantity.
What would have happened if we hadn’t used the shortcut formula? We would have gotten exactly the same answer, but it’s a bit of a pain. You can check this yourself: proceed directly from the definition, plug in \(\mu = p\), and expand. You’ll get the same answer: \[ \sigma^2 = Var(X) = \sum_{x \in \{0,1\}} (x - \mu)^2 p(x) = \sum_{x \in \{0,1\}} (x - p)^2 p(x) = \cdots = p(1-p) \] In other examples it’s even more of a pain, so I always use the shortcut formula when I’m doing calculations, and you can too, especially on exams. I really strongly recommend it. But it’s important to know both the definition and the shortcut formula.
Watch: Lecture 9 from 18:16.
Here’s a question for you. Suppose we have a random variable \(X\) whose variance is \(\sigma^2\), and suppose that \(a\) and \(b\) are constants. What do you think the variance of \(a + bX\) is going to be? We’re taking a linear function of \(X\), and I’m asking how the variance of this function of \(X\) relates to the variance of \(X\). It might not be obvious. Think about it for a minute before reading on.
This is actually quite a hard question, and it brings us to a very important point: variance and standard deviation are not linear. Expectation has this linearity property, but variance and standard deviation are different. It turns out that the variance of \(a + bX\) is equal to \(b^2\) times the variance of \(X\): \[ Var(a + bX) = b^2 \sigma^2, \qquad SD(a + bX) = |b|\,\sigma \] Notice that \(a\) had absolutely no effect on the answer. None at all. And notice that \(b\) had an effect, but the effect was squared. What this means is that shifting a random variable over, by adding \(a\) to it, just moves it to the left or the right. It doesn’t change the spread. That’s kind of intuitive. Multiplying a random variable, however, can change the spread, and indeed we see that it does. But notice that \(b\) could be negative. If it were, the negative sign would disappear on the right-hand side. You can’t have a negative variance, and that’s another way to remember that the square has to be there. The standard deviation is just the square root of this, but again, it can’t be negative, so we’ve got the absolute value sign.
Watch: Lecture 9 from 20:56.
Let’s prove it. This is not that hard. We’re going to go straight from the definition of a variance, and we’re going to consider \(a + bX\) to be a new random variable. If you want to, you can cross out \(a + bX\) everywhere and write \(Y\). Then you’ll notice that the first line below just says that the variance of \(Y\) is defined as the expected value of \(Y\) minus its mean, quantity squared. It just so happens that the \(Y\) we’re interested in is this linear function of \(X\): \[ \begin{aligned} Var(a + bX) &= E\left[\left\{(a + bX) - E(a + bX)\right\}^2\right]\\ &= E\left[\left\{(a + bX) - (a + bE[X])\right\}^2\right]\\ &= E\left[\left(bX - bE[X]\right)^2\right]\\ &= E\left[b^2 (X - E[X])^2\right]\\ &= b^2 E\left[(X - E[X])^2\right]\\ &= b^2 Var(X) = b^2 \sigma^2 \end{aligned} \] Let’s have a look. In the first line we have the expected value of \(a + bX\). We learned that when the function is linear, you can exchange the order of expected value and function. In particular, you can pull constants out of an expected value, and \(a\) and \(b\) are both constants here. The expected value of the constant \(a\) is just \(a\); it’s not random. And \(b\) is not random either; it’s a constant, so we can pull it out in front. So the expected value of the quantity \(a + bX\) is equal to \(a + b\) times the expected value of \(X\). That’s what we did in moving to the second line. Now we’ve got two terms inside the curly braces that are being squared, and we can simplify this a bit. We’ve got an \(a\) minus \(a\), and we’ve got \(bX\) minus \(b\) times the expected value of \(X\). So let’s kill the two \(a\)’s and pull the \(b\) out in front. This is a little tricky. Everything inside the parentheses is being squared, so if I want to pull \(b\) out in front, I’ve got to square it too. So in moving from the third line to the fourth, I factored out the \(b\), but because everything inside the parentheses is getting squared, the \(b\) got squared too.
Now we’re very nearly done. This part is possibly conceptually a little confusing, so I’m going to spend a moment explaining it. We have something a little bit complicated: a random variable minus its mean, quantity squared. Remember, the expected value of \(X\) is just a constant. So we take a random variable, subtract a constant, and square it. This is a function of the random variable \(X\): it’s \(g(X)\), where the function \(g\) is \((x - \mu)^2\). So it is a function of a random variable, which means it is a random variable itself. If you want to, you can cross it out for a moment and just call it \(Z\). And what about \(b^2\)? \(b\) is a constant, so \(b^2\) is a constant. Not random. So what this expression says is “the expected value of a constant times some slightly complicated random variable”. This means that we can use the linearity of expectation. It’s a special case of our \(a + bX\) result from before: here there’s no \(a\), or, if you like, \(a\) is 0, and instead of \(b\) we have \(b^2\). But that doesn’t matter, because \(b^2\) is just a constant. So we can pull it in front. Now what are we left with? This is pretty convenient, because you should recognize what’s left as the definition of the variance of \(X\). So we’ve shown that the variance of \(a + bX\) is simply equal to \(b^2\) times the variance of \(X\).
Notice that nothing we did here in any way used summations. It didn’t use any of the stuff to do with the probability mass function, and it in no way relied on the assumption that we’re working with a discrete random variable. So all of this is going to go through just fine for continuous random variables. This argument is totally general. And here’s the point of this derivation. Variance is defined in terms of expected value, and expected value has this really nice linearity property. Variance does not have the linearity property, because it involves a quadratic function, not a linear one. Nevertheless, at various steps in this proof, we had little pieces where we could use the linearity of expectation: in moving to the second line, and again in pulling out the \(b^2\). So while variance is not linear, we can use the fact that it’s defined in terms of expectation, and that expectation is linear, to figure out the variance of a linear function of \(X\).
Conditional expectation
Watch: Lecture 10 from 22:00.
The examples in this section and the following ones use the joint pmf from Discrete Random Variables I, shown here with its marginals:
| \(Y = 1\) | \(Y = 2\) | \(Y = 3\) | \(p_X(x)\) | |
|---|---|---|---|---|
| \(X = 0\) | 1/8 | 0 | 0 | 1/8 |
| \(X = 1\) | 0 | 1/4 | 1/8 | 3/8 |
| \(X = 2\) | 0 | 1/4 | 1/8 | 3/8 |
| \(X = 3\) | 1/8 | 0 | 0 | 1/8 |
| \(p_Y(y)\) | 1/4 | 1/2 | 1/4 | 1 |
There’s one more key idea that we’re going to need, and this one is very confusing. It’s extremely important, but the first time you see it, and maybe even the second and the third and the fourth time, this idea is just a lot more difficult to get your head around than some of the ones that we’ve seen before. But it turns out to be very important, in particular in econometrics, when you want to think about regression. The notion we’re going to be introducing now is that of conditional expectation, and here is the basic intuition. We use the notation \(E[Y|X]\). This is read “the conditional expected value of \(Y\) given \(X\)” or “the conditional expectation of \(Y\) given \(X\).” I’ll use the two terms interchangeably, because you’ll encounter them both. The conditional expected value of \(Y\) given \(X\) is our best guess of the realization that \(Y\) will take on, having observed the realization of \(X\). So this is sort of a strange idea. It’s a bit different from plain old expectation, the plain old expected value that we’ve already looked at. Intuitively, this is tied to the notion of trying to predict \(Y\) using what you know about \(X\).
Ordinary expectation, just plain old expected value like the expected value of \(Y\), is a constant. It’s a constant number that we arrive at by multiplying all of the probabilities of the different realizations that a random variable can take on by the realizations themselves, and adding them up. So that just gives us a number. Conditional expected value is different, and this is something you have to be very careful about. While plain old expected value is a constant, conditional expected value is a function, a function of \(X\). Hence, it is actually a random variable itself. Now, this is very confusing, so I’m going to try to explain it in words.
Capital \(X\) is a random variable. We don’t know what value it’s going to take on; it’s going to spit out some random number. So what the conditional expected value of \(Y\) given \(X\) says is: let’s take a look at the realization that \(X\) took on. Let’s see what it was. Depending on what value \(X\) took on, we’re going to report a forecast of \(Y\). So this is just a silly example, but suppose \(X\) could only take on the value zero or one. What the conditional expected value of \(Y\) given \(X\) would give us in that case is our best guess of \(Y\) in each of the two situations: when \(X\) is 0 and when \(X\) is 1. So this is a function of \(X\). You can even write \(E[Y|X] = g(X)\). It is a function where you plug in a realization of \(X\) and you get out a number. So it is a function of a random variable; hence it is itself a random variable. But the whole point here is that this is like a machine that’s going to tell us how to guess \(Y\) using what we know about \(X\). You plug in a realization for \(X\), and you get out a guess for \(Y\). The reason that this thing is a random variable and not a number is that \(X\) is spitting out realizations at random. So the function is deterministic: it’s a rule that translates the values that \(X\) can take on into our corresponding best guess of \(Y\) in each situation. But the values that we’re plugging into this function are random, because they are the realizations of \(X\). So that’s the intuition for why the expected value of \(Y\) given \(X\) is a random variable.
Now, once you actually plug in a particular realization, a particular value little \(x\) that \(X\) could take on (so suppose you actually observe \(X\), and you see that it took on the value 1), then this whole thing collapses to a constant, because, again, the conditional expectation is a function of \(X\). The function isn’t random. The only thing that’s random is which value we’re evaluating that function at. So once you fix the place where we’re evaluating the function, suddenly this is no longer random. So this is a little confusing. It’s more than a little confusing, actually. It’s very confusing. The conditional expected value function, \(E[Y|X]\), is a random variable. But when you plug a particular realization into it, you get a constant, \(E[Y|X = x]\), and that constant is our best guess of \(Y\) given the particular value little \(x\) that we have plugged in for our random variable capital \(X\).
So how do we calculate this conditional expectation? Well, understanding what it means is complicated, but calculating it actually doesn’t turn out to be, because we’re going to work not primarily with \(E[Y|X]\), but with \(E[Y|X = x]\), where we plug in a particular realization for \(X\). We’re just going to calculate the conditional probability mass function of \(Y\) given that \(X\) took on some particular realization little \(x\), and we’re going to go from there, because once we have a probability mass function, we can work out its expected value. In short: to calculate \(E[Y|X = x]\), take the mean of the conditional pmf of \(Y\) given \(X = x\).
Watch: Lecture 10 from 28:50.
So, back to this example. We’ve been beating it to death, and we’re going to continue beating it to death. We already worked out the conditional probability mass function of \(Y\) given that \(X\) equals 2 (see Discrete Random Variables I). So how do we work out the conditional expectation of \(Y\) given that \(X\) equals 2? Well, we’re going to multiply all the different realizations that \(Y\) can take on by the probabilities that \(Y\) takes on those realizations. But we’re not going to use the marginal probabilities. So we’re not going to use 1/4 times 1 plus 1/2 times 2 plus 1/4 times 3. We’re going to use the conditional probabilities for the different realizations of \(Y\) given this information about \(X\). So instead of 1/4, 1/2, and 1/4, we’re going to use 0, 2/3, and 1/3: \[ E[Y|X = 2] = 2 \times 2/3 + 3 \times 1/3 = 7/3 \] So here’s what this says. If you knew that \(X\) was equal to 2, then your best guess of \(Y\), the sort of average value of \(Y\) given that information, which is not going to be equal to the plain old ordinary average of \(Y\), is 7/3. So again, this notion is a little bit complicated, but the way we compute it is actually pretty simple.
Let’s try one more quick example: the conditional expectation of \(Y\) given that \(X\) is 0. Again, we’re going to take the realizations that \(Y\) can take on, 1, 2, and 3, multiply them by some probabilities, and add up. But we’re not going to use the marginal probabilities of \(Y\), 1/4, 1/2, and 1/4. Instead, we’re going to use the conditional probabilities of \(Y\) given that \(X\) is 0. So what are those probabilities? Well, those are pretty easy to work out. We have 1/8 over 1/8, which is 1; 0 over 1/8, which is 0; and finally 0 over 1/8, which is 0. So look at this. If we know that \(X\) is 0, then conditional on that information, \(Y\) is guaranteed to be 1. So this is a very easy one: 1 times 1 plus 0 times 2 plus 0 times 3 is 1. \[ E[Y|X = 0] = 1 \] This example is a little silly, but it’s meant to illustrate exactly what we mean by best guess. Here’s an example where you know \(Y\) for sure if you know that \(X\) is zero. So if our intuitive notion of conditional expectation is working properly, then in this case, where you actually get to know what \(Y\) is given this piece of information about \(X\), we’d sure like the conditional expectation to equal what our intuition tells us it should be. And it does. If \(X\) is 0, you know for sure that \(Y\) is 1, so your best guess of \(Y\) given that \(X\) is 0 is definitely 1. And there’s a very similar calculation for the conditional expectation of \(Y\) given that \(X\) equals 3, because if \(X\) equals 3, again, we know for sure that \(Y\) has to be 1. So we can skip all the calculations, and we see immediately that if you know that \(X\) is 3, then \(Y\) has to be 1, so \(E[Y|X = 3] = 1\). That’s just how this table is set up. And again, this is a particular example that I cooked up for illustrative purposes. Other examples might not look like this, but the concepts are going to be the same.
So now it’s your turn. I would like you to calculate the conditional expectation of \(Y\) given that \(X\) equals 1. Remember, what you’re going to be doing here is weighting the different realizations that \(Y\) can take on by some probabilities. The probabilities you’re going to use are those from the conditional pmf of \(Y\) given that \(X\) equals 1.
Here we are. We’re going to look at the row where \(X\) is 1. We see that there are three joint probabilities in that row, and by summing them we get the marginal probability that \(X\) takes on the value 1, which is 3/8. If we divide each of these joint probabilities by the marginal, we’re going to get 0; a fourth over three-eighths, which is two-thirds; and an eighth over three-eighths, which is one-third. So here are the conditional probabilities of \(Y\) given \(X = 1\), and these are the probability weights that we’re going to use when we add up the realizations that \(Y\) can take on. So we will have 0 times 1 plus 2 times two-thirds plus 3 times one-third, which equals 7/3: \[ E[Y|X = 1] = 2 \times 2/3 + 3 \times 1/3 = 7/3 \] And this may look familiar, because this calculation is actually the same as the one for the conditional expectation of \(Y\) given that \(X\) equals 2. That’s not going to be the case in all examples, but it happens to be the case in this simple one that I made up.
Watch: Lecture 10 from 35:30.
So we’ve now worked out our best guess of \(Y\) for every possible value that \(X\) could take on, for every possible realization of the random variable \(X\). Now we’re going to be able to make it clearer, I hope, exactly what I mean when I say that conditional expectation is a random variable. So again, let’s be clear: the things we’ve just calculated are numbers, because we’ve calculated the conditional expectation at a fixed realization of \(X\). But the full conditional expectation is a function of \(X\) that tells us what to guess about \(Y\), given the possible realizations of \(X\). And we now have enough information to put that object together: \[ E[Y|X] = \left\{\begin{array}{cc} 1 & X = 0\\ 7/3 & X = 1\\ 7/3 & X = 2\\ 1 & X = 3 \end{array}\right. \] So we have seen that if \(X\) takes on the value 0, our best guess of \(Y\) is 1. Whereas if \(X\) takes on the value 1, our best guess of \(Y\) is 7/3. And if \(X\) takes on the value 2, our best guess of \(Y\) is again 7/3. Finally, if \(X\) takes on the value 3, then our best guess of \(Y\) is 1. So you tell me \(X\), and I now have a function, a rule, that tells me exactly what \(Y\) to guess. You give me an \(X\), I give you a \(Y\). This is a function.
Now notice (and again, I cooked up this particular example to have a simple structure) that this function only takes on two different values. Remember, when I talk about the values a function can take on, I’m talking about the values the function spits out. So I’m talking about the best guesses. If we plug in 3, we get out 1. If we plug in 2, we get out 7/3. Notice there are only two different numbers that we can get out of this function: we can get out a 1, or we can get out 7/3. Now, it happens that there are multiple different inputs that will give us the same output, but that’s fine. Functions only have to pass the vertical line test, not the horizontal line test.
So where does the randomness come in? Well, the point is, the value that \(X\) is going to take on is random. In particular, we know that \(X\) takes on the value 0 with probability 1/8, the value 1 with probability 3/8, the value 2 with probability 3/8, and finally, the value 3 with probability 1/8. If you’re not sure where these probabilities came from, go back to the table and take a look at the margin for \(X\). So where does this leave us? Well, the conditional expectation of \(Y\) given \(X\) is a random variable. It says we’re going to look at the realization that \(X\) takes on, and then we’re going to give our best guess of \(Y\). Since the realization that \(X\) takes on is itself random, that means the best guess is going to be random. So the best guess is either 7/3 or 1, depending on what happened for \(X\).
So when is it going to be 7/3? Well, it’s going to be 7/3 either if \(X\) is 1 or if \(X\) is 2. These are mutually exclusive. So the probability that this function spits out the value 7/3 is the probability that \(X\) is either 1 or 2, and that is 3/8 plus 3/8, which is 6/8. That’s 3/4. So this random variable, the conditional expectation of \(Y\) given \(X\), takes on the value 7/3 with probability 3/4, or otherwise it takes on the value 1. So what’s the probability that it takes on the value 1? Well, we could do this the easy way and just say probabilities have to sum to 1, so it’s got to be 1/4. But let’s do it the hard way. It takes on the value 1 either when \(X\) is 0 or when \(X\) is 3. Those are two mutually exclusive events, so to get the probability of the union, we can sum the separate probabilities. We take the probability that \(X\) is 0, which is 1/8, and we add to that the probability that \(X\) is 3, which is also 1/8. 1/8 plus 1/8 is 2/8, which is 1/4.
So, to sum up, the conditional expected value of \(Y\) given \(X\) in this example is a random variable that can take on two values, 1 or 7/3. It takes on the value 1 with 25% probability, and the value 7/3 with 75% probability. All the randomness is coming from the fact that we don’t know which \(X\) we’re going to observe. If we know which \(X\) we’re going to observe, then we immediately know our best guess of \(Y\). But since \(X\) is itself spitting out random realizations, the best guess that we make conditional on \(X\) is going to be random. And we’ve now characterized exactly what that randomness looks like.
The Law of Iterated Expectations
Watch: Lecture 10 from 41:50.
If you thought that was hard, wait until you see this. There’s one more concept to look at here. It’s probably the most challenging one, but it’s so very useful in econometrics that I feel like I really ought to mention it. This is something called the Law of Iterated Expectations.
As we saw, the conditional expectation of \(Y\) given \(X\) is a random variable. We worked out in this particular example exactly what the realizations of that random variable were and what the probabilities of it taking on those different realizations were. Remember that in this particular example it could take on only the values 1 or 7/3, and the corresponding probabilities were 25% and 75%. So here’s a random variable, the expected value of \(Y\) given \(X\). It’s random. What’s its mean? What’s its expectation? Let me be clear: there are two kinds of expectations running around here. One of them is a plain old, ordinary, garden-variety, vanilla expectation, like the expected value of \(Y\). That’s a number, a constant number. In contrast, the conditional expectation, the expected value of \(Y\) given \(X\), is a function of the random variable \(X\) that gives us our best guess of \(Y\), depending on the realization we get for \(X\). So that best-guessing function is spitting out random best guesses, because it’s conditioning on something random, the value of \(X\). On average, what number is it spitting out? It turns out that if you take the plain old expected value of a conditional expectation, then you just get back the ordinary, unconditional expectation. To put it another way, the expected value of the expected value of \(Y\) given \(X\) is just equal to the plain old expected value of \(Y\): \[ E\left[E\left[Y|X\right]\right] = E[Y] \] This is called the Law of Iterated Expectations. It turns out that the proof of this is not that hard, but I don’t want to dwell on it too much. Just to be clear about what’s going on here, remember that the expected value of \(Y\) given \(X\) is a function of the random variable capital \(X\). So the outer expectation is averaging over the randomness in \(X\). The inside expectation says, “You tell me the \(X\), I’ll tell you my best guess of \(Y\).” My best guess is going to be random, because the \(X\) I observe is going to be random. And now let me average over the probabilities of all the different \(X\)’s. When I do this, and this turns out to be pretty convenient, I get back something very simple: the plain old ordinary expected value of \(Y\), the one you calculate by multiplying the realizations of \(Y\) by the marginal probabilities of \(Y\), the ones that come from the marginal pmf.
Let’s check it out. Here’s the marginal pmf of \(Y\): \(Y\) takes on the value 1 with probability 1/4, 2 with probability 1/2, and 3 with probability 1/4. That means that the expected value of \(Y\) is 2. I’m multiplying each of the realizations by the corresponding probabilities: 1 times 1/4 plus 2 times 1/2 plus 3 times 1/4 is 2. That should be straightforward. Now, what about the conditional expectation? Well, we worked this out above. It’s a random variable that takes on the value 1 with probability 25% and the value 7/3 with probability 75%. So what we are now going to do is calculate the expected value of this. We’re going to have 1 times a quarter plus 7/3 times 3/4. That’s what it means to take the expected value of the conditional expectation: we’re going to multiply the best guesses by the probabilities with which we’re going to be making those guesses. And 1 times a quarter plus 7/3 times 3/4 is indeed 2: \[ \begin{aligned} E[Y] &= 1 \times 1/4 + 2 \times 1/2 + 3 \times 1/4 = 2\\[4pt] E\left[E\left[Y|X\right]\right] &= 1 \times 1/4 + 7/3 \times 3/4 = 2 \end{aligned} \] So I haven’t proved this, but I’ve demonstrated numerically that it does work out in this example.
Covariance and correlation
Watch: Lecture 10 from 47:50.
We’ve talked about how to calculate the expected value of a function of one random variable, but now that we’re dealing with joint probabilities, two random variables at once, we want to have a way to calculate the expectation of a function of two discrete random variables. Here’s how you do it: \[ E[g(X,Y)] = \sum_x \sum_y g(x,y)\, p_{XY}(x,y) \] You multiply the function \(g\) evaluated at a particular pair of realizations \(x\) and \(y\) by the joint probability mass function of that pair of values. So again, it’s a probability, this time a joint probability, multiplied by the value of a function evaluated at the same pair of realizations for which you calculated the probability. And then we have a double sum, a sum over \(x\) and \(y\). What that means is you sum over all the pairs of realizations \(x, y\) in the table. So if the double sum notation looks intimidating, don’t worry about it. You can just think, “This means I sum over all the values in the table.” This formula looks very similar to the one that we’ve already seen for a single random variable. It’s just that the function takes two arguments now, as does the probability mass function, because it’s now a joint pmf. And correspondingly, there’s a double rather than a single sum.
The main reason I gave you this formula for calculating the expected value of a function of two discrete random variables is that there are a couple of functions that are going to be very important to us. First of all, a little notation. I’m going to use the symbol \(\mu\), the Greek letter m, with a capital \(X\) subscript to indicate the expected value of \(X\), and I’m going to use the symbol \(\mu\) with subscript capital \(Y\) to denote the expected value of \(Y\). So remember, these are plain old ordinary expectations. They’re numbers, based on multiplying all the realizations of \(X\) by the corresponding probabilities from the marginal pmf, and similarly for \(Y\). We’re going to define the covariance in terms of a function of two random variables: it’s the expected value of \(X\) minus its mean times \(Y\) minus its mean. We’re going to use the symbol \(\sigma\), the Greek letter s, with subscript capital \(X\) capital \(Y\). And we’re going to define correlation as you might expect: take the covariance and divide by each of the standard deviations. \[ \begin{aligned} \sigma_{XY} &= Cov(X,Y) = E[(X - \mu_X)(Y - \mu_Y)]\\[4pt] \rho_{XY} &= Corr(X,Y) = \frac{\sigma_{XY}}{\sigma_X \sigma_Y} \end{aligned} \]
So how do we actually calculate these? Well, I just gave you a formula for the expected value of a function of two random variables, and you could apply it here. Remember, \(\mu_X\) is a constant, \(\mu_Y\) is a constant, and \(X\) and \(Y\) are random variables, so \((X - \mu_X)(Y - \mu_Y)\) is just a function of two random variables, \(X\) and \(Y\). But there’s actually an easier way. Just like there was a shortcut formula for calculating variance, it turns out there’s a shortcut formula for calculating covariance, and here it is: \[ Cov(X,Y) = E[XY] - E[X]E[Y] \] This is much easier for doing calculations. Something I just want to point out to you is that you might think there are a lot of formulas to memorize. Don’t worry. Take a look at the shortcut formula for covariance. What happens if you plug in \(X\) and \(X\)? You get back exactly the shortcut formula for variance. So really, you only have to remember this one, and remember that the variance of a random variable is exactly the same thing as its covariance with itself. If you can remember this one, you’re done, and you can immediately use this rule for either covariances or variances, because if you plug in \(X\) and \(X\), you get the expected value of \(X^2\) minus the expected value of \(X\), whole thing squared.
Watch: Lecture 10 from 53:40.
All right, let’s do a calculation. Back to our infamous example. We can work out the expected value of \(X\) and the expected value of \(Y\) using the marginal probability mass functions: \[ \begin{aligned} E[X] &= 3/8 + 2 \times 3/8 + 3 \times 1/8 = 3/2\\[4pt] E[Y] &= 1/4 + 2 \times 1/2 + 3 \times 1/4 = 2 \end{aligned} \] We want to calculate the covariance of \(X\) and \(Y\) using the shortcut formula. For the shortcut formula, we need three things: we need the expected value of \(X\), we need the expected value of \(Y\), and we need the expected value of the quantity \(X\) times \(Y\). So how on earth do we calculate that? Well, \(X\) times \(Y\) is just a function of \(X\) and \(Y\). It’s one of the simplest ones you could come up with. So how do we calculate the expected value of this function, \(X\) times \(Y\)? Remember, the formula for calculating the expected value of a function of two discrete random variables is as follows. You go to the joint pmf for every possible pair of realizations, \(x\) and \(y\). You calculate the joint probability of those two realizations happening at the same time. And you multiply it by the function you’re interested in, evaluated at the same pair of realizations.
So let’s take a quick look. We’re going to have to go through the whole table, and for every single pair (and there are 4 times 3, so 12 pairs in the table here), we’re going to need to take the probability mass function and multiply it by \(x\) times \(y\) at that position. Well, I’m lazy. So I’m going to look at this table and say to myself, well, hang on a second. \(x\) times \(y\), what is that? Well, it involves multiplying a value of \(x\) by a value of \(y\), so maybe 1 times 2, or 2 times 3, or whatever. But look, in the first row, \(x\) is 0. So when I go to do my calculation, what will I have? 0 times 1 times the probability of landing there, which is 1/8, plus 0 times 2 times the probability of landing there, which is 0, plus 0 times 3 times the probability of landing there, which is 0. So in the row where \(x\) is 0, it actually doesn’t matter what the probabilities are. They’re not going to contribute anything to the sum, because they’re all being multiplied by 0. Well, that’s pretty convenient. So we can ignore this entire row. What about the other rows? Well, again, I’m lazy. I like taking shortcuts. So I don’t want to sum over all the remaining nine values. But remember what we’re going to be doing here. We’re going to be multiplying a value for \(x\), like 1, times a value for \(y\), like 1, times the probability of getting that particular pair, \((1,1)\). Well, that’s 0 at this particular location, so that’s not going to contribute anything to the sum either. So not only can we ignore the first row, because \(x\) is 0 there, we can ignore any of the entries where the probability, the joint pmf, is 0. So we don’t have to add up 12 things. We only have to add up 5 things.
So let’s visit each of these remaining five. We’re going to have to multiply the joint probability by the corresponding realizations for \(x\) and \(y\). The entry where \(x\) is 1 and \(y\) is 2 has probability 1/4, so we have 1/4 times 1 times 2. The entry where \(x\) is 2 and \(y\) is 2 also has probability 1/4, so we have 1/4 times 2 times 2. Then there are three entries that remain, and they all have probability 1/8. One corresponds to \(x\) being 1 while \(y\) is 3, so we’d have 1 times 3 times 1/8. One corresponds to \(x\) being 2 while \(y\) is 3, so we have 1/8 times 2 times 3. And one corresponds to \(x\) being 3 while \(y\) is 1, so we have 1/8 times 3 times 1. Factoring out the 1/4 and the 1/8: \[ E[XY] = 1/4 \times (2 + 4) + 1/8 \times (3 + 6 + 3) = 3 \] So again: probability times the value in the row times the value in the column. That’s all we’re doing, going around and adding them up. And if you do the arithmetic, you will discover that the expected value of \(X\) times \(Y\) is 3.
We’re finally ready to calculate the covariance. We have each of the ingredients that we need. Using the shortcut formula, the covariance between \(X\) and \(Y\) is going to be equal to the expected value of the quantity \(X\) times \(Y\), minus the product of the expected value of \(X\) times the expected value of \(Y\). So we have 3 minus 3/2 times 2, which is 0: \[ Cov(X,Y) = E[XY] - E[X]E[Y] = 3 - 3/2 \times 2 = 0 \]
Zero covariance versus independence
Watch: Lecture 10 from 1:00:16.
Now, that’s interesting. I bet you didn’t see that coming. Because remember, we tried to determine whether \(X\) and \(Y\), in this example, were independent, and we determined that they are not. Knowledge of \(Y\) tells you stuff about \(X\), and vice versa. I mean, just look at it. If you know that \(X\) is 0, then you know for sure that \(Y\) is going to be 1. So these random variables contain a lot of information about each other. At the same time, the covariance between \(X\) and \(Y\) is 0. Now, that’s a bit strange. And indeed, what about the correlation between \(X\) and \(Y\)? Well, to figure that out, we would need to work out the standard deviation of \(X\) and the standard deviation of \(Y\). But you know what? Whatever they are, it doesn’t turn out to matter, because the correlation is a fraction, and if the numerator is 0, the whole thing is definitely 0. So we don’t even need to calculate these standard deviations. We know the correlation between \(X\) and \(Y\) is going to be zero as well: \[ Corr(X,Y) = Cov(X,Y)/\left[SD(X)\, SD(Y)\right] = 0 \]
So here we have a bit of a puzzle. We have two random variables that are clearly dependent. They’re not independent, far from it. But they’re uncorrelated. So what’s going on? Well, it turns out that zero covariance, or zero correlation, does not imply independence. A way to think about this is essentially that covariance and correlation are a measure of dependence: they’re a measure of a relationship between two random variables. But that measure is linear, and not all the relationships that are out there in the world are linear. Zero covariance does not imply independence. However, independence does imply zero covariance and zero correlation. So it only runs one way. This is an extremely important thing to remember. You do need to remember, and this is totally crucial, that while zero covariance does not imply independence, independence does imply zero covariance.
For more on the different kinds of dependence between random variables, see my blog post The Independence Zoo.
Sums of random variables
Watch: Lecture 10 from 1:04:00.
We’ve already looked at the linearity of expectation for a single random variable. The key point was that while it is not in general true that the expected value of a function equals the same function applied to the expected value, there was one special exception, and that was a linear function. Well, exactly the same idea works when you have two random variables. In that case, a linear function looks like \(aX + bY + c\), where \(a\), \(b\), and \(c\) are constants: \[ E[aX + bY + c] = aE[X] + bE[Y] + c \] The argument is pretty much the same as the proof for a single random variable. Just like the shortcut formula, you don’t have to memorize the linearity of expectation twice, because the version for one random variable is just a special case of the version for two. In particular, if you take \(b = 0\), you get back exactly the formula that we saw before.
Now we’re going to use the linearity of expectation to derive the shortcut formula for variance: \[ \begin{aligned} Var(X) &= E[(X - \mu)^2] = E[X^2 - 2\mu X + \mu^2]\\ &= E[X^2] - 2\mu E[X] + \mu^2\\ &= E[X^2] - 2\mu^2 + \mu^2\\ &= E[X^2] - \mu^2 \end{aligned} \] The definition of the variance of \(X\) is the expected value of the function \((X - \mu)^2\), where \(\mu\) is the expected value of \(X\). This is just algebra: I’m expanding that quadratic, so I get \(X^2 - 2\mu X + \mu^2\). And why can I break it apart? Because \(X^2\) is a random variable, \(X\) is a random variable, and \(\mu^2\) is a constant. So we have a random variable, minus a constant times another random variable, plus a constant, and we can immediately apply our linearity of expectation rule. We get the expected value of \(X^2\), minus 2 times \(\mu\) times the expected value of \(X\), plus \(\mu^2\). Now, what is the expected value of \(X\)? That’s \(\mu\), which is just a constant. So we have the expected value of \(X^2\) minus \(2\mu\) times \(\mu\), which is \(2\mu^2\), plus \(\mu^2\). In other words, the expected value of \(X^2\) minus \(\mu^2\), where \(\mu\), again, is just our shorthand for the expected value of \(X\). The shortcut formula for covariance, \(Cov(X,Y) = E[XY] - E[X]E[Y]\), which we used above, is proved in an extremely similar way: \[ Cov(X,Y) = E[(X - \mu_X)(Y - \mu_Y)] = E[XY - \mu_X Y - \mu_Y X + \mu_X \mu_Y] = \cdots = E[XY] - E[X]E[Y] \]
The linearity of expectation is extremely powerful. It doesn’t only apply when there are two random variables; it applies when there’s any number of random variables, any number at all. Crucially, it does not matter if these random variables are independent or dependent. It doesn’t matter in the slightest. A particularly common example is a sum of a bunch of random variables, say \(X_1\) up through \(X_n\). We can repeatedly apply the linearity of expectation to break that up into the expected value of \(X_1\), plus the expected value of \(X_2\), all the way up to the expected value of \(X_n\): \[ E[X_1 + X_2 + \cdots + X_n] = E[X_1] + E[X_2] + \cdots + E[X_n] \] And here’s the crucial point: regardless of how these random variables might be related to one another, it doesn’t matter. They can be dependent, they can be independent, they can have any complicated relationship with each other, or no relationship. We can always split up an expectation in this way. In other words, the expected value of a sum is always equal to the sum of the expected values. This is a very powerful idea, and it’s a special case of the linearity of expectation that we’re going to use a lot.
Now, where do we see sums of random variables? We’ve already seen one particularly prominent example: we took the Bernoulli random variable and summed it up to get a binomial. We want to talk about that a bit more now, using some slightly fancier notation. This is the notion of independent and identically distributed random variables. We write \[ X_1, X_2, \ldots, X_n \sim \text{iid Bernoulli}(p) \] You can read the symbol \(\sim\) as “distributes as”. The first i in iid stands for independent, the second stands for identically, and the d stands for distributed. So here’s what it means. If someone tells you that a bunch of random variables are iid, “independent” means that they are all completely independent of each other. Knowing the realization of one doesn’t tell you anything about the others. To put it another way, the joint pmf equals the product of the marginal pmfs. Knowing one of them tells you absolutely nothing about the others, just like independent coin flips. “Identically distributed” means that each of them is the same kind of random variable. In particular, each has the same values for any parameters. What this does not mean is that \(X_1\), \(X_2\), and so on equal each other. This is a crucial thing that students often get confused about. If you flip a coin 50 times, the outcomes need not all be equal. It’s random. What’s the same between the coin flips is the kind of randomness you’re dealing with: in each case, you have a 50% chance of getting heads and a 50% chance of getting tails. More generally, if you have a bunch of iid Bernoulli random variables, each one of them can take on the value 0 or 1, and the probability of getting a 1 is \(p\). The fact that one of them took on a particular value doesn’t tell you anything about the others, and it is not the case that they’re all going to take on exactly the same value as each other. What’s going to be the same is the kind of randomness that each one is subject to, and because of this, they each have the same pmf, CDF, mean, variance, and so on. It’s not true that all random variables are independent and identically distributed, but a lot of interesting things we’re going to look at later in the course have this structure.
So remember how we defined the binomial random variable: as the sum of \(n\) independent Bernoulli random variables, each with probability of success, in other words of a 1, equal to \(p\). The two parameters are \(p\), the probability of success, and \(n\), the number of trials, which is like the number of coin flips, and the support set is \(\{0, 1, 2, \ldots, n\}\). That takes a lot of words to explain. Fortunately, we have some new notation. Let \(X_1\) through \(X_n\) be iid Bernoulli random variables, each with probability of success \(p\), and define \(Y\) as the sum \(X_1 + X_2 + \cdots + X_n\). Then \(Y \sim \text{Binomial}(n, p)\), with pmf \[ p_Y(y) = \binom{n}{y} p^y (1-p)^{n-y} \]
Now, say you wanted to calculate the expected value of a binomial. Using the pmf, that’s going to be a real pain. You’d need to multiply the pmf by \(x\) and sum from \(x = 0\) to \(n\). It’s possible to do that calculation and work out the expected value, but it’s a whole lot of work, and it requires you to know something about binomial coefficients. There’s an easier way, however, because of the way we defined the binomial as a sum of iid Bernoullis. It’s so easy that it’s pretty much trivial. We just use the linearity of expectation. Since we defined \(Y\) as the sum \(X_1 + X_2 + \cdots + X_n\), to calculate the expected value of \(Y\) we just need the expected value of this sum, and we can immediately split that apart using the linearity of expectation. Each of \(X_1\), \(X_2\), all the way up to \(X_n\), is a Bernoulli(\(p\)) random variable, and we’ve already shown that the expected value of a Bernoulli(\(p\)) is just \(p\). So we’re adding up \(p\) over and over. How many times are we adding it up? As many times as there are \(X\)’s, and there are \(n\) of them, so we have \(n\) times \(p\): \[ E[Y] = E[X_1 + X_2 + \cdots + X_n] = E[X_1] + E[X_2] + \cdots + E[X_n] = p + p + \cdots + p = np \] That was easy. The linearity of expectation allowed us to work out that the expected value of a binomial random variable is \(np\) without doing any hard calculations at all. We totally sidestepped any unpleasant math. And this is because the expected value of a sum of random variables is equal to the sum of their expected values. It doesn’t matter what those random variables are: they don’t need to be independent, and they don’t need to be identically distributed. Just to be clear, the step where we split up the expected value used the linearity of expectation, and that did not in any way rely on the fact that the \(X\)’s were iid. It would be true no matter what the \(X\)’s were. The part where we used iid (actually, we didn’t use the i, only the id) was where we said that each of them has the same expected value, \(p\).
The variance of a sum
Watch: Lecture 10 from 1:17:25.
The linearity of expectation always holds: the expected value of a sum always equals the sum of the expected values. And the reason I’m headlining this so very seriously is that there’s a related idea that isn’t true, and I don’t want you to get confused. So this is really important. While it’s true that the expected value of a sum always equals the sum of the expected values, this is not true for variances. The variance of a sum is not, in general, equal to the sum of the variances. Well, if that’s what it’s not, then what is it? We’re going to work it out.
Suppose we want to work out the variance of a sum, \(aX + bY\), where \(a\) and \(b\) are constants. So this is a random variable, \(aX + bY\), and I want to find its variance. So I’m going to take the random variable minus its mean, square that, and take the expected value. Before I regroup, I’m going to use the linearity of expectation to evaluate the inner expectation, the expected value of \(a\) times \(X\) plus \(b\) times \(Y\). I can pull an \(a\) in front, so I have \(a\) times the expected value of \(X\), and I can pull a \(b\) in front, so I have \(b\) times the expected value of \(Y\). Each of these is getting subtracted. So I’ve skipped a few steps here, but what I’ve said is that we have \(aX\) minus \(a\) times the expected value of \(X\), which I’m going to call \(\mu_X\). Similarly, we have \(bY\) minus \(b\) times the expected value of \(Y\), which I’m going to call \(\mu_Y\). So now we have two pieces, nicely put together, and we’re going to square the whole thing. This is just algebra. Maybe it looks nasty; it’s not. Just review your formula for expanding a binomial in this way: we have the first term squared, plus the second term squared, plus twice the product of the first and second terms.
Okay, so what have we got here? \(a^2\): \(a\) is a constant, so \(a^2\) is a constant. And then we have this complicated mess, \((X - \mu_X)^2\). Well, I don’t know what it is, but it’s a function of \(X\), and \(X\) is a random variable, so this is also a random variable. What about the next term? \(b\) is a constant, so \(b^2\) is also a constant. And what is \((Y - \mu_Y)^2\)? Well, I don’t know, but \(Y\) is a random variable, so this whole thing is a random variable, too. Finally, what about the last term? \(2ab\): those are constants, so the whole thing, \(2ab\), is a constant. \(X\) minus \(\mu_X\) times \(Y\) minus \(\mu_Y\): I don’t know what that whole thing is exactly, but I know \(X\) and \(Y\) are random variables, so this product is also a random variable. Now I’m just going to use the linearity of expectation. And the point is that it doesn’t only work with two random variables; it works with many. So \(a^2\) is a constant I’m going to pull in front, \(b^2\) is a constant I’m going to pull in front, \(2ab\) is a constant I’m going to pull in front, and I’m going to split up the expected value across these plus signs. So we have used the linearity of expectation, even though variance is not itself linear: certain steps in this calculation do involve linear functions.
Okay, so what have we got? \(a^2\) times some object, \(b^2\) times some object, \(2ab\) times some other object. Wait a minute, these look familiar. \(X\) minus its mean, squared, expected value of that: that’s just the variance of \(X\). \(Y\) minus its expected value, squared: that’s just the variance of \(Y\). Finally, what’s this? The expected value of \(X\) minus \(\mu_X\) times \(Y\) minus \(\mu_Y\). I’ve got the product of \(X\) minus its mean and \(Y\) minus its mean. This is just the covariance of \(X\) and \(Y\). So we have shown that the variance of \(aX\) plus \(bY\) is equal to \(a^2\) times the variance of \(X\), plus \(b^2\) times the variance of \(Y\), plus \(2ab\) times the covariance of \(X\) with \(Y\): \[ \begin{aligned} Var(aX + bY) &= E\left[\left\{(aX + bY) - E[aX + bY]\right\}^2\right]\\ &= E\left[\left\{a(X - \mu_X) + b(Y - \mu_Y)\right\}^2\right]\\ &= E\left[a^2(X - \mu_X)^2 + b^2(Y - \mu_Y)^2 + 2ab(X - \mu_X)(Y - \mu_Y)\right]\\ &= a^2 E[(X - \mu_X)^2] + b^2 E[(Y - \mu_Y)^2] + 2ab\, E[(X - \mu_X)(Y - \mu_Y)]\\ &= a^2 Var(X) + b^2 Var(Y) + 2ab\, Cov(X,Y) \end{aligned} \]
Now, we’ve already seen a simplified version of this formula, where there’s only \(X\) and there’s no \(Y\). Notice that if you put \(b = 0\), this formula collapses to that one. So again, one more thing you don’t have to memorize. Notice, though, that the variance of this sum depends not only on \(a\) and \(b\); it depends on the covariance between \(X\) and \(Y\). It depends on the linear relationship between \(X\) and \(Y\). Sometimes people like to write this in terms of correlations. \(\sigma_X^2\) is notation for the variance of \(X\), \(\sigma_Y^2\) is notation for the variance of \(Y\), \(\rho\) is notation for the correlation between \(X\) and \(Y\), and \(\sigma_X\) and \(\sigma_Y\) are the standard deviations. This is just using the fact that you can always write the covariance as the product of the correlation and the standard deviations, \(\sigma_{XY} = \rho \sigma_X \sigma_Y\): \[ Var(aX + bY) = a^2 \sigma_X^2 + b^2 \sigma_Y^2 + 2ab \rho \sigma_X \sigma_Y \] So it’s a matter of taste which way you prefer to write things, and you’ll see both.
A good way to practice and make sure that you understand what’s going on would be to go back through this derivation with \(aX + bY + c\), where \(c\) is a constant, and see if anything changes. Hint: nothing is going to change, and you want to think about why. Another special case that’s really worth looking at is the variance of \(X - Y\). So there, \(a\) is 1 and \(b\) is \(-1\). It’s well worth thinking about that example and trying to figure out exactly what’s going on there. Key point: variances definitely can’t be negative, so think carefully about that example.
So the point here is that the variance of a sum is not, in general, going to be equal to the sum of the variances. Even if \(a\) and \(b\) were equal to 1, so we’d have the variance of \(X\) plus \(Y\), that would not, in general, be equal to the variance of \(X\) plus the variance of \(Y\), because there’s this covariance term hanging around. However, if \(X\) and \(Y\) are independent, then, as we explained above, their covariance is going to be zero. What that means is that the last term would disappear. So if \(X\) and \(Y\) are independent, then the variance of \(X\) plus \(Y\) will equal the variance of \(X\) plus the variance of \(Y\): \[ Var(X + Y) = Var(X) + Var(Y) + 2\,Cov(X,Y) = Var(X) + Var(Y) \] Now, I’m going to be very careful here. I’m saying independence, but all you really need is for there to be no covariance. In most of the examples that we’re going to look at, we’re actually going to be assuming independence. So independence is stronger than you really need. Independence definitely implies zero covariance, and zero covariance kills this term. But in order to have the variance of a sum equal the sum of the variances, you would only need there to be no covariance between the random variables. And this holds not just for two random variables; it holds for a whole bunch. So if you have a bunch of independent random variables, then the variance of their sum does equal the sum of the variances. Not true in general, but it’s true in this important special case: \[ Var(X_1 + X_2 + \cdots + X_n) = Var(X_1) + Var(X_2) + \cdots + Var(X_n) \quad \text{if } X_1, \ldots, X_n \text{ are independent} \]
And this brings us to a crucial distinction. No matter what random variables you’ve got, it is always true that the expected value of a sum equals the sum of the expected values. They can be dependent, they can be independent, whatever you want. However, for variance, it is not true in general that the variance of a sum equals the sum of the variances. It is true in a special case, and the special case is when the random variables are independent, or at least uncorrelated.
Watch: Lecture 10 from 1:27:50.
We can finally calculate the variance of a binomial random variable, using what we’ve just figured out about the variance of a sum of independent random variables. Again, the binomial random variable is defined as the sum of a bunch of iid Bernoulli trials, and the first i in iid is independent. Because the \(X\)’s are independent, the variance of their sum, in this special case, does equal the sum of their variances. So going from the first line to the second below, we’ve used independence. That’s crucial. Now we want the variance of \(X_1\), the variance of \(X_2\), and so on. Well, \(X_1\) is just a Bernoulli(\(p\)) random variable, and so are \(X_2\), \(X_3\), all the way up to \(X_n\). And we’ve already calculated the variance of a Bernoulli random variable: \(p(1 - p)\). So we are adding up the same exact thing over and over. The third line is where we’ve used “id”, identically distributed. We have \(p(1 - p)\) plus \(p(1 - p)\), over and over. How many of them are there? As many as there are \(X\)’s. And so, without doing a whole bunch of complicated calculations, we’ve figured out that the variance of a binomial random variable is \(n\) times \(p\) times \(1 - p\): \[ \begin{aligned} Var(Y) &= Var(X_1 + X_2 + \cdots + X_n)\\ &= Var(X_1) + Var(X_2) + \cdots + Var(X_n)\\ &= p(1-p) + p(1-p) + \cdots + p(1-p)\\ &= np(1-p) \end{aligned} \]