Discrete Random Variables I
This page is preparation for Class 2. The sections from The joint pmf through Independent random variables are required. They work through calculations that start from a joint distribution. The lecture video D6 does not work through such calculations, and you will need them for problem sets and exams. The earlier sections are optional. They cover random variables, the pmf, the CDF, and the Bernoulli and binomial random variables, which the lecture videos D2 and D4 also cover. The page as a whole covers the material in lecture videos D2, D4, and D6: random variables, the probability mass function, the cumulative distribution function, the Bernoulli and binomial random variables, and joint, marginal, and conditional distributions. It is a lightly edited version of my lectures for Econ 103, the introductory statistics course I taught at the University of Pennsylvania. It works through each idea with concrete examples. Where a recording exists, each section links to the point in the video where it begins, so you can read, watch, or both.
Expected value, variance, conditional expectation, and covariance are covered on Discrete Random Variables II. That page is required preparation for Class 3.
Notation
My lectures and the slides in the course lecture videos use different symbols for the same objects. I write \(p(x)\) for the probability mass function (pmf) of a discrete random variable \(X\). With two random variables \(X\) and \(Y\), I write \(p_{XY}(x,y)\) for the joint pmf, \(p_X(x)\) and \(p_Y(y)\) for the marginal pmfs, and \(p_{Y|X}(y|x)\) for the conditional pmf of \(Y\) given \(X = x\). The course lecture slides write \(f(x)\), \(f(x,y)\), \(f(y)\), and \(f(y|x)\) for these. They call \(f(x)\) a PMF in D2. In D6 they call the joint, marginal, and conditional distributions PDFs, even when the random variables are discrete.
Random variables
There is no video for this section or the next three. They follow the slides of my Econ 103 Lectures 7 and 8.
A random variable is neither random nor a variable. A random variable \(X\) is a fixed function that assigns a number to each basic outcome of a random experiment. A realization \(x\) is a particular numeric value that a random variable could take on. We write \(\{X = x\}\) to refer to the event that the random variable \(X\) took on the value \(x\). The support set, or support, is the set of all possible realizations of a random variable.
We use capital Latin letters for random variables, for example \(X\), \(Y\), \(Z\), and the corresponding lowercase letters for their realizations, for example \(x\), \(y\), \(z\). You can think of a random variable as a machine that spits out random numbers: although the machine is deterministic, its inputs, the outcomes of a random experiment, are not.
For example, the coin flip random variable assigns numeric values to the random experiment of flipping a fair coin once: heads is assigned 1 and tails 0. Its realizations are the numbers 0 and 1, so its support set is \(\{0, 1\}\).
There are two kinds of random variables. A discrete random variable has a discrete support set, for example \(\{0, 1, 2\}\) or \(\{\ldots, -2, -1, 0, 1, 2, \ldots\}\). A continuous random variable has a continuous support set, for example \([-1, 1]\) or \(\mathbb{R}\). We start with the discrete case, since it’s easier, but most of the ideas we learn will carry over to the continuous case.
The probability mass function
The probability mass function (pmf) is a function that gives \(P(X = x)\) for any realization \(x\) in the support set of a discrete random variable \(X\). We use the following notation for the pmf: \[ p(x) = P(X = x) \] Plug in a realization \(x\), get out a probability \(p(x)\). For the coin flip random variable, \(p(0) = 1/2\) and \(p(1) = 1/2\).
Whenever you write down the pmf of a random variable, it is crucial to also write down its support set. Recall that this is the set of all possible realizations of the random variable. Outside of the support set, all probabilities are zero. In other words, the pmf is zero outside the support.
If \(p(x)\) is the pmf of a random variable \(X\), then
- \(0 \leq p(x) \leq 1\) for all \(x\)
- \(\displaystyle\sum_{\text{all } x} p(x) = 1\)
where “all \(x\)” is shorthand for “all \(x\) in the support of \(X\)”.
The cumulative distribution function
The cumulative distribution function (CDF) gives the probability that a random variable \(X\) does not exceed a specified threshold \(x_0\), as a function of \(x_0\): \[ F(x_0) = P(X \leq x_0) \] This definition is the same for continuous random variables. The threshold \(x_0\) is allowed to be any real number. In particular, it doesn’t have to be in the support of \(X\)!
For a discrete random variable, you sum the pmf to get the CDF: \[ F(x_0) = \sum_{x \leq x_0} p(x) \] Why? The events \(\{X = x\}\) are mutually exclusive, so we sum to get the probability of their union for all \(x \leq x_0\): \[ F(x_0) = P(X \leq x_0) = P\left( \bigcup_{x \leq x_0} \{X = x\} \right) = \sum_{x \leq x_0} P(X = x) = \sum_{x \leq x_0} p(x) \] For the coin flip random variable, with \(p(0) = p(1) = 1/2\), \[ F(x_0) = \left\{\begin{array}{ll} 0, & x_0 < 0\\ 1/2, & 0 \leq x_0 < 1\\ 1, & x_0 \geq 1 \end{array}\right. \]
CDFs have the following properties, which are also true for continuous random variables:
- \(\lim_{x_0 \rightarrow \infty} F(x_0) = 1\)
- \(\lim_{x_0 \rightarrow -\infty} F(x_0) = 0\)
- Non-decreasing: \(x_0 < x_1 \Rightarrow F(x_0) \leq F(x_1)\)
- Right-continuous
Since \(F(x_0) = P(X \leq x_0)\), we have \(0 \leq F(x_0) \leq 1\) for all \(x_0\).
The Bernoulli random variable
The Bernoulli random variable is a generalization of the coin flip. Its support set is \(\{0, 1\}\), where 1 is traditionally called “success” and 0 “failure”. Its pmf and CDF are \[ p(0) = 1 - p, \quad p(1) = p, \qquad F(x_0) = \left\{\begin{array}{ll} 0, & x_0 < 0\\ 1 - p, & 0 \leq x_0 < 1\\ 1, & x_0 \geq 1 \end{array}\right. \] We write \(X \sim \text{Bernoulli}(p)\) to mean that \(X\) is a Bernoulli random variable with \(P(X = 1) = p\) and \(P(X = 0) = 1 - p\). The tilde is read “distributes as”. A parameter is any constant that appears in the definition of a random variable, here \(p\).
The binomial random variable
Watch: Lecture 9 from 27:18.
The Bernoulli is going to be the building block for lots of more complicated and interesting random variables. Here is the first of those. It is called the binomial random variable, and it is what we get if we sum up a bunch of independent Bernoulli random variables. And I literally mean that. Imagine doing your little Bernoulli random variable experiment, which is a possibly unfair coin flip: it could have any probability of heads. Do it a bunch of times, and you’re going to get a bunch of zeros and ones. Sum them up, and whatever answer you get, that is a realization of a binomial random variable.
So let’s dive into some of the details. We’ll start with a really simple case, a fair coin flip. Suppose we flip a fair coin three times. What is the probability that we get exactly two heads? This shouldn’t be too hard for you to figure out, and it’s a good review of some of the probability material from earlier, so try to think about what the answer might be.
The best way to solve these kinds of problems is to make sure we know all the basic outcomes that satisfy the condition. We’re flipping a fair coin three times, so the sample space of this experiment is all of the sequences of three heads and tails in a row, and we’re interested in the ones that contain exactly two heads. There are three, because we have a choice of where to put the one tails: it could be at the end, in the middle, or at the front. So three basic outcomes make up this event: \(\{HHT, HTH, THH\}\). Since the coin flips are independent, the probability of each of these basic outcomes is just a half times a half times a half, or 1/8. Since they are all mutually exclusive, we can sum to get the overall probability of this event: 1/8 plus 1/8 plus 1/8 is 3/8. So the probability of getting exactly two heads when you flip a fair coin three times is 3/8, which equals 0.375.
Okay, let’s now make this a little bit more complicated. Suppose we flip an unfair coin three times. This unfair coin does not have a 50% chance of coming up heads; it only has a 1 in 3 chance. Same question: what is the probability that we get exactly two heads? In the preceding example, we leaned heavily on the fact that the basic outcomes were equally likely. That’s not going to be the case here. But we’re still interested in the same sequences. There are still three ways to get exactly two heads, because they correspond to where you put the lone tails. So now we need to calculate each of these probabilities separately, because they involve something slightly different, right? Well, maybe not. Let’s take a look. Heads, heads, tails: that’s 1/3 times 1/3 times \((1 - 1/3)\), which is 2/27. Well, gosh, take a look: \[ P(HHT) = (1/3)^2(1 - 1/3) = 2/27, \qquad P(HTH) = 2/27, \qquad P(THH) = 2/27 \] It sort of seemed like the situation was going to get a lot more complicated when I said the probability of heads was 1/3. But actually, these sequences all have exactly the same probabilities. So let me be clear about what changed. In the previous example, every possible sequence of three heads and tails was equally likely. Here that’s not true, because tails are more likely than heads. However, the three outcomes that we’re interested in, namely the ones with exactly two heads, all still have the same probability as one another, because they each have the same number of heads. The fact that heads and tails have different probabilities comes out in the wash. So we have \(2/27 + 2/27 + 2/27 = 6/27\), which is \(2/9\), or about 0.22.
Perhaps you’re starting to see a pattern here. If not, we’ll do one more. Suppose we flip an unfair coin four times, where again the probability of heads is 1/3. What is the probability that we get exactly two heads? Let’s try to enumerate the basic outcomes that satisfy this condition. There are six of them: HHTT, HTHT, HTTH, THHT, THTH, and TTHH. The reason I know that there are six of them is that this is equivalent to choosing where to put the heads in a sequence of four slots. We’ve got a pool of four slots, and we’re trying to choose a committee of two slots. So there are \(\binom{4}{2}\) ways to allocate the two heads, and once you’ve done that, you’re finished, because the rest of the slots are going to be filled with tails. So we would have to calculate the probability of each of these six basic outcomes and then sum them up. So there will be six separate probability calculations, right? Well, no, actually, because once again, just as we saw in the preceding example, these all have the same numbers of heads and tails. Each of them has probability 1/3 times 1/3 times 2/3 times 2/3. It’s not true that every sequence of heads and tails in this experiment is equally likely. But these particular ones all have the same probability, so we just need to calculate the probability of one of them and multiply it by six: \[ \binom{4}{2} (1/3)^2 (2/3)^2 \] We have \((1/3)^2\) because we have two heads and the probability of heads is 1/3, and \((2/3)^2\) because we have two tails and the probability of tails is 2/3. Finally, we multiply the whole thing by the number of different ways that we could have gotten exactly two heads, which is \(\binom{4}{2} = 6\). If you follow that reasoning to its natural conclusion, you are going to end up with the probability mass function of the binomial random variable. I encourage you to go back to the coin flip example and try working out the probability for different numbers of flips and different probabilities of heads and tails. You can also try different numbers of heads. When you do that enough, you’ll notice that all the answers you get are special cases of the formula below.
Watch: Lecture 9 from 36:08.
This is how we end up with the binomial random variable. We define \(X\) as the sum of \(n\) independent Bernoulli trials, each with probability of success \(p\). So we’re going to repeatedly flip some kind of coin, which might not be fair. We’re going to do it \(n\) times, and we’re going to sum up the results, where heads count as 1. Whatever we end up with, it’s random: it’s basically a random sum. And that sum is important enough that we give it a special name. We call it the binomial random variable, and we write \(X \sim \text{Binomial}(n, p)\).
It has two parameters, because we need to fully specify exactly what we did in the background with all those separate coin flips. We need to specify how many times we flipped the coin, and what the probability of heads is. Or, to put it in slightly more precise, although less concrete, terminology: how many independent Bernoulli trials did we carry out, and what was the probability of success on each of those trials? So we have two parameters here, \(p\) and \(n\), and we need to know both in order to know the probabilities of the different outcomes of this random variable.
Now, what is the support going to be? We’re going to flip a coin \(n\) times, and every time we get heads, we’re going to call that a 1. We’re going to count up the number of heads. Say we did it 10 times. What’s the maximum number of heads you could get in 10 coin flips? It’s 10, so the largest possible realization you could get for a binomial random variable is \(n\). What’s the smallest? If you flip a coin 10 times, what’s the smallest number of heads you could get? It’s clearly 0. You could, although it’s unlikely, get 10 tails in a row. And by the same reasoning, you could get any number of heads between 0 and 10. Generalizing that to \(n\) coin flips, we see that the support of the binomial random variable is \(\{0, 1, 2, \ldots, n\}\). And by an argument that is essentially identical to what I just showed you for a very concrete example, this is the pmf: \[ p(x) = \binom{n}{x} p^x (1-p)^{n-x} \]
I’ve found that this formula looks intimidating to students, so let’s remember what a pmf is. This is crucial. When you see a complicated formula, don’t try to understand it all at once. Ask yourself, what is it a formula for? \(p(x)\) is a probability mass function, so you take any number in the support and plug it in. So \(x\) could be anything from 0 up to \(n\), and what we’re asking is: what is the probability that this random variable takes on the value \(x\)? You could plug in 1: \(p(1) = \binom{n}{1} p^1 (1-p)^{n-1}\). Or you could plug in 2: \(p(2) = \binom{n}{2} p^2 (1-p)^{n-2}\). So \(x\) just denotes which realization we’re calculating the probability for, and \(n\) and \(p\) are the parameters. \(p\) is the probability of getting a success on one of the underlying Bernoulli trials, and \(n\) is the total number of trials. So if we had a Binomial(10, 1/2), you’d plug in \(n = 10\) and \(p = 1/2\). In the example that just preceded this, we calculated \(p(2)\) in a setting where \(n\) was 4 and \(p\) was 1/3. So notice: to get a total of \(x\), you got \(x\) successes, which gives \(p^x\), and the rest of your trials, \(n - x\) of them, were failures, which gives \((1-p)^{n-x}\). And then there are all the different ways to arrange those in a sequence.
This is not immediately intuitive. You have to think about it. You shouldn’t just assume you can look at this and know what it is. I seriously, seriously, seriously encourage you to plug in some numbers. Choose an \(n\). Choose a \(p\). Write down what the support is for that choice of \(n\), and try calculating some examples, just like we did above. You will eventually see what’s going on here. All this formula uses is the basic probabilistic reasoning we’ve already developed; it’s just that we’re writing it in a slightly more abstract format. Bear in mind that the reason we use abstraction is not to make life more difficult. It’s actually to make life simpler, because once we realize that we’re dealing with a binomial random variable, we’ve got all kinds of results we can use, and we don’t have to redo all the calculations as though it were a special case. So the point of abstraction is to get more power. But in order to cope with abstraction successfully, it’s really helpful to look at some concrete examples until you start to see the pattern. So again, do not just try to memorize this. Try to understand where it is coming from.
So we’ve got the pmf of a binomial random variable. What about the CDF? I’ve made an app for you: fditraglia.shinyapps.io/binom_cdf. It shows the pmf and the CDF for any binomial random variable that you want. You can change \(n\) between 1 and 20, and you can choose the probability of success. You can see the different shapes that you get for the pmf on the left and for the CDF on the right. And you can see exactly how you get from the pmf to the CDF by changing the threshold \(x_0\) and watching the bars light up as we climb up the staircase of the CDF. So spend a bit of time playing with this; it’ll give you some good intuition for what the binomial random variable looks like. If you set \(n = 1\), you’re back to the familiar case of a Bernoulli random variable. So a Bernoulli random variable is just a special case of the binomial, the special case where \(n = 1\), and bearing that in mind is very helpful.
So, to summarize: a binomial random variable counts up the total number of successes, that is, ones, in \(n\) independent Bernoulli trials, each of which has probability of success \(p\).
The joint pmf
Watch: Lecture 9 from 1:07:00.
We’ve looked at discrete random variables, but we’ve only looked at them one at a time. What we’re now going to do is look at more than one random variable at once. This is going to allow us to look at things like covariance and correlation: how are these things related to one another? To do this, we’re going to start simple. We’re going to start again with discrete random variables, and we’re going to define an object called the joint probability mass function. We’ve already talked about the probability mass function, but now we need a probability mass function that’s a little bit more general, one that allows us to look at two random variables at once.
So let’s suppose that we’ve got \(X\) and \(Y\), which are two discrete random variables. We’re going to define a function called the joint probability mass function. It’s defined as follows: \[ p_{XY}(x,y) = P(X = x \cap Y = y) \] I’m going to use these subscripts to make it clear that the first argument refers to \(X\) and the second one refers to \(Y\). This is just to help us remember that. You’ll see in some of the problems we work out that remembering which is which turns out to be helpful. What this thing does is you plug in any pair of realizations \(x, y\), where little \(x\) is in the support of the random variable \(X\) and little \(y\) is in the support of the random variable \(Y\), and this function gives you the probability that \(X\) equals little \(x\) and \(Y\) equals little \(y\). So this is going to give us a joint probability: the probability of the intersection of these two events, that the random variable \(X\) takes on this value and that the random variable \(Y\) takes on that value. You plug in two numbers, a realization for \(X\) and a realization for \(Y\), and you get out a probability.
We can express a joint probability mass function in a table. Here is a very simple example where I have a random variable \(X\) that can take on the values 0, 1, 2, 3, and a random variable \(Y\) that can take on the values 1, 2, or 3. I’ve just made up this pmf. I just invented it.
| \(Y = 1\) | \(Y = 2\) | \(Y = 3\) | |
|---|---|---|---|
| \(X = 0\) | 1/8 | 0 | 0 |
| \(X = 1\) | 0 | 1/4 | 1/8 |
| \(X = 2\) | 0 | 1/4 | 1/8 |
| \(X = 3\) | 1/8 | 0 | 0 |
So this particular pair of random variables has probability 1/8 that \(X\) is 0 at the same time as \(Y\) is 1, it has probability 1/4 that \(X\) is 2 at the same time as \(Y\) is 2, it has probability 0 that \(X\) is 3 at the same time as \(Y\) is 2, and so on. Each of these numbers in the table is a number from the joint pmf, so the whole joint pmf is expressed here in a table.
Quick question for you: what is the joint probability that \(X\) equals 1 and \(Y\) equals 2? All we have to do is go to the table and look where \(X\) is 1 and \(Y\) is 2. We have a quarter. So the probability that \(X\) is equal to 1 at the same time as \(Y\) is equal to 2 is 1/4. What about this one: what is \(p_{XY}(2,1)\)? Notice I’ve reversed the numbers. The subscripts are the hint. The first element refers to \(X\); the second element refers to \(Y\). Once again, all we have to do is look at the point in the table where \(X\) is 2 and \(Y\) is 1, and we see that the probability is zero. The probability that \(X\) takes on the value 2 at the same time as \(Y\) takes on the value 1, for this particular pair of random variables that I invented, is zero: \[ p_{XY}(1,2) = P(X = 1 \cap Y = 2) = 1/4, \qquad p_{XY}(2,1) = P(X = 2 \cap Y = 1) = 0 \]
So that’s very simple. Joint probability mass functions are not that different from the plain old probability mass functions we’ve already looked at. They spit out probabilities. You plug in pairs of realizations, and you get out probabilities. Probabilities have to be between 0 and 1, and if you sum over all the possible pairs, you need to get 1:
- \(0 \leq p_{XY}(x,y) \leq 1\) for any pair \((x,y)\)
- \(\sum_x \sum_y p_{XY}(x,y) = 1\)
This double sum means you’re going to sum over all the possible pairs \(x\) and \(y\). The point here is that the probability that something happens is 1. All the pairs \(x, y\) in the table are mutually exclusive and collectively exhaustive events, so if you sum over the whole table, you’d better get 1. If something does not satisfy these requirements, then it is not a valid joint probability mass function.
So, going back to the example I just showed you, is this a valid joint pmf? First of all, let’s visit every single pair in the support and look at the probabilities. None of them are negative, and none of them are bigger than 1, so we’ve satisfied the first condition. Now let’s go through and add them all up, column by column: \(1/8 + 0 + 0 + 1/8\) is 1/4; \(0 + 1/4 + 1/4 + 0\) is 1/2; \(0 + 1/8 + 1/8 + 0\) is 1/4. And indeed, \(1/4 + 1/2 + 1/4\) is 1. So this is a valid joint pmf.
Joint versus marginal pmfs
Watch: Lecture 9 from 1:13:30.
The next crucial concept here is the difference between a joint and a marginal pmf. A moment ago, I referred to the pmfs we’ve looked at before as plain old pmfs. That’s just because I hadn’t introduced this new piece of terminology. When you’ve got more than one random variable floating around, you’ve got to be specific about what you mean when you say pmf. When we say joint pmf, we’re talking about \(p_{XY}(x,y)\), which I’ve just spent a couple of minutes discussing. But we could also ask questions, not about the joint probabilities that \(X\) and \(Y\) are doing some particular things at the same time, but about \(X\) by itself or \(Y\) by itself. To indicate that that’s what we mean, I’ll have a single subscript: \[ p_X(x) = P(X = x), \qquad p_Y(y) = P(Y = y) \] So we have marginal pmfs, which is when you look at one random variable at a time, and the joint pmf, which is when you look at both at once.
A crucial point is that it turns out the joint pmf contains strictly more information than the marginal pmfs. In other words, if you know the joint pmf, you can always calculate the marginal pmfs. But even if you know both of the marginal pmfs, you can’t calculate the joint. The reason for this, intuitively, is that when you learn about \(X\) in isolation and you learn about \(Y\) in isolation, you never get to learn about whether \(X\) and \(Y\) are dependent on each other, and if so, how. But when you look at the joint pmf, you explicitly get to see what \(X\) is doing when \(Y\) is doing this. So you can’t calculate the joint from the marginals, but you can calculate the marginals from the joint.
And to calculate the marginals from the joint, it’s actually very easy. All you have to do to get the marginal pmf for \(X\) evaluated at a particular realization little \(x\) is to sum up the joint pmf over all possible \(y\)’s. Similarly, if you want to get the marginal pmf for \(Y\) evaluated at a particular realization little \(y\), all you’ve got to do is sum up the joint pmf over all possible \(x\)’s: \[ p_X(x) = \sum_{\text{all } y} p_{XY}(x,y), \qquad p_Y(y) = \sum_{\text{all } x} p_{XY}(x,y) \] If you’re curious about where this comes from, here’s the argument, which is just based on mutually exclusive events and being able to sum their probabilities: \[ p_Y(y) = P(Y = y) = P\left( \bigcup_{\text{all } x} \left\{ X = x \cap Y = y \right\} \right) = \sum_{\text{all } x} P(X = x \cap Y = y) = \sum_{\text{all } x} p_{XY}(x,y) \] If you don’t like that, don’t worry about it. Let me show you this in the table. I think it’s going to make a lot more sense.
The reason they’re called marginals, and I’m not kidding, is because you calculate them by summing into the margins of the table. So what do I mean? Say I wanted to calculate the probability that \(X\) is 0. Well, there are three different ways \(X\) could be 0. It could be that \(X\) is 0 and \(Y\) is 1. It could be that \(X\) is 0 and \(Y\) is 2. Or it could be that \(X\) is 0 and \(Y\) is 3. Notice that these pairs, \((0,1)\), \((0,2)\), and \((0,3)\), are mutually exclusive. They can’t happen at once. If you’re at the pair \((0,1)\), then you’re definitely not at the pair \((0,2)\), because \(Y\) is 1, not 2. So that means that if we want to get the probability that \(X\) is 0, all we’ve got to do is sum up the probabilities of each of the different ways it could be 0. And those ways correspond to the different values of \(Y\). This is exactly the logic from the argument above; it’s just probably a bit easier to see in the example. So an eighth plus zero plus zero is an eighth.
We can do the same thing to get the probability that \(X\) is 1. There are three ways that could happen. We could be at \((1,1)\), \((1,2)\), or \((1,3)\). These are all mutually exclusive, so we can sum their probabilities: 0 plus 1/4 plus 1/8 is 3/8. Similarly, for \(X\) is 2, 1/4 plus 1/8 is 3/8. Similarly, for \(X\) is 3, we just have 1/8: \[ \begin{aligned} p_X(0) &= 1/8 + 0 + 0 = 1/8\\ p_X(1) &= 0 + 1/4 + 1/8 = 3/8\\ p_X(2) &= 0 + 1/4 + 1/8 = 3/8\\ p_X(3) &= 1/8 + 0 + 0 = 1/8 \end{aligned} \] So notice, in order to get these probabilities for \(X\), we just summed across the rows into the margin. These are the marginal probabilities for \(X\): this is the marginal pmf. The probability that \(X\) is 0 is 1/8, the probability that \(X\) takes on the value 1 is 3/8, the probability that \(X\) takes on the value 2 is 3/8, and the probability that \(X\) takes on the value 3 is 1/8. So that’s it. These are the marginals for \(X\), and we got them from the joint for \(X\) and \(Y\). A good sanity check on your results is that, if you did it right, this should be a legitimate pmf by itself, and so it had better add up to 1. And we see that it does.
We can do exactly the same thing for \(Y\), but in this case you’re going to be summing down the columns, because you’re going to say: for a fixed value of \(y\) that I’m interested in, what are all the different \(x\)’s that I could have going along with it? So why don’t you give that a try? Why don’t you try calculating the marginal pmf of \(Y\) at each of the possible values?
Okay, so here we go. If we want to come up with the probability that \(Y\) is 1, there are four ways that could happen: \(Y\) is 1 when \(X\) is 0, when \(X\) is 1, when \(X\) is 2, and when \(X\) is 3. These are all mutually exclusive, so we can sum the probabilities. An eighth plus an eighth is a fourth. Exactly the same logic for \(Y\) equal to 2: a fourth plus a fourth is a half. And for \(Y\) equal to 3 as well: an eighth plus an eighth is a fourth. \[ \begin{aligned} p_Y(1) &= 1/8 + 0 + 0 + 1/8 = 1/4\\ p_Y(2) &= 0 + 1/4 + 1/4 + 0 = 1/2\\ p_Y(3) &= 0 + 1/8 + 1/8 + 0 = 1/4 \end{aligned} \] So we’ve just found out that the marginal pmf for \(Y\) is the following: \(Y\) has probability 1/4 of taking on the value 1, probability 1/2 of taking on the value 2, and probability 1/4 of taking on the value 3. And again, as a sanity check, we can add these three numbers up, and we do get 1. Here is the table with both sets of marginals in the margins:
| \(Y = 1\) | \(Y = 2\) | \(Y = 3\) | \(p_X(x)\) | |
|---|---|---|---|---|
| \(X = 0\) | 1/8 | 0 | 0 | 1/8 |
| \(X = 1\) | 0 | 1/4 | 1/8 | 3/8 |
| \(X = 2\) | 0 | 1/4 | 1/8 | 3/8 |
| \(X = 3\) | 1/8 | 0 | 0 | 1/8 |
| \(p_Y(y)\) | 1/4 | 1/2 | 1/4 | 1 |
Conditional pmfs
Watch: Lecture 10 from 1:30.
The next ingredient, when looking at discrete random variables, is to define what is called a conditional probability mass function. A conditional pmf is the probability that the random variable \(Y\) takes on the realization little \(y\), given that we know that the random variable \(X\) has taken on the realization little \(x\): \[ p_{Y|X}(y|x) = P(Y = y | X = x) \] Again, I’m using these subscripts not because I want to make the notation complicated, but because I want to make it very clear, in examples where we plug in numbers, which number corresponds to what. The bar here, just like we’ve seen when we looked at conditional probability, tells us what we’re conditioning on. The thing after the bar is the thing we’re conditioning on, and the thing before the bar is the thing whose probability we’re calculating. So this says that we’re going to look at a particular realization, little \(y\), and determine the probability of that realization happening, given a particular realization that we’ve already observed for \(X\). The reason that we’re going to want to do this is that it will allow us to see how the probability mass function of \(Y\) changes if you know \(X\). How does the distribution of \(Y\), how does the randomness inherent in \(Y\), change with \(X\)?
That’s the definition. It isn’t, however, the formula, because the conditional pmf is defined in terms of some objects that you already know. The key point is to remember that we defined conditional probability as the ratio of the probability of an intersection to the probability of the thing we’re conditioning on. And indeed, that same idea holds here, because these random variables \(X\) and \(Y\) taking on particular values, well, those are nothing other than events. So if you want, you can call \(Y = y\) the event \(A\), and you can call \(X = x\) the event \(B\). Then this says the probability of \(A\) given \(B\), and you already know how to calculate that: it’s the probability of the intersection of \(A\) and \(B\) divided by the probability of \(B\). And using the notation of the joint probability mass function and the marginal, we see exactly how to write down the conditional pmf: \[ p_{Y|X}(y|x) = P(Y = y | X = x) = \frac{P(Y = y \cap X = x)}{P(X = x)} = \frac{p_{XY}(x,y)}{p_X(x)} \] It is the ratio of the joint to the marginal, and we can do this in either direction. So we can work out the conditional pmf of \(Y\) given \(X\), as above, or the conditional pmf of \(X\) given \(Y\): \[ p_{X|Y}(x|y) = \frac{p_{XY}(x,y)}{p_Y(y)} \] In each case, the numerator is the same: that’s just the joint pmf of \(X\) and \(Y\). The only thing that changes is the denominator, because we’re changing what we condition on. And this is exactly the same as what we did looking at two events \(A\) and \(B\). We know that if you change what you’re conditioning on, it changes the denominator, but it leaves the numerator unchanged, because the probability of an intersection does not depend on the order in which you write the two events in the intersection.
Watch: Lecture 10 from 6:00.
Let’s try doing some calculations by continuing with our numerical example. We’re going to try to figure out what the pmf of \(Y\) is going to be when we condition on the fact that \(X\) has taken on the value 2. In order to do that, we’re going to need to use the marginal probability that \(X\) equals 2. To work out the probability that \(X\) will take on the value 2, we sum into the margins of the table: 0 plus 1/4 plus 1/8, which gives us 3/8. So this is the event we’re conditioning on, an event that has probability 3/8.
So to work out the probability that \(Y\) is 1, given that \(X\) is 2, what do we need to do? Well, we’re going to take the joint probability that \(X\) is 2 at the same time as \(Y\) is 1, and divide that by the marginal probability that \(X\) takes on the value 2. We already know what the denominator is: it’s 3/8, the probability that \(X\) is 2. Let’s try to find the numerator. We need \(X\) to be 2 and \(Y\) to be 1. Well, that probability is 0. So actually, we didn’t even need to bother dealing with the denominator, because the fraction is going to be 0 as long as the numerator is 0. So that’s pretty straightforward, and we can use exactly the same idea to work out the probability that \(Y\) takes on some other value, given that \(X\) is 2. To get the probability that \(Y\) is 2 given that \(X\) is 2, we’re going to take the joint probability that \(X\) and \(Y\) are both 2, and if we go into the table, that’s a fourth, and divide that by the marginal probability that \(X\) takes on the value 2, and that’s three-eighths. So we get a fourth over three-eighths, which is two-thirds. And finally, for the probability that \(Y\) is 3 given that \(X\) is 2 (again, notice the denominator stays the same for all these calculations, because we’re holding the event that we’re conditioning on, \(X = 2\), fixed), we first calculate the joint probability that \(X\) is 2 at the same time as \(Y\) is 3. From the table we see that’s one-eighth, and an eighth over three-eighths is a third: \[ \begin{aligned} p_{Y|X}(1|2) &= \frac{p_{XY}(2,1)}{p_X(2)} = \frac{0}{3/8} = 0\\[4pt] p_{Y|X}(2|2) &= \frac{p_{XY}(2,2)}{p_X(2)} = \frac{1/4}{3/8} = 2/3\\[4pt] p_{Y|X}(3|2) &= \frac{p_{XY}(2,3)}{p_X(2)} = \frac{1/8}{3/8} = 1/3 \end{aligned} \]
So you can see it’s not too difficult to do these calculations, but I want to point out something else, which is that there’s kind of a nice graphical way to see what’s going on if you look at the table. The marginal probability that \(X\) takes on the value 2 is just the sum of the three numbers in the \(X = 2\) row. What are we doing here? We’re just taking each of those numbers in turn and dividing it by their sum. So all that we’re doing is rescaling the probabilities that \(Y\) takes on the values 1, 2, and 3 in a fixed row for \(X\) by the total amount of probability in that row. Conditioning on \(X\) being 2 means we’re fixing the row \(X = 2\). That’s the only row we’re going to consider. Then we’re going to look at all the different probabilities for the different values \(Y\) can take on within that row of the joint pmf, and we’re just going to rescale them so they add up to 1, because notice, if you add 0 to 1/4 to 1/8, you don’t get 1. A conditional probability mass function is still a probability mass function. As such, if you sum up the probabilities of all the different realizations, you should get 1. So we know that these three numbers don’t add up to 1; they don’t form a legitimate probability mass function. What would we need to divide them by to make sure that they sum up to 1? Well, the answer is the sum of all three. That’s exactly the thing you would need to divide by to make sure that if you add them all up, you get 1. And that’s exactly what we’re doing here. So, kind of like the Venn diagram picture of conditional probability, there’s a very simple, intuitive idea going on here. We’re sort of reducing the space of possibilities by using a restriction, namely \(X\) is 2. We look only at the probabilities that correspond to \(X\) being 2, but now, since we’ve reduced the list of possibilities we’re considering, our probabilities no longer add up to 1, so we need to find a way to rescale them. And that’s all that’s happening here. So if you understand how this worked before for simple events \(A\) and \(B\), this shouldn’t present any problem.
So you’ve now seen a joint probability mass function, a marginal, and a conditional, and you’ve seen how they relate to each other in a simple example. So now it’s your turn. We’ve calculated the conditional probabilities for \(Y\) given a particular value for \(X\). Now I would like you to do the reverse: calculate the conditional probability mass function of \(X\) given a particular value for \(Y\). In particular, calculate the probability that \(X\) is 0, 1, 2, and 3, given that \(Y\) is 2.
Watch: Lecture 10 from 12:20.
The logic here is pretty much the same as in the one I just worked out for you. We’re going to be conditioning on \(Y\) being 2, which corresponds to the middle column of the table. All we’re doing is restricting our attention to the part of the table where \(Y\) is 2, and then we’re going to take each of the joint probabilities and rescale it to make sure that the probabilities add up to 1. So what is the probability by which we need to rescale? Well, that’s just the probability that \(Y\) equals 2. That’s 1/2, because we know that to get a marginal probability, we sum into the margins of the table: 0 plus 1/4 plus 1/4 plus 0 is 1/2. So the probability that \(X\) is 0 given \(Y\) is 2 is 0 over 1/2, which is just 0. The probability that \(X\) is 1 given that \(Y\) is 2 is 1/4 over 1/2, which is just 1/2. The probability that \(X\) is 2 given that \(Y\) is 2 is 1/4 over 1/2, which is again 1/2. And finally, the probability that \(X\) is 3 given that \(Y\) is 2 is 0: \[ p_{X|Y}(0|2) = 0, \qquad p_{X|Y}(1|2) = \frac{p_{XY}(1,2)}{p_Y(2)} = \frac{1/4}{1/2} = 1/2, \qquad p_{X|Y}(2|2) = 1/2, \qquad p_{X|Y}(3|2) = 0 \] This is a pmf. It tells you the probabilities of all the realizations that \(X\) can take on, 0, 1, 2, 3, given a particular value for \(Y\). And indeed, if we add up these numbers, 0 plus 1/2 plus 1/2 plus 0, we do get 1. So these problems are very nice in the sense that it’s easy to check your work.
Independent random variables
Watch: Lecture 10 from 15:00.
When we studied probabilities in terms of simple events, from die rolls and coin tossing, we talked about the idea of independence. Two events \(A\) and \(B\) are independent if the probability of the intersection of \(A\) and \(B\) happens to be equal to the probability of \(A\) multiplied by the probability of \(B\). That’s not true for all events, but when it is true, it means those two events are independent. We also talked about how independence can be thought of in terms of a conditional probability. If \(A\) and \(B\) are independent events, then knowing that \(A\) has happened doesn’t tell you anything new about the event \(B\), and vice versa: knowing that \(B\) has happened doesn’t tell you anything new about the event \(A\). So an equivalent definition of independence has to do with a conditional probability being equal to the plain old probability. If the probability of \(A\) given \(B\) equals the probability of \(A\), then \(A\) and \(B\) are independent events.
Now we’re going to use exactly the same reasoning to define independent random variables. We say that two discrete random variables are independent if and only if their joint probability mass function is equal to the product of their marginal probability mass functions: \[ p_{XY}(x,y) = p_X(x)\, p_Y(y) \quad \text{for all } x \text{ in the support of } X \text{ and all } y \text{ in the support of } Y \] So what this says is that the joint pmf is equal to the product of the two marginal pmfs for any pair \(x, y\) that you choose to put in there. You can visit any pair of realizations \(x, y\) in the whole support, and if this relationship holds for all the pairs, not just for some, but for all of them, then we say that these random variables are independent.
Now remember, I said that you cannot, in general, derive a joint pmf from knowledge of the marginal pmfs alone. The special case in which you can is if the two random variables are independent. This does not contradict what I said before, because what I said was that the joint probability mass function tells you about the dependence between the two random variables, and that’s the information that’s not contained in the marginals. Well, guess what? If the two random variables are actually independent, that means there’s no dependence between them. So that means there’s no information in the joint probability mass function beyond what’s in the marginals. And that’s pretty much what it means to say that we get the joint by multiplying the marginals together.
And again, just like in the case of simple events, where we defined independence in terms of a conditional probability, it follows from our definition of a conditional pmf that another way to characterize independence is to require that both of the conditional probability mass functions equal the corresponding marginals. So if the probability mass function of \(Y\) given \(X\) is just equal to the plain old marginal probability mass function of \(Y\), and the probability mass function of \(X\) given \(Y\) is also just equal to the plain old probability mass function of \(X\), for all the different points in the support, then these random variables are independent: \[ p_{Y|X}(y|x) = p_Y(y) \quad \text{and} \quad p_{X|Y}(x|y) = p_X(x) \quad \text{for all } x \text{ in the support of } X \text{ and all } y \text{ in the support of } Y \]
Here’s an example. It’s the one we’ve already looked at several times now, with the same exact probabilities. Take a look at the table and tell me: are the discrete random variables \(X\) and \(Y\) independent? Think about this for a minute.
Well, there are a lot of different ways to see that the answer is no: \(X\) and \(Y\) are not independent. I’ll show you a couple of different ones. The way that I would do it is I would quickly peek at the table, and I would say, “Oh, look at this. If I know that \(X\) has taken on the value 0, then that completely rules out the possibility that \(Y\) will be 2 or 3. And look, if I know that \(X\) has taken on the value 1, then that completely rules out the possibility that \(Y\) will take on the value 1.” So there’s no way that these random variables can be independent, because knowledge of \(X\) rules out some of the values \(Y\) can take on.
That’s a more intuitive way to explain it, but we can also go straight from the calculation based on the definition of independence. Here’s a quick way to do it. Let’s just pick a particular pair: \(X\) is 2 and \(Y\) is 1. Remember, two discrete random variables will only be independent if the joint probability mass function equals the product of the marginal probability mass functions at every single pair of values in the support. So it’s going to be enough if I can find one pair where things don’t work to show that \(X\) and \(Y\) are not independent. The joint probability that \(X\) is 2 while \(Y\) is 1 is 0. But if we look at the margins of the table, we’ll notice that the marginal probability that \(X\) is 2 is 3/8, while the marginal probability that \(Y\) is 1 is 1/4. And 3/8 times 1/4 is definitely not equal to 0: \[ p_{XY}(2,1) = 0, \qquad p_X(2) \times p_Y(1) = (3/8) \times (1/4) \neq 0 \] So we have found a counterexample to the claim that the joint is equal to the product of the marginals. Therefore, we know that \(X\) and \(Y\) are not independent. So there are a number of different ways you can do this. This one uses the definition formally. The way I described before is a little bit more informal, but it’s conveying the same information, namely, that knowledge of \(X\) tells you something about \(Y\), and vice versa, because of the way this table is set up.
So we’ve now covered the three main ideas when looking at the joint distribution of two discrete random variables: the joint probability mass function, the marginal probability mass function, and the conditional probability mass function. We’ve also characterized what it means for two discrete random variables to be independent of each other.