This post is part of the ppmmc series. Please make sure to have read the previous posts in the ppmmc series so that the context is understood.

A Very Introductory Introduction to Markov Chains — Part 3

Recall in the Thursday post, we considered two games offered by Mr. Conner Tist from which we had to choose one.

It turn out that both games are losing games for us, but we had to choose the one that was “less” losing. Another way of thinking about this is by asking the question, “How much will I win or lose per game on average.”

Imagine if we played six billion games of game 1. Let’s say that one billion of those games resulted in a win, two billion resulted in a draw, and three billion resulted in a loss. Then given the point payouts for win, draw, lose, we would have netted a loss of two billion credits. Since we played a total of six billion games, we lost one-third of a credit per game, on average. This is one way to think about expected return.

The technical nuance when we talked about an expected return is that we mean over an infinite number of plays. So how do we handle “infinite” plays? Analytically, that means getting a handle on the proportion of times the events of interest occur. Thus, in game 1, a win occurs one-sixth of the time. A draw occurs one-third of the time. A loss occurs one-half of the time. So, that means one-sixth of the time we gain one point. One-third of the time we neither gain nor lose a point. And one-half of the time we lose a point. Therefore, we may calculate our expected return, \(r\) as $$r = -1\cdot\frac{1}{2} + 0\cdot\frac{1}{3} + 1\cdot\frac{1}{6} = \frac{-1}{3}$$

Again, let’s say we couldn’t figure this out on paper. What then? Simulate! We know the game rules, so let’s just program the game and play it, say, one million times. Here is what our game code could look like:

As we can see, the first value returned is the total win / loss. The second value returned is how much was won / lost on a per game basis. Theoretically, we calculated a per game loss of one-third of a point. When we ran a few trials of one million simulations per trial, we observed a per game loss that was close to one-third. So it seems, that our theoretical calculations are inline with our simulated calculations.

So, now let’s see what our exact calculations says about game 2. Remember game two looked like:

And in the previous post we calculated that the probability of winning was 25%. Since there is no possibility of a draw in game 2, we have that the probability of losing is 75%. Therefore, our expected return is $$r = -1\cdot\frac{3}{4} + 1\cdot\frac{1}{4} = \frac{-1}{2}$$

So, it looks like game 2 is worse than game 1. In game 2 we will lose, on average, one-half a point per play, while in game 1 we will lose, on average, one-third a point per play.

What do our simulations say? Here is a program for game 2.

These numbers also seem to be inline with what our theoretical calculations gave.

We’ll stop here, but this is what’s in store for the rest of this week:

• We are going to introduce more complex games and start getting more involved with Markov Chains.
• We will also slowly introduce some statistical notions, like standard deviation and confidence intervals. As we saw in these simulations, there is some variability in the results; we want to understand what that variability is.

Here are some exercises and further ideas:

• Instead of getting an infinite number of rerolls, what would happen if we were only allowed to reroll once? Is this game better than game 1? is it better than game 2? Can you solve this theoretically?
• In game 2, instead of receiving +1 for a six, what if we received +1 per number of rerolls? Thus, on the first roll, if we roll a six, we get +1, but if we have to reroll and then roll a six, we get +2. Which game is better game 1 or this new game 2?

Don’t hesitate to get in touch if you have questions or would like more information about setting up a workshop or training in your school.