21 February 2012

When common sense is senseless: the Monty Hall Solution

In the previous article, I presented the Monty Hall Problem, a famous problem in elementary probability, based on "Let's make a deal", a television game show of years gone by. If you haven't read that article, what follows is likely to make little sense. It would therefore probably be better to read that article before continuing to read this one.

As a reminder, here is a video clip presenting the game and the problem:



The player has two options: switch to the other closed door or stay with her/his original choice. What should he/she do to maximise her/his chances of winning? There are three possible answers:
  1. Switching increases the chances of winning
  2. Staying with the original choice increases the chances of winning
  3. It makes no difference at all
It turns out that most people think it is a silly question and that option 3, it makes no difference at all, is the correct answer common-sense dictates. If you have chosen this option, you have a lot of friends.

And, you know what? It is indeed the correct answer for some people. So far the good news for common sense.

And now the bad news: while it is the correct answer for some people, it is not the correct answer for you!

Now, think of it. How can this be? This makes no sense at all, does it?

You are left with two choices. One door hides a goat, the other hides a car. If you stay with your original choice, door 3, you will win either a car or a goat. That's one chance in two of winning the car, yes?

Wrong.

Should you switch, you end up with the same possibilities: the door hides either a car or a goat. Again, one chance in two of winning the car, yes?

Wrong.

As a result, you have one chance in two if you stay, and you have one chance in two if you switch. Clearly, option 3, It makes no difference at all, is the correct choice, yes?

Wrong.

As I said, this option is perfectly valid and the right choice for some people, but again, it is not the right choice for you.

Well then, what is the right choice for you?

It turns out that option 1, switching increases the chances of winning, is the right choice for you.

Is your head starting to spin? Take heart, what I said is perfectly true, it isn't complicated and while it isn't common sense, it does make perfect sense.

First of all, let's see what happens if you switch in the example we started in the previous article. The player decides not to switch and stays with her/his first choice:



After confirming the player's choice, the host opens the corresponding door:


Wrong choice. The player did not follow the advice, and lost. The car was behind door 2. See?


Of course, as you have probably guessed, there is more to the story than this. What about an explanation, for example? Who says I did not manipulate the conditions and the player to make it just so? The answer is, of course, that I did indeed manipulate the example to make the point as clearly as possible. However, this does not mean that the advice was wrong.

Let's look at the previous game once more. Remember, first, there are three closed doors. Two conceal a goat, one conceals the coveted car:



The player, who wants to win the car, must now choose one of the three doors:



Hence, the player has one chance in three of choosing the door with the car. One way to look at this problem is to look, not only at the choice one makes, but also at the choices that one does not make:


By selecting door 3, the player also chooses not to select doors 1 and 2. This means that the player has two chances in three of losing the car, i.e. the probability of losing the car is twice as high as the probability of winning the car.

Another way of describing this situation, is that the probability that the car is behind one of the doors in the rectangle, is two in three, whereas the probability that the car is in the stand-alone square is one in three, or half the probability of the rectangle.

But now, the host magnanimously opens one of the two doors in the rectangle to reveal a goat:


Nothing has changed, except that one door which was previously closed, is now open. The likelihood that the car is behind one of the doors in the rectangle is still  two in three. The likelihood that the car is behind door 3 is also still one in three.

But, we now also know that door 1 does not hide the car. This means that the likelihood that the car is behind door 2 is now twice as high as the likelihood that it is behind door 3, since we can safely discard door 1.

In other words, switching is the adviseable option, since that increases one's chances from one in three to two in three. Winning is not guaranteed when switching, however. It is still possible to lose, but it is more likely that one will win.

Very few people are able to find out the correct solution. As I wrote before, the vast majority of people think that switching or staying makes no difference at all. They are wrong, but that is what common sense tells them.

Common sense helps us stay alive in nature, by helping us decide when to run from a lion who wants to eat us, and when not to waste energy running from a full-bellied lion who merely wants to sleep. In more complex situations however, common sense can easily lead us astray.

This problem is a very simple one, yet almost nobody solves it correctly, because it is so counter-intuitive. Welcome to the land of "conditional probabilities". Here, the probability of an event is not only dependent on an existing situation, but also on what happened before.

This is why I wrote that, for some people, switching or not switching would make no difference at all. Who are these people? The people who walked into the room, only to see one open door and two doors closed and who were therefore unaware of what occurred before. For them, the probabilities of having the car behind door 2 or door 3 are equal.

After seeing the above explanation, many people still refuse to accept the solution. Maybe you are one of them (or not). While problems involving probabilities can often be very complex, the Monty Hall problem is not, and this has an advantage. It is actually possible to enumerate every single possible case for this problem.

Consider this: there are three starting possibilities. The car can be behind door 1, door 2 or door 3. Then, the player can select one of those three doors, leading to nine possibilities (three times three). Furthermore, the player can either stay with her/his choice or switch, two possibilities that lead to eighteen possible combinations in total (three times three times two):


If you only count nine lines, you are -of course- correct. However, don't forget that there are two columns: nine switches and nine stays. That makes eighteen combinations in total.

Although it is probably more boring than watching paint dry, it may be interesting to study each of the 18 cases while they are being played out. To this end, please view the video I uploaded to YouTube:



Since the player cannot know whether he/she should switch or stay, the best strategy is to switch, as the table shows. Switching means that the probability of winning is twice the probability of winning when not switching.

It is really hard for humans to wrap their heads around the Monty Hall Problem. This is, however, not necessarily the case for other animals. Pigeons, for example, do a lot better [2].

Conclusion
I wrote this article as a warning.

The Monty Hall Problem shows us that common sense is not to be trusted. Yes, it can lead us to the correct solution when we face a problem. Unfortunately, it can also lead us to the wrong solution. In other words, it makes no sense to trust common sense or "instinct", as the Monty Hall Problem very clearly demonstrates.

Another warning can be seen in the way I treated this problem: a good skeptic always investigates a problem. In spite of popular belief, skepticism is not about denying what others take at face value. Skepticism is about investigating whatever comes our way. Sometimes, this is very easy, such as for the Monty Hall Problem.

Sometimes, it gets a lot more complicated, but whatever the case may be, the skeptic does his utmost to look at a situation from all angles, to investigate all the possibilities and only then does he/she to come to a tentative conclusion that is itself always open for correction, modification or refutation.

The attitude of the skeptic is the attitude of the doubter, the scientist, not that of the quack or the religionist. As a result, the skeptic must be prepared to be ridiculed by the quack and the religionist, for they will use the skeptic's doubts against her/him to fool the public into believing that the skeptic is an idiot, and many people will fall for this.

The skeptic has but one weapon against this: evidence. Quacks often like to compare themselves to Galileo or other famous scientists who were first ridiculed or even put to death for their seemingly preposterous theories. However, they forget one important element.

For Galileo to become world-famous and part of the bedrock of science, he needed one element the quack and the religionist do not have: the evidence to back up the claim, evidence that shows he or she is right.

Quacks and religionists are not interested in such sordid details as "evidence", and that is why they are almost always wrong, often quite terribly so.

The Monty Hall Problem has a very interesting and controversial history. However interesting, the history of the problem is not really relevant to the problem per se and since Wikipedia [1] has an excellent page on this, I will not talk about it here.

Jeffrey S. Rosenthal is one of Canada's best-known experts on probability and statistics. At the time of the fraud scandal at the Ontario Lottery and Gaming corporation, he was often in the news, because he was the expert called in to investigate the then-alleged fraud.

Not only has he written a hilarious and very interesting book, called Struck by Lightning: The Curious World of Probabilities, in which he talks about the Monty Hall Problem, he also has written an article [3] on the problem, which can be downloaded from his personal website.


References
[1] Several authors, Monty Hall Problem, Wikipedia, retrieved 20 February 2012
[2] Walter T. Herbranson, Julia Schroeder, Are Birds Smarter Than Mathematicians? Pigeons (Columba livia) Perform Optimally on a Version of the Monty Hall Dilemma, Journal of Comparative Psychology, 2010, Vol. 124, No. 1, 1-13
[3] Rosenthal, Jeffrey S. (2005a). "Monty Hall, Monty Fall, Monty Crawl".Math Horizons: September issue, 5–7. Online reprint, 2008

----------
Fine print:
 I have written this article on my personal "authority" as a computer programmer and skeptic. Any errors, inaccuracies, omissions or other flaws are therefore mine, and mine alone. While I have done my best to be as accurate as possible, this article is nevertheless open to correction, modification or refutation. Please do not hesitate to comment or send me an e-mail if you think something is wrong. I will do my utmost to reply to every comment or message within a reasonable amount of time. Please consult a properly qualified professional before using any information in this and any other of my articles.

-----
This page was last updated on 21 February 2012

5 comments:

  1. I've come across this problem before. You did a fine job presenting it. The Monty Hall challenge would make a good topic for a class in journalism school.

    ReplyDelete
  2. Thanks, Rockinon. I agree, that it would make a good topic for journalism school. It shows so clearly that even something exceedingly simple must be thoroughly investigated in order to avoid painful mistakes.

    ReplyDelete
  3. A very nice clear and thorough job of presenting the material and laying it all out so exhaustively that no one can fail to see the point. The best and simplest explanation is the grouping of the open door with one closed door, pointing out that the chances of one of them guarding the gold is 2/3, therefore with one open, the other must have a 2/3 chance of being right. You did an excellent job of making this clear with your images.

    However, perhaps you should make the statement up front that the mistaken choice is valid for some people, even if as you say at the end, if there is no information on what has happened to open one of the doors, then the chances are fifty-fifty either of them will be the right one. Since the requirement that Monty open one door, knowing that that door is a failure, is part and parcel of the question posed, then the possibility of someone not knowing why the first door was opened is never part of the problem, and thus the 50/50 estimation is never valid in this context.

    It leaves you looking as if you were endorsing "I'm OK You're OK" social approval for wrong answers, which I am sure you don't!

    ReplyDelete
  4. Correction: Sorry, I left out "not" as in "should not make" in the first sentence of the second para. Please correct.

    ReplyDelete
  5. Thank you for your nice comment, AnthonyL. Sorry for not replying earlier. It seems that your comments have escaped me somehow.

    You made a good suggestion, but if I want to be complete I cannot take it.

    The point is this: when someone enters the room *after* one door has been opened, there are only two doors left. The car is behind one of these two doors. *If* this individual is oblivious as to what has happened before, the probability of finding a car behind the left door is 1/2. The same is true for the right door. This person has no reason to switch because he/she sees only two closed doors and one of them is hiding a car. The switching part makes sense only to the person who was there all the time.

    Of course, that only applies to a person who does not know what is going on. Knowledge is part of the equation. As it should be.

    ReplyDelete