Eddie Thomas, a better philosopher than he is a statistician, poses the following problem. He is interviewing five candidates for two jobs. Each candidate’s chance of receiving a job offer, a priori, is 40%. After interviewing four candidates Eddie wants to offer a job to the best of the four to protect against his taking another job elsewhere. He is puzzled because it now looks like the final candidate has a 25% chance of a job offer, being one in four remaining, while his chances should be unaffected, remaining at 40%.
This is a type of restricted choice problem. The classic illustration of true restricted choice is the old game show Let’s Make a Deal. Monty Hall shows the player three doors, behind one of which is the grand prize, a Hawaiian vacation or a brand new Cadillac Eldorado. The player chooses one door. Monty then reveals another, behind which the prize is not hidden, and asks the player if he wants to switch. The player should always switch. His chance of choosing the grand prize in the first place was 1 in 3. If he switches, it is 2 in 3, because Monty’s choice of which door to open has been restricted by the choice the player already made. Many people don’t believe this even after it has been explained to them, but it’s true, and can be verified easily by experiment if you doubt it.
Here, on the other hand, because of restricted choice, the probabilities only appear to change. To receive a job offer you must be one of the best two of five candidates. Consider it from the point of view of the first four candidates. Each one has a 25% chance of receiving an offer after four interviews, and a 15% chance (.75 X .20) of receiving an offer after five interviews, for a grand total of 40%, as you would expect. So one way to look at it is that the remaining candidate must also have a probability of receiving an offer of 40% for the probabilities to add up to 200% (two job offers).
Since this will satisfy no one, least of all Eddie, I’ll try another approach. To receive an offer he has to be better than the three remaining candidates. However, his three remaining competitors are not randomly chosen; they have already failed to finish first among the first four. Choice has been restricted. Each one of the three, not having received the first offer, has a far less than 40% chance remaining of securing an offer. In fact, they now have half that chance; for they can only finish second, at best, among the five, while the fifth candidate can still finish either second or first. Therefore their remaining chance is half of their original 40%, since one of the two offers has been closed to them, or 20%, rather than 25%. The fifth candidate still has a 40% chance.
God of the Machine: all probability, all the time!
"Since this will satisfy no one, least of all Eddie…"
Are you doubting your powers of demonstration or my powers of comprehension? I think I know the answer!
I do, however, have an alternative solution now in my comments to my original post. Have at it.
I’m not doubting your powers of comprehension at all. It’s just that I’ve tried to explain apparent probability paradoxes before, and certain demonstrations, though true, are simply not intuitively persuasive. I couldn’t get my girlfriend, who is no idiot, to believe the Monty Hall solution until she did 20 trials with a penny and three Dixie cups.
However, your solution in your comments is also wrong. A lottery would produce exactly the same result and would in no way prejudice the chances of the last candidate to draw.
I don’t see how a lottery could not be prejudicial. If, after the first four candidates come, we draw one of those four, then the fifth candidate has been excluded from an opportunity to be selected. This has to lower his odds.
No, you’re still wrong. In fact the fifth candidate is certain either to be selected or not, depending on whether the first four have already drawn the two winning tickets. The chance that they have is, of course, 0.6. Maybe the easiest way to see this is to write out the possibilities, which is manageable in this case. W will represent a winning ticket, L a loser. So we have, with equal probability, 10 possible orders:
1 2 3 4 5
W W L L L
W L W L L
W L L W L
W L L L W
L W W L L
L W L W L
L W L L W
L L W W L
L L W L W
L L L W W
Each position wins exactly 4 times out of 10. Order is irrelevant.
I get the Monty Hall scenario but I still can’t see how the job chances of the 5th interviewee work out to 40%. The situations do seem fundamentally different, because the Monty Hall contestant still has a chance to change their position, while the job candidate can’t change their aptitude for the job.
This is how I see it: on Monty Hall, you have a 1/3 chance of being right with the first pick, which means there is a 2/3 chance that pick is wrong and that the winner is in what we’ll call the field. So when Monty Hall gives you a chance to change your mind and pick the field–and eliminates all the losers from the field–it’s a no brainer.
The 5th job candidate, who ranks either 1, 2, 3, 4, or 5 in the field and needs to place 1 or 2 to get the job, starts out with a 2 in five chance. But once the first job is given out (to a guy who was either 1 or 2), now to get the job, the 5th candidate has to be the best of the remaining 4–and I don’t see how his chances of that can be better than 1 in 4.
Another way of looking at it (though this is a bit murky in my mind) is this: by being revealed as either #1 or #2, the first job winner has reduced by 50% the original chances of interviewee #5 he would win in each of the 2 ways he could have won: by placing in position #1 and by placing in #2. (The first job winner now has a 50% claim–a 50-50 chance–on both those positions). So his chances are now 2 half-shares, and since the field has been reduced to 4, his chances are 2/2 (or 1) in 4.
Where did I go wrong?
This is just a wonderful post. Of course Aaron, you are correct in your reasoning. The interesting thing about this whole exercise is how easily bright people can be mistaken in their analysis.
When I first came upon the Let’s Make a Deal reasoning, I had to verify it by experiment. The interesting thing about this field of study is the relationship between analysis and experiment. Experiment is invaluable to help test conclusions you may think are airtight.
Your last paragraph is almost right, except that the first job winner has not reduced the chances of the fifth candidate, who can still place either first or second. (If you interview last, somebody’s going to be leading.) It is the first three candidates whose chances have been reduced: they can place second at best, having already been judged with reference to the first winner and found wanting.
Since the first three losers can occupy only one of the two winning slots, their chances are cut in half, to 20% from their original 40%. The remaining 40% chance belongs to the fifth candidate, for whom things are just as they were.
"The classic illustration of true restricted choice is the old game show Let’s Make a Deal. Monty Hall shows the player three doors, behind one of which is the grand prize, a Hawaiian vacation or a brand new Cadillac Eldorado. The player chooses one door. Monty then reveals another, behind which the prize is not hidden, and asks the player if he wants to switch."
Didn’t the other doors hide goats? When Monty would open one of the doors not chosen by the player, he would reveal a goat. It’s a much better story with goats.
I admit that it would have been difficult for your girlfriend to fit goats under Dixie cups.
Aaron:
Ah, the belated light bulb just went off! When the first job is filled, nothing had changed for candidate #5, since all that proves is that he can’t be both #1 and #2–which he knew anyway. But he can still turn out to be either #1 or #2. But the other four now know they can’t be #1, so they have only one of their original 2 chances in 5 left.
Floyd: I forgot about the goats. But the girlfriend tells me it was sometimes a cow, which seems less funny than a goat somehow.
John: Excellent. Now we just have to bumrush Eddie.
I still need to think about your solution to the original problem, but I am holding out on the lottery alternative. If we select a candidate randomly after the first four, then add the fifth candidate to the pool and select randomly again, then the probabilities work out as follows:
Candidates 1-4:
1 – (3/4)*(3/4) = 7/16
Candidate 5:
1/4
We must be understanding the lottery alternative differently.
I understand a lottery in the customary sense: five people choose tickets, two of which are winners, from a hat. In this case order does not matter.
In your construct we throw four names in a hat, choose one, then throw in a fifth name and choose again. In this case you’re correct. But that’s one weird lottery, and it’s certainly not analogous to the job interview case.
The job search is not a lottery unless you assume those doing the hiring are clueless–which, in real life, is not out of the question. But the right 5-in-a-hat analogy is: put 5 numbers in a hat (randomly chosen from all the numbers there are). The 2 highest numbers are winners. You take out 4 and set aside the highest of those 4 (you want to get him quick before he bolts to another analogy). The question is: at this point, what chances does the 5th number have of winning. The answer is: the same as he always had. If you play this with a real hat & numbers, #5 will be the highest of the remaining 4 about 40% of the time.
It is a weird lottery, but it mimics what would logically follow if every candidate was equally likely to get a job, because equally likely would have to mean equally qualified. In such a case, only an arbitrary selection process would be fair, but then it would be unfair to pick a candidate just from the first four.
That we can choose a best candidate from the first four means that in fact that they are not actually equally likely to get the job before the process starts. The candidates may seem equal given our state of ignorance, but that doesn’t translate into actual equality. Thus I contend that we cannot work out the probabilities from the information given, because we don’t know enough about the selection process (how candidates are ranked) and what their qualifications are.
Eddie:
It is true that candidates have different aptitudes for the jobs, but that is not relevant when we haven’t seen them yet. Surely you must agree that if someone tells you that five candidates have applied for two jobs, at that point a candidate has a 40% chance of getting a position.
Your "lottery" does not model your interview process. It may help if you actually write down all the possible rankings of candidates. We’ll label the candidates A thru E (E being the candidate who is interviewed fifth), and write them in their rank ordering:
ABCDE
ABCED
ABDCE
ABDEC
etc.
There are 120 possible orderings, but the order of the last three doesn’t matter, so we can simplify matters by writing the combinations like this:
ABxxx (6)
where the 6 stands for the 6 possible orderings that start with AB.
After four interviews, a job is offered to A, B, C, D. What are the rankings when one candidate, say "A", is made an offer? He is ranked ahead of B, C, or D, so he is ranked first:
ABxxx (6)
ACxxx (6)
ADxxx (6)
AExxx (6)
or second behind E:
EAxxx (6)
There are 30 ways for each of A thru D to get an offer. This adds up to 120. A, B, C and D have a 25% chance of getting a job after four candidates have been interviewed.
Now E is interviewed and another offer is made. E can get an offer if he is ranked first:
EAxxx (6), EBxxx (6), ECxxx (6), EDxxx (6)
or second:
AExxx (6), BExxx (6), CExxx (6), DExxx (6)
These are 48 possibilities, which is 40% of 120. E has a 40% chance of getting an offer.
What of the three remaining candidates? They are known not to be ranked first (someone was already chosen in preference to them). Therefore they can only be ranked second, and they cannot be ranked second when E is first. Here are the possible choices for A:
BAxxx (6)
CAxxx (6)
DAxxx (6)
18 choices, which is 15% of 120. A thru D have 15 + 25 = 40% chance of getting a job.
"It is true that candidates have different aptitudes for the jobs, but that is not relevant when we haven’t seen them yet. Surely you must agree that if someone tells you that five candidates have applied for two jobs, at that point a candidate has a 40% chance of getting a position."
I strongly disagree. Imagine a biased die where the number 6 comes up twice as much as any other number. If I don’t know the die is biased, I would predict beforehand a fairly equal distribution over a large number of rolls. My ignorance, however, does not make it so, and I will find that out after the rolls have been made. For the 40% figure to stand up, we have to presume each candidate is equally likely to be selected, but my later ranking of the candidates proves this not to be the case.
"Your ‘lottery’ does not model your interview process."
It models the process insofar as there is an offer made to one candidate out of the first four, before the fifth candidate has been seen. It doesn’t model the actual process because it assumes each candidate is equally likely to be chosen, i.e., the selection is arbitrary.
"I strongly disagree. Imagine a biased die where the number 6 comes up twice as much as any other number. If I don’t know the die is biased, I would predict beforehand a fairly equal distribution over a large number of rolls. My ignorance, however, does not make it so, and I will find that out after the rolls have been made. For the 40% figure to stand up, we have to presume each candidate is equally likely to be selected, but my later ranking of the candidates proves this not to be the case."
Someone tells you that your department is interviewing five candidates and will hire one. If someone asks you "what do you think are the odds that the fourth candidate we interview will be the one we hire?", the answer is 20%. Now #4 may turn out to be extremely qualified, or extremely unqualified. But if you repeated the process many times, #4’s chance of getting a job is always 20%, even if individual #4’s are geniuses or drooling idiots. To say "I can’t answer that question because the fourth candidate could be very good or very bad" is wrong; the odds are 20%.
If it offends you to model the problem as a die roll, because dies are random and candidates have abilities that can be compared, think of the problem like this: What are the odds that a particular interview slot (such as "fourth") will contain the most qualified candidate?
I agree that the most qualified candidate is equally as likely to be in the 4th spot as anywhere else, but that is a function of the randomness of that ordering process, not a function of the candidates having an equal chance of being hired.
Lord love a duck! You analyze Emily Dickenson and do statistics. I’m impressed, Aaron.
I’ve seen the Monty Hall thing before but still don’t get it. Unless the first door is opened, I don’t see how the choices are affected. If you keep the first choice, your odds are 1/3 of getting the good prize and 2/3 of getting a lesser one. If you bypass in favor of one of the others, you now have a 50-50 chance at getting the best remaining prize–but the great prize might have been behind the bypassed door.
The reason you switch is that Monty’s choice is constrained. He can never open the door with the grand prize behind it. If you had a 1/3 chance of choosing right the first time, it doesn’t change because Monty opens another door. Consequently your chance is 2/3 of winning the grand prize if you switch. The same reasoning applies to lesser prizes. No matter what Monty shows you, you always switch.