urn:lsid:ibm.com:blogs:entries-a27fff3d-0d1b-4145-b91e-6726fb6cadf1Insurance - Tags - expert Incorporate new approaches to address uncertainty and complexity.12013-06-26T15:55:07-04:00IBM Connections - Blogsurn:lsid:ibm.com:blogs:entry-c06a5d82-c67a-484b-b6bd-7c55dbdfa326How to test your doctorChristian Bieckchristian.bieck@de.ibm.com270001WK4FactiveComment Entriesapplication/atom+xml;type=entryLikes2012-11-12T09:15:55-05:002012-11-12T09:36:05-05:00<p align="left">Imagine engaging with some friends in a friendly game of roulette – “friendly” because you have a special wheel without a zero (i.e. no house edge). You are about to consider where to place your bet and need to decide on red or black. What is the probability that the next roll will come up red?</p> <p align="left">That one was easy, right? Since there is no zero, the probability is exactly 50%, so you pick whichever color you like best at the moment.</p> <p align="left">Now suppose you watch for a while first, and you see red and black both coming up in a seemingly random order. Suddenly, the wheel goes on a streak, with the ball falling on red 10 times in row. Where do you place your bet now, and why? I.e. what is the probability of another red?</p> <p align="left"><a href="https://www-304.ibm.com/connections/blogs/gbs_insurance/entry/can_we_ignore_residual_risk_an_insurance_perspective?lang=en_us">As I wrote before, humans in general are bad at probabilities</a>, because that is simply not the way we are wired. Like all calculations, using probabilities is hard (for most people), is slow, requires thinking and thus expends energy. (The latter two for all people.) Contrast that to using “intuitions” where we can make judgment calls in an instant, usually effortlessly and not even consciously.</p> <p align="left">Daniel Kahneman, a psychologist who won the 2002 Nobel prize winner for economics, calls these two models of thinking System 1 and System 2 in his book “Thinking, Fast and Slow”. (Great reading <img alt="Smiley" class="wlEmoticon wlEmoticon-smile" src="https://www-304.ibm.com/connections/blogs/gbs_insurance/resource/Windows-Live-Writer_Doctors-and-probabilities_E0C2_wlEmoticon-smile_2.png" style="border-bottom-style: none; border-left-style: none; border-top-style: none; border-right-style: none"> ). System 1 is the fast, effortless, intuitive mode, while System 2 is deliberate and requires energy and concentration. We perceive ourselves as rational beings where System 2 is in the driver seat, but most of the time, that is not the case – and for good reason. For one, System 1 is pretty good at doing most of our daily tasks, even including moderately complex ones like driving a car (ever had the experience of getting in the car after work, getting out when you reached home and not remembering most of the trip in between?). If we needed to engage System 2 all the time, we would be much too slow and expend way too much energy on most tasks.</p> <p align="left">On the other hand, only System 2 can think about thinking – not surprising that it thinks it is in the drivers seat all the time. In reality, whenever we can get away with just using System 1, we do. (Quote: “We are lazy.”)</p> <p align="left">Back to roulette – do you remember your answer to the second question?</p> <p align="left">After 10 times red your hand will automatically put your chips on black – intuitive System 1 taking over. But even if you do turn on System 2 – which I assume both of today’s readers did – there are ways in which intuition can kick in. After all, the chance of 10 reds in a row is 1 / (2 ^ 10), right? I.e. about 0.1%. Another red would further half that probability – 11 reds in a row are just at 0.05% chance. Time to place the bet on black, then?</p> <p align="left">Ok, I didn’t fool you. The prior turns of the wheel have no effect whatsoever on the next outcome, so chances are still at 50%. Objectively, there is no reason to bet on black. (Can’t hurt, though, either.)</p> <p align="left">Let’s move to the world outside games of chance. The next riddle is the one to test your doctor with.</p> <p align="left">Let’s assume a disease that affects 1% of the population. (At random, to make it simpler.) Let’s also assume there is a cheap way to test for the disease so that everybody who sees a doctor gets tested. The test is fairly reliable, detecting the disease in 90% of the infections. Unfortunately, there are false alarms – 10% of those not infected will show up as positive (i.e. infected) on the test. </p> <p align="left">Now imagine a friend of yours has the test, and shows up positive – what do you tell him his probability of actually having the disease are? Take a moment.</p> <p align="left">The intuitive answer is 90%, right? That is what most people, even doctors and people trained in statistics will say. (Seems there were studies on that, but I can’t find the reference right now. Anybody?) Unfortunately, 90% is not only wrong, it is quite far off from the correct answer. </p> <p align="left">It quite safe to say that the correct answer is counterintuitive. Why? When we have a hard time answering a question even using System 2, our System 1 tends to use a heuristic that Kahneman calls ‘substitution’ – we just answer a simpler question. The US had millions of people employing that heuristic very recently: in elections, the question that actually should be the relevant one is “who of the candidates will be the better leader the country/the state/etc.” Since that is a tough question, we substitute for easier ones like “does he/she belong to ‘my’ party” or “do I like him/her”. We do the same when looking for doctors, insurance agents or other professionals where it would be difficult to judge expertise.</p> <p align="left">What is the substitution here? There are three probabilities in the riddle (1%, 90% and 10%), all of which are relevant. Those who answered 90% substituted the full riddle for a different one, i.e. one which only had the 90% probability that an infected person has a positive test, and nobody else does.</p> <p align="left">Let’s assume you have 1,000 people getting tested. Of those tested, 10 (=1%) will be infected, 990 will not. </p> <p align="left">Of the 10 infected, 9 will test positive.</p> <p align="left">Now we need to use the last information – those of false positives. Of the 990 people tested who are not infected, 10% will still test positive, i.e. 99 people. Which means you have 9+99 = 108 positive results, but only 9 of those are actually infected. The probability that a positive results shows a real infection is 9 / 108, i.e. 8.33%. Quite a difference to 90%, isn’t it?</p> <p align="left">Humans, even experts, are bad at probabilities.</p> <p align="left">Ok, some might say, math is hard, and even experts might get something like this wrong from time to time. Surely, in general you can trust expert intuitions? We read about them being right all the time – the firefighter calling alarm just before the floor collapses and other cases.</p> <p align="left">Well, intuitions (i.e. System 1) can be trained. If the firefighter was on his first mission, chances are that the call was pure luck, but in an experienced fireman who had seen all the signs before will make the right calls with a high percentage reliability. Unfortunately, confirmation bias also kicks in here – we only tend to read about cases were the call was right, not the other ones. (Confirmation bias might get a separate post one day.)</p> <p align="left">Which is why a tool like <a href="http://en.wikipedia.org/wiki/Watson_%28computer%29">Watson</a> is such an interesting phenomenon. (Search for ‘Watson jeopardy’ on Youtube if you want to see it in action.) At the very least, it can be a valuable augmentation for human expertise, especially in areas where experience is hard to come by, e.g. because a disease is very rare and the cases widespread. For Watson, nothing is counterintuitive, because it only has a System 2 – plus, it has no ego…</p> <p align="left">Can Watson replace human expertise? Let’s talk about that another time.</p><p>(NB: The logic used in the medical example is called 'Bayesian'. For the nerds, check out <a href="http://xkcd.com/1132/">this joke</a>.)<br></p>
Imagine engaging with some friends in a friendly game of roulette – “friendly” because you have a special wheel without a zero (i.e. no house edge). You are about to consider where to place your bet and need to decide on red or black. What is the probability...001627