© Sebastian Rushworth.com
I think that anyone who has even a partial understanding of what doctors do understands that the practice of medicine, although based on scientific knowledge, isn't a science. Rather it is an art form. And as with all art forms, there are those who excel, and those who plod along, occasionally producing something nice or useful. Most people are probably aware of the fact that if you go to five different doctors with a problem, there is a significant probability that you will get five different answers. Medicine is so complex, with so many different variables to consider, and doctors themselves are so varied in terms of how they think and what they know, that the end result of any one consultation will often vary wildly.
One of the things that always needs to be estimated in any individual consultation is probability. What is the probability that the breast lump is cancer? What is the probability that the fever is due to a serious bacterial infection? When faced with these questions, I think most doctors are more like an experienced chess player than a robot. They act on a feeling, not on a conscious weighing of probabilities. Doctors with a nervous disposition therefore order more tests and prescribe more antibiotics, while those with a more relaxed disposition order fewer tests and prescribe fewer antibiotics.
But how good is the average doctor?
That is what a study recently published in JAMA Internal Medicine sought to find out. The study was conducted in the United States, and funded by the National Institutes of Health. 492 physicians working in primary care in different parts of the United States filled in a survey, in which they had to estimate the probability of disease in four different common clinical situations, both before and after a commonly used test.
The situations were mammography for breast cancer, x-ray for pneumonia, urine culture for urinary tract infection, and cardiac stress testing for angina. For each scenario, the physicians were provided with a vignette detailing the situation and providing information on the age, gender, and underlying risk factors of the patient. Based on this they were asked to estimate the probability of disease before the test and then after the test, in both a situation where the test came back positive and one where the test came back negative. Here's an example from the survey:
Ms. Smith, a previously healthy 35-year-old woman who smokes tobacco presents with five days of fatigue, productive cough, worsening shortness of breath, fevers to 102 degrees Fahrenheit (38.9 degrees centigrade) and decreased breath sounds in the lower right field. She has a heart rate of 105 but otherwise vital signs are normal. She has no particular preference for testing and wants your advice.
How likely is it that Ms. Smith has pneumonia based on this information? ___%
Ms. Smith's chest X-ray is consistent with pneumonia. How likely is she to have pneumonia? ___%
Ms. Smith's chest X-ray is negative. How likely is she to have pneumonia? ___%
The average age of the participants was 32 years, and they had been in practice for an average of three years. In other words, these were mostly young doctors who had recently graduated medical school. It is reasonable to think that they would do better on this type of test than older doctors, since what they were taught in medical school is still relatively fresh in their memories and is also more updated and correct. Additionally, medical school today emphasises probabilistic thinking and concepts like sensitivity and specificity far more than it did in the past.
So, what were the results?
In the pneumonia scenario, the doctors overestimated the pre-test probability of pneumonia by 78%. In other words they thought the likelihood that the patient had pneumonia was almost double what it actually was. Not good. Unfortunately, that was their best performance. When it came to angina, they overestimated the pre-test probability by 148%. When it came to breast cancer, they overestimated the pre-test probability by 976% (i.e. they thought it was ten times more likely than it actually was). And when it came to the urinary tract infection scenario, they overestimated the pre-test probability by 4,489%! (i.e. they thought it was 45 times more likely than it actually was).
Doh! What are doctors being taught in medical school these days?
What I think is particularly interesting here is that the error was always in the same direction - in each of the four scenarios the doctors thought that the disease was more likely than it is in reality. If this reflects real world outcomes, then that would mean that doctors probably engage in an enormous amount of over-treatment. Obviously, if you think a patient likely has a urinary tract infection, you're going to prescribe an antibiotic. And if you think a patient likely has angina, you're going to prescribe a nitrate. You might even refer the patient for some kind of interventional procedure.
To be fair, this study was conducted in the overly litigious United States. Doctors who know that they are likely to face lawsuits if they miss a diagnosis are probably going to over diagnose and over-treat. But my personal experience tells me this is not just a US-based problem. I've seen plenty of patients here in Sweden with asymptomatic colonization of their urinary tract prescribed unnecessary antibiotics, to take just one example. I think the over-estimation has more to do with cognitive bias than with fear of litigation. Once you anchor on a diagnosis, say pneumonia in someone with a fever and a cough, you will almost certainly overestimate the probability of that diagnosis.
Let's move on. When it comes to how much a test changes estimation of probability, the doctors overestimated the effect of a positive lung x-ray by 92%, of a mammography by 90%, and of a cardiac stress test by 804%! They were relatively on the mark, however, when it came to estimating the impact of a positive urine culture, only overestimating by 10%.
When it comes to how much a negative test changes the estimation of probability, the doctors actually did ok, being close to the mark for both the chest x-ray, urine culture, and cardiac stress test, but wildly underestimating the predictive value of a negative mammogram (in other words, they thought breast cancer was far more likely than it actually was after getting back a negative mammogram, so again, they overestimated the probability of disease).
What can we conclude from this? Doctors have a pretty poor understanding of how the tests they use influence the probability of disease, and they heavily overestimate the likelihood of disease after a positive test. They are however generally better at understanding the impact of a negative test than they are at understanding the impact of a positive test.
Finally, the survey asked the doctors to consider a hypothetical scenario in which 1 in 1,000 people has a certain disease, and estimate the probability of disease after a positive and negative result for a test with a sensitivity of 100% and a specificity of 95%. Sensitivity is the probability that a person with the disease will have a positive test result. Specificity is the probability that a person without the disease will have a negative test result.
Regular readers of this blog will have no problem figuring this out. If you test 1,000 people, you will get one true positive (since the sensitivity is 100% you will catch every single positive case) and 50 false positives (with a specificity of 95% that means five false positives per 100 people tested). The odds of any one person with a positive test actually having the disease will thus be roughly 2% (1/51). So what did the doctors answer?
The average doctor in the study thought that the odds of a person with a positive test actually having the disease was 95%. In other words, they overestimated the probability by 4,750%!
Apart from that, they thought that a person with a negative test still had a 3% probability of disease, even though the sensitivity was listed as 100% (which means that the test never fails to catch anyone with the disease). Oops. I should add that there were no meaningful differences in how correct the answers were between attendings (more senior doctors) and residents (more junior doctors).
What can we conclude?
Doctors suck at estimating the probability of common conditions in scenarios they face on a daily basis, are not able to correctly interpret the tests they use, and don't understand even very basic diagnostic testing concepts like sensitivity and specificity. It's kind of like a pilot not being able to read an altitude indicator. Be afraid. Be very afraid.
Medical schools should be thinking long and hard about the implications of this study. What it tells me is that medical education needs a massive overhaul, on par with the one that happened a hundred years ago after the Flexner report. We don't send pilots up in to the air without making sure they have a complete understanding of the tools they use. Yet that is clearly what we are doing when it comes to medicine. Admittedly the practice of medicine is much more complex than flying a plane, but I don't think that changes the fundamental point.
Reader Comments
It is called "medical malpractice", and it is the third leading cause of death in America today, behind cancer and heart disease: [Link]
You don't use statistics to calculate the speed of a falling object, or voltage and current in an electrical network, because you know the cause-and-effect laws.
Applying statistics means "I don't know the relevant variables, and I cannot observe them. Thus I watch for the visible behavior, and make assumptions".
In the wake of the "Spanish Flu" and also lately, doctors tried to infect healthy persons with body fluids from seriously ill patients. The germ theory cannot explain why every single one of that experiments failed to infect the test object, nor can statistics.
My degree is in math, my favorite subject was Probability and Statistics, because it is so useful to so much we must deal with. And I got A's in college math courses and was a member of the math honor society. Not meaning to brag, just make it clear I know what I'm talking about when it comes to this particular subject. Life is so much about making risk/benefit calculations that having a clear understanding of this process should be universal. As you say, it's not. Witness the disasters of the past year. People and societies fumble through the years doing absolutely horrible things because of this failure.
We always used determined equations - unless the relations are unknown or unobservable, i.e. processes we call stochastic. Which applies to very few subjects in this field.
Thus my conclusion - the heavy use of statistics in medicine is an admission of defeat. Wait until they make Monte-Carlo simulations their gold standard ...
If I have the Lyme rash, which I've had, my doctor knows the probability of doxycycline fixing me up is very high, higher than other antibiotics. Which has been determined statistically based on empirical evidence. You don't need hard scientific proof when doxy fixes most every Lyme infection and other antibiotics don't. That's how you use statistics correctly. There's a lot of that in medicine and there's nothing wrong with it. Without it they'd be in the dark even more than they are. The trouble is there's a lot of deceptive use of statistics in medicine today because it's thoroughly corrupt. Correct application of probability and statistics is essential in a field like medicine when in so many cases there's no way in hell they'll ever figure out exactly what's going on at a biological level - but it's very useful to know what works.
But the application of statistics implies an unknown or unobservable causal relations (like Brownian motion).
And, as you might know, correlations do not establish causations.
Good advice. That's been our perception of the medical care system in the US for a long time. If you need trauma care, go to the doctor and they know what to do. Antibiotics can be a miracle. Anything else, like lifestyle-related conditions, you are more likely to be harmed than helped. Most people foolishly think they know what they're doing. Their treatments help just often enough to keep the racket going.
I did not return for any further "help" from this guy.
Number one, when I go to a doctor for help, it is not so (s)he can "practice" anything. I want answers that make sense, explanations of cause and effect. Treatment is secondary to understanding how the discomfort began.
Second, I'm not seeking help for the primary purpose of being given "medicine." If that is all they know how to do, they are wasting my time and acting more the part of a charlatan than doctor.
Third, when I departed I asked myself the question, What does neurology of the brain have to do with pain in the scalp?
I since put the pieces of the symptoms together and reasoned that if these hyper-tensive drugs worked by softening the vessel walls, then what was happening was the vessels in my scalp were ballooning during said activity. They were expanding so far it was stretching everything in the vicinity, the same as cutting off the circulation in a digit, so that it expanded beyond tolerance. Men are quite familiar with this kind of pain from prolonged erections.
How is it that a doctor of medicine and not of health, cannot even put 2 & 2 together about the effects of their "medicine?"
And the AMA is concerned about "quacks."
Albeit MDs might be well-meaning (mostly), they are trapped in their dogma, and brainwashed in their med school. What they have not been taught simply cannot exist. They cannot see they are nothing but licenced drug peddlers.
Critical thinking and self reflection is not at all related to "IQ" or education. This insight took me the longest to understand.