Comment: Why statistics can mislead on vaccination’s efficacy

By Jordan Ellenberg / Special To The Washington Post

Is the vaccine wearing off? It’s an exhausting thought for those of us who believed the battle against covid-19 would be won once enough needles plunged into enough arms.

But outbreaks of the delta variant have blossomed even in places with high levels of vaccination, including Israel, Britain and my own home of Madison, Wis. Recent reports from Israel that nearly 60 percent of people hospitalized with severe covid-19 were fully vaccinated raised particular alarms about the limits of the protection that vaccines provide.

While there is other evidence that the vaccines’ protection against infection (not severe illness) declines over time — and also some evidence that booster shots work — we shouldn’t be overly concerned about the Israeli hospitalization statistics. The explanation involves a famous old statistical curveball called “Simpson’s paradox,” which isn’t really a paradox at all. It’s just a reminder of why you have to be careful with data.

Simpson’s paradox is a warning that the whole of the data often looks weirdly different than the sum of its parts. In the case of Israel — as a number of epidemiologists and other scholars have pointed out — what explains the surprising hospitalization figures is largely the relative ages of vaccinated and unvaccinated people. The University of Pennsylvania biostatistician Jeffrey Morris wrote an especially thorough and widely shared blog post making this point; although the post was not shared nearly as widely as the worry that vaccines aren’t helping you stay out of the intensive care unit. Anxiety travels halfway around the world, while a spreadsheet is still getting its boots on.

Remember that a lot of Israelis are vaccinated, around 80 percent of the adult population. That’s important. If everyone were vaccinated, then all hospitalized people would be vaccinated; and that obviously wouldn’t mean vaccination was useless. In real-life Israel, as of Aug. 15 — using Morris’ summary of official data — 301 fully vaccinated people had an illness severe enough to require hospitalization. They represented just 53 out of every million fully vaccinated Israelis. At the same time, 214 hospitalized people were not vaccinated. Those people made up a much bigger fraction of the smaller population of unvaccinated people: 164 out of every million. So an unvaccinated Israeli is about three times as likely to end up in the hospital as their vaccinated compatriot.

But that’s not as impressive a difference as you might expect, is it? When we get that shot, we’re hoping for and expecting a lot heftier risk reduction than a factor of three. The really counterintuitive nugget at the heart of Simpson’s paradox lies deeper. I learned last summer, from mathematician Dana Mackenzie, a way to make the power of the “paradox” really clear. It has to do with racial differences in rates of covid-19 infection and deaths.

As of July 2020, Mackenzie noted, confirmed cases of covid-19 in the United States broke down along racial lines like this: 34.6 percent were Hispanic, 35.3 percent were non-Hispanic white and 20.8 percent were Black.

The distribution of deaths from covid-19 looked different, however: 17.7 percent were Hispanic, 49.5 percent were non-Hispanic white and 22.9 percent were Black.

These numbers are startling on their face if you know anything about health disparities in the United States, which in almost all cases involve differentially bad health outcomes for people of color. Why was the proportion of white deaths so high?

The answer is simple. White people with covid-19 are more likely to die of covid-19 because old people with covid-19 are more likely to die of covid-19, and white people tend to be old (relative to non-white people). Break down cases by age groups and things look completely different, Mackenzie noted. Among Americans between 18 and 29, in July 2020, white people made up 30 percent of covid-19 cases but just 19 percent of the deaths. Among people 85 and up, 70 percent of covid-19 cases and 68 percent of deaths were white people. In fact, within every single age band of adults recorded by the Centers for Disease Control and Prevention, a covid-19 case in a white person was less likely to be fatal than it would be for the typical American that age. Yet when you combine the groups together, the disease appears to be falling harder on white people because it was so much likelier to be fatal for the elderly, a group made up of a larger share of white people.

Something very similar is happening in Israel. Vaccinated Israelis, like white Americans, are older as a group than unvaccinated Israelis. And that’s why they’re going to the hospital at a rate higher than you might naively hope. Among Israeli adults under 50, as of Aug. 15, 3.5 million were vaccinated and 1.1 million were not. That’s still a considerable number of vaccine holdouts. Among those 3.5 million vaccinated younger people, just 11 were hospitalized; about three per million. Meanwhile, of the unvaccinated in this age range, 43 were in the hospital, or 39 per million.

Note that hospitalizations of young people for both the vaccinated and unvaccinated are low, because younger people rarely suffer the severest illness from covid-19. Still, vaccination reduced the rate of hospitalization more than 10-fold in the population under 50.

Now look at the population 50 and older. There are 2.1 million vaccinated Israelis over 50, and 290 were in the hospital Aug. 15. That’s 136 per million, a rate that dwarfs anything younger people are experiencing. And unvaccinated older Israelis? There are very few people in that category: just 186,000. But of that group, 171 were hospitalized; a grievously higher rate of 919 per million. In the older population, vaccinated people were less than one-sixth as likely to be hospitalized as the unvaccinated.

Clearly, the vaccines are impressively effective. Still, overall, more vaccinated Israelis were in the hospital than unvaccinated ones. This is a natural consequence of the fact that if you prioritize the most vulnerable people for vaccination — which is what you should do — then vaccinated people will be disproportionately drawn from the vulnerable population. That means more of them than you might otherwise expect will end up sick. (Since Aug. 15, the picture has changed somewhat: Israel now has more unvaccinated hospitalized patients than vaccinated hospitalized patients, possibly related to the rollout of booster doses at the end of July.)

Compare the situation in Israel with that in Oklahoma, whose weekly epidemiology and surveillance report gives an admirably thorough breakdown of the state of the pandemic. In Oklahoma, almost a third of people over 65 remain unvaccinated, a far higher proportion than in Israel. That means there’s a huge reservoir of Oklahomans who enjoy neither the protection of youth nor mRNA technology. And that results in a lot of hospitalization in unvaccinated people. So in Oklahoma, fewer than 1 in 10 people hospitalized with covid-19 are vaccinated. That makes the vaccine look better at a glance than the statistics from Israel. But from the standpoint of public health, you’d still rather be in Tel Aviv than Tulsa.

– – –

Jordan Ellenberg, a math professor at the University of Wisconsin, is the author of “Shape: The Hidden Geometry of Information, Biology, Strategy, Democracy, and Everything Else.”