Methodology and Analysis for Bloomberg’s Covid Vaccine Tracker

Below are the key takeaways from the live blog event “Methodology and Analysis for the Covid-19 Vaccine Tracker”, followed by a complete transcript of blog entries in the order they were originally posted.

01/21 13:07 ET

Bloomberg has dozens of reporters around the world tracking the rollout of the biggest vaccination campaign in history. Here we’ll dig into how we’re doing it and describe some of the idiosyncrasies in the statistics. We’ll also provide news and analysis from the data we’re collecting around the world.

Tom RandallSenior Reporter

01/25 11:00 ET

California has consistently ranked at the bottom of U.S. states the percentage of vaccines used.

As of Sunday evening’s update, the state had received 4.91 million doses of Covid-19 vaccine delivered, according to the CDC, and had used only 2.32 million — less than 50%.

The state says missing data may be to blame — specifically that many of the people giving vaccines on the front lines aren’t correctly logging them in the state’s records.

California Department of Public Health Director Tomás Aragón says the state knows that doses were being used, but hadn’t figured out why they weren’t showing up in its counts. So it launched a review to find out what was going on.

“On the Bloomberg Index we were ranked low and so we’re thinking: Oh, what’s happening?” Aragón said in an interview Friday. “What we were hearing on the ground, people were telling us, we need more vaccine.”

A state team found that many vaccines administered were not being reported. Some health clinics were using a records system that requires them to click a “submit” button at the end of the day -- and they were forgetting to do so. Other providers using paper forms weren’t typing them up because it took too long. (Each form includes 40 data fields.)

“A lot of these were just piling up. So now we’re working really hard to get those entered,” Aragón said.

California’s numbers are starting to improve. Instead of dead last, the state has moved up to sixth from the bottom compared to other states, as of Sunday’s count. (Kansas ranks last among states, with 43% of delivered vaccines used. The U.S. average is 54%. Some states have used more than 80%.)

Aragón said California might eliminate paper records except in situations where it’s absolutely necessary. It’s also in the process of rolling out a new system from Salesforce and Accenture that will feed directly into the state immunization registry.

Angelica LaVitoHealth-Care Reporter

01/26 18:08 ET

Curious about how we collect our data? Wondering why we made certain decisions for the tracker? We have answers.

We’ve moved our FAQ to its own page. You can now find it here.

Drew ArmstrongHealth Editor

01/28 12:36 ET

Almost two weeks after national vaccinations began, Brazil is still not reporting consolidated figures. Officials say there are technical issues preventing the Health Ministry from publishing data it receives from local governments. So for now, it’s up to Brazil’s 26 states (and the Federal District) to publish their own numbers.

Several have set up websites that are updated daily or even in real time, like Sao Paulo and Rio Grande do Norte. Some are still sending tallies by email -- Rio de Janeiro, for example. Others aren’t publishing information at all.

Some Brazilians have taken data collection into their own hands, including a popular website that reports vaccination totals. Bloomberg’s tally differs, because we don’t use third-party data sources in our collection. Bloomberg’s count includes 15 states which account for roughly 77% of Brazil’s population.

So far, regions aren’t providing breakdowns of which vaccines are being given. And while some of the local data dashboards are set up to report first and second doses, none have done so -- it’s still to early for the booster shots. It may take a while: Amid a widespread shortage of vaccines, local governments have been debating whether to widen the interval between doses to reach more people.

As the data from official sources gets better, we’ll keep adding to it.

Julia LeiteBrazil Country Chief

01/29 15:27 ET

Something a bit strange may shake out in the next few days with U.S. state vaccine dashboards.

This week, Pfizer announced that it would be counting six doses per vial as being shipped to states, not five doses. That’s because there’s “extra” 3ml of vaccine in each bottle that health-care workers have been able to extract.

That change may cause some quirks in a few states that are using up almost all of the vaccine that’s been sent to them. On Friday, West Virginia reported on its dashboard that it’s used more first doses of vaccine than it’s received.

That’s not an error, it’s just an effect of having received lots of vials that counted as five-dose-vials when they were delivered, but health-care workers on the ground are eeking out a sixth dose from them.

This numerical weirdness should resolve over time, but expect it to show up in a few more places until then. We don’t expect the changes to throw off the big totals in our dashboards much, but it’s worth being aware of at the state level and if you’re poking around on your own state’s website.

Drew ArmstrongHealth Editor

02/03 11:23 ET

A variety of vaccination figures are circling around The Netherlands. If you’re confused, join the club.

Up until now, the total number of administered doses was reported on a government dashboard that collected data from the institutions giving the shots. But the health ministry has now adjusted its method. The result? Two different numbers.

It turns out the previous number - which, again, was based on the injections reported by local health authorities, hospitals and long-term care facilities — wasn’t considered to be reliable enough. So the ministry came up with an estimate based on the number of doses distributed, regardless of whether they’d actually been injected into an arm yet. It notes on the dashboard that doses are administered within four days of delivery.

There was “under-reporting,” from the hospitals and long-term care facilities, the ministry said. “In order to give the best possible picture of the number of injections taken, the dashboard will temporarily show a mathematical assumption of how many injections have been taken.”

This resulted in two figures — the “estimated amount” of vaccines given, and the “reported number” from the health institutions. Adding to the confusion, the so-called reported number was significantly lower than the single figure that was reported on the government dashboard until recently.

The new figures led to an outcry among hospitals saying they certainly did not under-report injected doses.

If you’re confused, you’re not alone, and there’s more. On Monday, Dutch Health Minister Hugo de Jonge told reporters that he wants an review of the calculation method. (Not much later the Dutch government announced it had accidentally double-counted 17,550 of its “estimated” doses and was removing them from its calculated total.)

To make things worse, the ministry has warned that another change may be on the way: It plans to use automated data from long-term care facilities that will feed into the central registration system of national health authority RIVM. “The moment this adjustment of the working method is implemented, it may result in a sudden change in the figures, just like the change on January 31, 2021.”

Our policy is to count reported doses whenever possible. Shots go into arms, they get reported, we count them. It’s better for us to undercount something and catch up when the data get better than it is to be wrong and then have to correct ourselves.

As for what it does to our numbers, you’ll see that our Dutch totals have dropped from their previous totals, reflecting the new, more conservative methodology we chose to use. And be on the lookout for more changes in the future.

Ellen ProperAmsterdam Business Reporter

02/12 18:30 ET

Today we revised our method for estimating the average daily rate of vaccinations. The changes are intended to increase the accuracy of our daily estimates for countries that report their data infrequently—most notably China and Russia.

Prior to today’s change, whenever we received a new update, those data would only affect the country’s seven-day rolling average beginning with the day we received the new data. The new method interpolates the data retroactively when calculating a running average—and then carries that average forward until we receive the next update.

For example: Imagine a country only reports once a week, and within that period it administered 70,000 doses of vaccine. First, we retroactively apply 10,000 doses per day to the daily rate. Second, we would assume that the country will continue to vaccinate 10,000 people a day until we receive its next data update.

The effect on our estimated daily rate at the time we adopted the new method was to reduce the daily rate by 1.6%. Had we done it earlier in the week before recent updates by China and Russia, it would have increased the daily rate by as much as 8.7%. We’ve tested several methods of calculating the estimated daily rate over time and have found this one to provide the most accurate and consistent estimates.

Tom RandallSenior Reporter

02/19 13:14 ET

Over the last week, the average rate of shots given in the U.S. has been stuck around 1.6 million to 1.7 million a day. What’s keeping the U.S. from getting its rate up?

Two things -- supply, and snow.

On the supply side, the U.S. is at a (temporary) ceiling. About 10 million to 15 million doses a week have been getting shipped out, which equates to 1.4 million to 2.1 million doses a day. The recent seven-day averages for vaccine administrations are smack in the middle of that number, at 1.6 million to 1.7 million doses a day.

The second cause is the winter storms that blanketed the U.S. in snow and ice. As much as 73% of the country was covered in snow, the greatest extent on record, with data going back to 2003. Many vaccine appointments were put on hold.

Texas is one of the biggest contributors to vaccine administrations and was hit particularly hard.

Before the storms, Texas was averaging around 135,000 doses a day. On Monday and Thursday this week, it recorded 22,000 and 21,000 doses, respectively. That’s more than 110,000 fewer doses on those days, dragging down both state and national numbers.

Methodology and Analysis for Bloomberg’s Covid Vaccine Tracker

This helps explain why on Sunday, the U.S. reported a record 2.4 million vaccinations and then today reported a far more pedestrian 1.7 million. A slump? No. It was just the weekend, data-lagging its way into Monday (and probably tomorrow as well).

The good news is that it was the largest number of Monday-reported vaccinations seen so far. That suggests the winter storm make-up is still happening, and the overall vaccinations continue to trend upward.

Drew ArmstrongHealth Editor

03/08 10:37 ET

Later today (we think) the U.S. Centers for Disease Control will reveal how it plans to count Johnson & Johnson’s one-dose shot in its vaccine data.

Up until now, states and the CDC have all been collecting and disseminating data on two-dose vaccines. A one-dose vaccine makes the data more complex. People who get Pfizer Inc. or Moderna Inc.’s vaccine are partially vaccinated after one shot, and fully vaccinated when they have both.

With the new vaccine, people who have one dose of J&J will be fully vaccinated. This makes a mess of data categories reported by states, and will require a revision by the CDC as well.

For now, our understanding is that while doses of J&J’s vaccine have been administered all over the U.S. (we’ve seen it show up in some state data), those data still haven’t made their way into the CDC’s counts.

We expect that will change as soon as today. When that happens, we may also see a one-day bump in doses administered as the CDC dumps several days worth of J&J vaccinations into one day’s reporting. (Remember, we count vaccinations when they’re reported, not when they happen, and many of the entities that report the data do the same.)

It’s not clear yet the size of that bump, but it could be large (and distorting). J&J planned to ship around 3.9 million doses of its vaccine in the first week. While Mondays and Tuesdays have historically been slow days for vaccine reports, we could see a very big one-day number of doses administered because of the backlog.

If that happens, it will skew our seven-day average and make things look a little weird for a bit. Skewing things further, states have been told they won’t get a shipment of J&J’s vaccines in the second week of their availability -- so we could see a bump, then a lull.

Things should normalize as J&J shipments smooth out in the back half of the month, by which point around 20 million doses from the company will have been distributed.

Drew ArmstrongHealth Editor

03/08 20:14 ET

Earlier today, I wrote that I thought there could be a substantial one-day bump the doses administered count when J&J’s vaccine got added to the CDC’s reporting.

CDC updated their methodology to include J&J tonight, and the increase is less a bump than a blip.

So far, according to CDC’s data, only about 200,000 doses of J&J’s vaccine have been logged in the federal system, out of around 3.9 million doses set for distribution for last week.

In addition, we’d originally been guided by CDC that J&J’s dose count would be included starting only today. In fact, we’ve learned, they’ve been in the numbers since Friday -- just not broken out. (It’s not yet clear whether they were being counted as first or second doses.)

So how are these J&J doses being reported?

Under CDC’s new categories, people are counted as having “at least one dose” or being “fully vaccinated.” J&J vaccine recipients get counted in both totals, according to the CDC methodology. (One potential point of confusion: Some people get a first dose of Pfizer Inc.’s or Moderna Inc.’s two-shot vaccines, and then later a dose of J&J’s.)

Our tracker uses a similar description. In the meantime, expect the J&J numbers to grow from here. Just not as fast as I thought.

Drew ArmstrongHealth Editor

03/09 18:22 ET

Today we turned the Bloomberg Vaccine Tracker upside down—literally.

When we first published the tracker on Dec. 18, we started with a map of vaccinations in the U.S. We decided not to open with a global outlook for several reasons, chief among them that there were few countries outside the U.S. that had any vaccinations to speak of. The global map was mostly a sad gray—the color we use to indicate that there’s no data to report.

Until recently, it has made sense to keep the focus on the U.S. Many of the trends we’ve seen in the U.S. soon repeat themselves in vaccine campaigns around the world. Today the U.S. has administered nearly 100 million doses—more than the next few countries combined—and manufacturers plan to double the pace in the weeks to come. A quarter of adults in the U.S. have received at least one dose, and a hard winter feels like it’s coming to an end.

A similar degree of optimism hasn’t quite yet covered the globe, but the situation is improving daily. We’re now tracking active vaccination campaigns in 118 countries. As America’s chief scientific advisor Anthony Fauci has pointed out, even highly vaccinated countries will continue to live with outbreaks until an effective vaccine reaches everyone. It is a global disease that requires a global solution.

So we’ve made some changes. Starting today, you’ll see our global map first. Interest in the U.S. remains high, so we’ve defaulted some of our global charts to U.S. views, which you can change in the drop-down country lists. All of the other U.S. data and charts still exist—just a bit further down the page. Readers who want to go straight to the state-by-state figures can bookmark this link.

Finally, we’re working on some new tools to track the vaccination campaign, both for our global readers and for the U.S. specifically. The “big flip” of our vaccine tracker will help make these projects possible. Stay tuned, and use our Tips and Feedback tool to let us know what you think.

Tom RandallSenior Reporter

03/13 20:00 ET

On Saturday, the U.S. reported a huge number of vaccinations: 4.6 million, more than 50% higher than any previous daily report.

There’s always a reason to look at big increases or decreases with caution, and today is no different.

Usually, the U.S. Centers for Disease Control and Prevention pulls their numbers from state data at 6 a.m. (Remember, vaccinations are logged under this methodology when they are reported, not when they happen.)

Today, the CDC pulled the data much later in the day, according to the agency. That’s a small methodological change that means a big (seeming) increase in doses. But it essentially robs doses from Sunday’s report, presuming CDC goes back to its normal data pull and reporting schedule.

If it had pulled the data at the normal time, today’s reported doses would have been 2.98 million, according to the agency. That’s still a big number -- a record, in fact -- but it’s no 4.6 million.

Look for Sunday’s total to be lower than usual as a result.

In the meantime, the moving 7-day average remains the best measure to watch, since it helps smooth these types of changes. And that average, it’s worth noting, keeps going up.

Drew ArmstrongHealth Editor

03/18 18:28 ET

by Tom Randall

We’ve introduced a new way to measure vaccinations around the world: “people covered.” This is a calculated figure based on the number of doses administered and the mix of vaccine types used. This figure accounts for the doses required for each type of shot—some require two doses, others just one.

Here are a few examples to show how it’s calculated. Keep in mind that Pfizer and Moderna require two doses per vaccination, while J&J is a single-dose vaccination:

Pfizer: 10M doses administered = 5M people covered

Moderna: 10M doses administered = 5M people covered

J&J: 10M doses administered = 10M people covered

Previously we mostly relied on per capita doses administered when comparing the pace of vaccinations between states or countries. The introduction of J&J’s single-dose vaccine made that metric imprecise.

Why don’t we just report the number of people “fully vaccinated”?

In our data tables, we continue to report people who have received “at least one dose” and those who are “fully vaccinated”—categories favored by some public health agencies including the U.S. CDC. Each of these metrics has its drawbacks.

The category, “at least one dose,” assigns the same value for a single Pfizer dose as for a J&J dose—even though a Pfizer vaccine recipient will require a follow-up shot. Similarly, counting people only when they are “fully vaccinated” ignores the value of having begun a two-dose regimen.

Another advantage to the new metric is that it can be more broadly used to compare countries, even when complete data aren’t available.

Every country reports vaccinations in a different way. Many don’t include metrics such as “fully vaccinated,” but almost every country reports doses administered. Our calculated rate of “people covered” provides a single, comprehensive metric to compare countries around the world.

Tom RandallSenior Reporter

04/16 10:47 ET

Knowing which vaccines are being supplied in the just as important as knowing how many shots are going into arms.

But because of a bifurcated U.S. distribution system, the public is getting less and less detailed information from the federal government on how many doses each vaccine manufacturer is actually delivering. It’s a situation that makes it harder to know if drugmakers are meeting their delivery commitments and also how the federal government is allocating shots.

To understand the U.S. vaccine distribution, you first need to know that it’s split into two parts.

The first is state-based. The Department of Health and Human Services each week announces vaccine allocations. States order doses, and they’re shipped out. This has been the biggest part of the system, accounting for 17.6 million of 28 million doses made available this week.

We have great data for this program: Each week, the Centers for Disease Control and Prevention publishes a file showing how many doses have been allocated to each state and which drug company made them.

The second part is what we call the federal-direct program. These are doses that the U.S. sends directly to pharmacies, federally qualified health centers, kidney-care centers and other locations. These doses make up an increasingly large part of weekly shipments — 10.4 million out of 28 million doses made available this week.

There is far less public data here. The White House announces a grand total once a week but doesn’t provide breakdowns of which vaccines are going where, or in what quantities. While the CDC has begun publishing data on how many vaccines are flowing to those channels, it doesn’t provide data on which vaccines are being sent.

Methodology and Analysis for Bloomberg’s Covid Vaccine Tracker

As a result, the Bloomberg Vaccine Tracker’s supply chart contains an increasingly large category of “unknown” doses — shots that are being delivered to the federal government but without public information on the manufacturer.

We’ve repeatedly asked HHS for this information, and have been told for several weeks that they’re working on it. We still don’t have it.

Getting a complete picture is important for several reasons. It tells us, in near-real-time, if vaccine makers like Pfizer Inc., Moderna Inc. and Johnson & Johnson are meeting their production commitments.

Detailed data on the federal program would also reveal if the U.S. is moving away from its strategy of allocating vaccines largely on a per-capita basis to distributing them by demand or need, something that governors and mayors have called for as surpluses build in some places. It would also give us clarity on what the impact will be, and where, from the pause in J&J vaccinations over safety concerns.

This administration, to its credit, has been increasingly responsive to our vaccine data questions. The CDC has produced more information about how vaccines are used, and has been more transparent about explaining that information.

But data drives accountability. And without knowing how many vaccine doses are being handed over to the U.S. and from which companies, it’s harder to know if the promises made by drugmakers are being met, harder to know how those shots are being allocated, and harder to know where the U.S. vaccine campaign is headed.

Drew ArmstrongHealth Editor

04/26 16:13 ET

This week, we’ve changed our data source for Netherlands. The reason we’re doing it provides a little insight into the complexities of vaccine tracking.

From the start of the campaign, the Netherlands has published two different figures for cumulative vaccinations. One is described as the “reported number” and the other as a “calculated number.” These figures diverge significantly—the calculated number is currently 66% higher. Starting this week, we will switch to using the higher figure.

This isn’t the only time a country has presented a “calculated” figure. This pandemic was the first time in history when there was an urgent need to report vaccination numbers around the world in real time. Most countries didn’t have a system for doctors to centrally report each shot, so some places, like the Netherlands, calculated their vaccinations using supply-chain data.

At the Bloomberg Vaccine Tracker, we chose not to use calculated figures unless there was no alternative. The whole point, after all, was to know how many vaccines were actually being consumed. But as other countries switched to reporting actual vaccinations, the Netherlands persisted with its calculated figure.

The Dutch insist that the calculated figure is the best one to use. The reported figure doesn’t include many hospitals, general practices, and long-term care institutions—so for those groups the country relies on estimates based on doses distributed by a separate agency.

This calculated figure is commonly accepted by other vaccine trackers, including the World Health Organization, and is used by government officials and Dutch news. There are few subjects that generate more consistent feedback on the Bloomberg Tracker survey. We have decided to make an exception and join the consensus.

It means there will be a period of data weirdness for the next few weeks as our Dutch charts absorbs the sudden addition of a few million doses.

Tom RandallSenior Reporter

05/05 17:14 ET

We’re making a data change to the part of our graphic that shows how many vaccine doses have been promised and delivered in the U.S.

There are several reasons to do this. The first is an error on our part. Some doses from the beginning of the U.S. rollout were being removed each week due to a mistake made in the way we stored and presented data. As a result, our cumulative total for vaccines available was short by millions of doses.

The other issue is that, with more and more doses going out via federal programs for which we don’t receive detailed data, the chart presents an increasingly misleading picture of whether or not drugmakers are meeting their delivery promises. That problem would only continue to get worse.

As a result, we’ve put up a correction noting the removal of the incorrect data. What remains is a chart of the promised deliveries of each drugmaker. If we locate a better dataset that allows us to accurately portray those deliveries over time, we’ll add it to the chart.

Drew ArmstrongHealth Editor

06/11 09:41 ET

Since the start of the Bloomberg Vaccine Tracker, we’ve focused on collecting data by hand. It was a (labor-intensive) choice we felt was necessary to understand information that can vary significantly from state to state and country to country.

In the last few weeks, we’ve begun to automate some of that data work. We’re doing this for several reasons.

In some places, country data sources are stable enough that they can be scraped. This saves huge amounts of time for our team. In addition, we now have reliable data from the World Health Organization for many regions, including countries that report less frequently or less comprehensively.

We’ve begun using the WHO’s data and our own scrapers for the majority of the countries we cover, and using the extra time to focus on countries that remain difficult to automate.

That does not mean that there’s no human touch. We’re always on the lookout for problems with automated data, and carefully vet new sources before turning them on. But at this point in the project, we felt we were ready to let the computers do some of the work.

We’ve updated our source notes and FAQ to reflect these changes, and we’ll keep you informed of others going forward.

Drew ArmstrongHealth Editor

©2021 Bloomberg L.P.

BQ Install

Bloomberg Quint

Add BloombergQuint App to Home screen.