I'm going to jump right in. I want to talk about the IHME-UW (Institute for Health Metrics and Evaluation at the University of Washington) modeling and why it concerns me.
Don't get me wrong. I look at these almost every day. I keep track of several states and the national estimates on a regular basis because I want to see how they change. IHME is local to me and there is a certain regional pride in how quickly they put the word out. I want them to be right.
But I'm afraid they are not -- or at least that people who look at the model get the wrong idea -- and this image sums up why:
This is today's modeled estimate of total deaths in Washington State for the period up to 1 August, assuming there is not a second wave of infections or a change in policy and/or compliance. It's a wonderful image, including easy to see and understand where we are in relation to the peak day of deaths, with a clearly understandable image showing how the model predicts deaths will drop, including a range of uncertainty. You an't see it, but higher up on the page there are links to a FAQ, updates, and the original article describing the model -- everything you need to know.
And you will probably still walk away from this image with a severe misunderstanding. That misunderstanding? That this model predicted 42 deaths on 6 April.
It did not.
As of 6 April, this model was predicting a peak death toll in WA state of 19 deaths on 2 April -- a prediction that was already underestimating the peak by at least 5 deaths (24 on 4 April) and which underestimated the peak that day of 45 deaths. These numbers are shown on the graph, and if you read carefully and think carefully you will understand that the solid red line is not the model prediction, it is reality.
All models are wrong. Some are useful.
I am not an expert in epidemic modeling. Far from it. I'm a polymath with significant training in statistics, modeling, and systems analysis. My objection to this model is not the underlying model* (which is widely considered optimistic in the current circumstances), the update to the model in early April which resulted in substantial changes (lessening the predicted severity in every example I have tracked), nor the implementation of the model.
It's the display of the results.
If you are not careful, you will walk away from this image thinking that the model is far more accurate (that is, reflects reality better) than it actually is. That is dangerous.
Because all models are wrong, you should be looking at a bunch of different models. And I do. But there is some indication that national policy is being set by looking at this particular model.
Setting national policy on the basis of any single model is a serious mistake. Even the best, most sincere of modelers end up with blind spots. We can get married to something specific in the model, our own cleverness, a type of model, or something else. We can get things wrong. Sometimes those things are subtle, sometimes obvious. One reason to use multiple models, preferably developed by groups or individuals without regular communication, is to avoid relying entirely upon something with an error in it. Multiple models give you a better chance to see such errors.
* OK, I have some concerns about the model too, but that's a separate conversation.
Here are some additional links I think you should read:
The IHME Epidemiological Model, by Cheryl Rofer at Nuclear Diner. Cheryl is retired from Los Alamos and is way smarter than I am. The links in her piece are also worth following.
Some basic works (all PDF) on epidemic modeling:
* Columbia University "Introduction to Epidemic Modelling"
* American Journal of Epidemiology "Epidemic Modeling: An Introduction"
* University of Trento "Mathematical Modeling of Epidemics"