The Reliability Roulette of Testing (updated)

George Rebane

[This is the addended transcript of my regular KVMR commentary broadcast on 21 July 2020.]

It’s again time to bring up testing for the Covid-19 virus.  Let me start by saying that testing has shown no value in formulating reasonable pandemic response policies by public officials.  It has become an election year political football, a game played by both parties and all our healthcare institutions starting with the Centers for Disease Control and Prevention and the National Institutes of Health.  And there’s a lot more to be said to support such a strong and wide-reaching assertion.

First, let’s understand that there are two kinds of testing that are of interest – testing for the presence of covid infection, or simply infection testing, and testing to detect whether someone has had the covid virus and could now be immune, or simply immunity testing.  There are many different tests that are being administered here abroad for both infection and immunity.  Infection testing of an individual or a random sample from a target population – say, Sacramento County – gives an unreliable snapshot that becomes dated almost as fast as the test results are available.  Their major use is for on-the-spot screening of people going into a controlled environment like a hospital.  Infection testing of random samples is pretty much worthless for two intertwined reasons, the frangibility of abysmally unreliable population test results.  In simpler words, they’re inaccurate to begin with, and they go stale immediately.

So let’s consider test reliability.  Here’s a hot flash for most people listening tonight.  There is little to nothing known of any given test’s sensitivity and specificity numbers.  They range all over the place and, even if known, results vary greatly depending on how and when the test is administered.  Test sensitivity is the fraction of positive test results that come from testing infected people – high fractions or percentages are good.  Test specificity is the fraction of negative test results that come from testing uninfected people.  Again, a high fraction or percentage is good.  Specificity is also calculated by subtracting the test’s false alarm rate from one or a hundred percent.  A false alarm is a positive test result obtained from a healthy person.

Test sensitivities range all the way from 80% to 99%, and their specificities range from, say, 50% to 95%.  Any given test has both numbers which describe its overall reliability.  Now we come to the problems.  Sensitivity and specificity are usually on opposite ends of a see-saw.  When one is high, the other tends to be low – in short, there is no free lunch – and there’s the rub.  We start by knowing the reliability only to within a certain range, if that.  And when we do the math to calculate either the probability of a person having the disease, or what fraction of the target population is currently infected, the results vary all over the place.  Definitely they are not of the kind from which one can support reasonable response policies.  For the listener not afraid of a little algebra, I have posted  all the math details for these arguments on Rebane’s Ruminations, where I respond to readers’ comments and questions.

So where does that leave testing.  I’m afraid it leaves it exactly where it has been for the last five months – a political football played in an election year mudbath by both sides in an attempt to sway the lightly read or simply ignorant voter with meaningless numbers, claims of ‘settled science’, and just plain lies such as the attempt to convince you that more tests and testing is somehow better.  The science about all parts of Covid-19 is anything but settled, and therefore policies are constantly changing.  No one can point to any test results that have reasonably impacted any response policies across the land.  The only reliable indicators that do impact policies, but not always reasonably, are body counts and current active cases as tallied in healthcare facilities.  And some games are even played with these stats.  Everything else is pretty much garbage in – garbage out.

So what to do?  Your intrepid commentator recommends taking reasonable care by wearing a mask when out and about, and the older you are, the more crowds you should avoid.  Think about covid droplets in the air and use common sense to reduce risks.  But for heaven’s sake don’t crawl back into a hole; get out there to live, consume, and work – vaccines will be along soon enough.  Until then keep enjoying life, but a bit more carefully.

My name is Rebane, and I also expand on this and related themes on Rebane’s Ruminations where the addended transcript of this commentary is posted with relevant links, and where such issues are debated extensively.  However, my views are not necessarily shared by KVMR.  Thank you for listening.

[Addendum]  Let me start by confirming President Trump’s claim that testing increased the apparent or measured incidence of C19 (my abbreviation for the Covid-19 virus).  All the media and even many conservative lamebrains gave the president crap for making that assertion.  And his crack epidemiologist team, who should have known better and supported him, just remained quiet and let him swing in the wind.  Here’s how that claim actually works out to be correct.

Say, the prevalence of C19 infections is actually 5% of the target population which is sampled and tested with a test that is 95% sensitive and 85% specific.  That means out of every 1,000 people in the random sample only 50 are infected.  Yet when the test is applied to those 1,000, it turns out that 190 test positive, and it is reported that suddenly there is almost a four-fold increase in the number of covid cases.  But that 190 is a specious number caused by administering a less than reliable test.  And if the cowering ‘experts’ in Washington don’t step up and explain the matter to the public, the president is made out by the lamestream to look like an idiot who claims that C19 testing infects healthy people with the virus.

I’m not sure how many RR readers are astute enough to understand any of this, but I have covered it all in nauseous detail in my recent yawn-garnering commentaries on testing and the pdfs that can be downloaded from these.  The simple fact (yes, simple!) is that the expected number of positive tests is calculated from E[N(T)] = {P(T|V)P(V) + [1-P(-T|-V)]*[1-P(V)]}*Nsample.  Substituting from above E[N(T)] = {0.95*0.05 + [1-0.85]*[1-0.05]}*1000 = 190.  And we can show countless other examples of how raw test results are grossly misleading, and doubly so when the test reliability numbers are not even known or known to a usable range.

Now if you want to consider the real zoo actually taking place in ALL halls of governance, consider adding to the above formulation that the actual prevalence is not known.  Say, for example, its 7% instead of 5%.  Then you do have the added error factor that even unreliable tests, such as the one above, will discover more previously unknown infected people who show up in the raw count of positive test results. (see here on how prevalence affects test results). As I said in my commentary, population test results are near worthless for making/adjusting response policies.  If you want reliable data, just count the bodies and filled hospital beds.

In sum, anyone claiming to speak from science when presenting such results should give the error bounds of the stats.  That none of this takes place speaks to the utter political bullpucky that these reported numbers evoke.  So now reexamine the pronouncements, policies, and their bases from self-sure politicians like Governors Newsom and Cuomo.

For readers who care to look deeper into the test reliability issues, please see my test commentaries, and read some of the following citations.  Also, googling ‘covid test reliability’ wouldn’t hurt.  But you’ll find getting actual data on reliability is like pulling hen’s teeth.

CDC’s testing startpage (here).  Note the dearth of test reliability discussion or data.

DARK Daily is a site for professional researchers and lab technicians – ‘Multiple Studies Raise Questions About Reliability of Clinical Laboratory COVID-19 Diagnostic Tests’.

Kaiser Health News – ‘As Problems Grow With Abbott’s Fast COVID Test, FDA Standards Are Under Fire’.

New York Intelligencer – ‘How Reliable Are COVID-19 Tests?’

Cochrane appears to be a non-profit information library for healthcare professionals (here)

ARUP Laboratories – ‘How Accurate Are COVID-10 Tests? Many Factors Can Affect Sensitivity, Specificity of Test Results’.

[update]  Tonight, as if to underline the point of this commentary, FN had the President’s covid news conference followed by an interview with Dr Birx, the administration’s national response coordinator.  We were proudly told that the US has now administered over 50M tests, and millions more are on the way.  Both the president and Dr Birx tried to sound as if more testing would slow down or stop the spread of C19.  But during both interviews, neither was able to say one word about how 3-4 day delayed test results would do anything to slow the pandemic or what the test results are being used for.  And after citing “test positivity” data (raw count of test positives, in some places up to 10%), Dr Birx totally avoided mentioning that even this time-late data was useless because it came from people who self-selected for testing from a hodge-podge of locations administering different types of tests – i.e. the whole mess is confounded due to administering tests of various reliabilities to a totally non-random sample of test takers.  In short, our testing program continues to be beyond sophomoric.  The bottom line from our political leaders is that if only we all can have strong faith in testing, then that will be enough to rescue us from the pandemic.  But hey, it worked for Peter Pan, Wendy, and the kids – all they had to do was believe, and they could fly.

Let me finish with some more positive news.  Dr Birx informed the nation that the latest data and analysis shows that the country's pre-infected rate is 10X the previously published figure of somewhere around 3%.  So at around 30% of Americans already having survived the C19 virus, we are very close to the rate at which herd immunity really starts kicking in, flattening all curves, and heralding the effective end to the epidemic - it will probably always remain with us like the annual flu.  But what this revelation indicates is that the new shutdown regulations imposed by various states and localities is even more arbitrary and ineffective than ever.  But then, how will they ever give up all that heady power and their newly discovered level of control over us?

Comments

[Bill Tozer]

Good Informative post, Dr.Rebane.

I learned much and my eyes did not glaze over...for once. :). With that said, I really hate to veer off testing and get into the mudslinging of C-19 in general and divert from The Reliability Roulette of Testing. However, I have a C-19 link and I don’t now where else to put it....maybe Letters?

‘The Media and the Virus’

American press coverage of Covid-19 was first dismissive, then alarmist—but always condescending.

https://www.city-journal.org/press-coverage-of-covid-19

[Dan, RN]

Thank you for the solid advice. I will no longer get tested before going back to work at Eskaton. I'll tell the management you said so.

[George Rebane]

DanRN 337pm - "(The tests) major use is for on-the-spot screening of people going into a controlled environment like a hospital." So tell us Dan, when did they drop the literacy requirement for RNs? It appears that in your case Eskaton should administer both a covid infection test and a reading test. Having an illiterate RN administering to elderly patients is probably just as dangerous as having one who is infected.

[Gregory]

G-d bless good RN's but they do not shine in literacy or numeracy. They do a good job doing what the doctors order... this I say with the experience of my first wife having the mixed blessing of having former students of hers in competency mathematics (mostly the arithmetic of fractions) at Sierra College attending to her in one of her final stays at SNMH's Cancerland. She was proud to have been central to their return to a college path and the attainment of their goal to be an RN, but a bit aghast at how little they knew when they started her class.