> Completely skimming it, the test sites had false positive rates of 0.6 - 1.8%.
In other words, the test is almost completely useless. Given how low the base rate of terrorists sneaking through bombs is, a positive test result on one of these machines is >99.9% likely to be a false positive.
Let's do the math. We'll assume there are never any false negatives and just look at the positive results. Let's simplify "0.6-1.8%" and just call that "1%". Out of 100,000 bags, let's assume that ONE contains a bomb being snuck through by a terrorist. 1/100,000 is our postulated base rate of terrorism. 1/100 is our false positive rate on the test.
So let's put 100,000 bags through the machine. There will be 1,000 false positives and one true positive. which means that if some bag "tests positive for explosives" the odds are a-thousand-to-one against that being a valid result.
"But," I hear you cry, "we RUN IT THROUGH AGAIN when we get a positive result!"
Sure, that would work GREAT if false positive results were COMPLETELY RANDOM. But they're not. More likely than not, the false positive is being triggered by something that is or was actually in the bag. So when you run it again, there's a good chance that it'll trigger again. The "false" part of "false positive" is that the thing in the bag that it's triggering on...isn't an explosive. It's just some other chemical.
It's not just the high false positive rate, it's the combination of a high false positive rate with a really low base rate. (If there were a lot more terrorists, the test might be worth doing!)
The combination of those two facts means that getting a hit on explosives gives you almost no increase in information. Any look at plausible numbers makes the test nearly useless at finding explosives, though it's still at least theoretically possible the test could serve as a sort of deterrent, albeit primarily a deterrent against movie-plot threats.
My guess is that the true purpose of this test is to make the people who sell explosives-testing equipment comfortably well off. Any other purpose would be served roughly as well with a box that triggered based on random number selection.
I am extremely, bitterly familiar with the base rate fallacy (I was an intrusion detection researcher, and spend 4+ years working on statistical anomaly detection the Internet backbone).
The fallacy doesn't make a value judgement. It points out something counterintuitive about the accuracy of a filter or test. That thing is important, but not dispositive. If the base rate is low and the false positive rate is percentagewise high but the overall number of hits is manageable, low-power statistical tests can have utility as pre-filters.
I have the same thought every time I go through airport security ("whoever designed this probably doesn't know about the base rate fallacy"), but if the system is only ejecting 1-2 candidates per station per hour for expensive "offline" screening, it's not untenable.
It's not untenable in the sense that it's possible to get that amount of work done, but that doesn't make it worth doing. TSA is fundamentally trying to solve an unsolvable problem. No matter how many resources they throw at it, there will always be ways to evade their checks. A smart terrorist will find a way to get through undetected, find a way to avoid the check entirely (say, by bribing a TSA employee or airport employee), or will just bomb the next available target of opportunity - perhaps the security line itself. There is simply no plausible scenario in which the TSA's checks actually prevent terrorism.
(Yes, one could postulate really stupid terrorists who somehow don't realize they'll get caught going through security, but terrorists who are that stupid are likely to have their plans fail without the TSA's help. See also: the shoe bomber.)
The double-negative is bad style, but the fact is actually reassuring. It means that the typical TSA agent sees a ton of positive results each week and has never seen a valid result. In other words, the agent is not fussed at all by the positive test outcome; they just have to double check to clear you.
It's expected. Even in the most optimistic of circumstances, you have to believe that innocent people will get caught in an anti-terrorism net. Same as most things- innocent people get arrested far away from airports, too.
The question (and failure) is what happens after that. But false positives in themselves isn't surprising.
