How to Spot Bad Science

Bad science is, unfortunately, everywhere. Sometimes it results from reporters who don’t understand the story they’re covering and mangle the facts (or who willfully misrepresent findings to twist the story to their own particular worldview). Sometimes people want to make a fast buck, so they lie about the efficacy of their product or service. Sometimes people are just, well, delusional. Here are a few quick things to look out for that may tip you off to the presence of Bad Science.

“Studies say” Well-modeled studies are the backbone of good science. Without testing hypotheses, we can never know if an idea is a stroke of brilliance that revolutionizes our understanding of the world around us or if it’s just an interesting concept that doesn’t actually work. Poorly modeled or deliberately biased studies, however, can be twisted to say just about anything. It can be hard for a scientific layperson to tell the difference sometimes, but there are some red flags to look for. One study that wildly contradicts established science and makes no effort to temper its enthusiasm or confirm results before making a big announcement is likely to turn out to be unsubstantiated. We saw this two years ago with the “discovery” of arsenic-based lifeforms; everyone got majorly excited only for the entire claim to be debunked within days. Contrast that with the Higgs boson announcements, which were only made after months of independently corroborating the findings and were couched in terms that made clear that nothing was 100% certain about the discovery.

The biggest clue that something may not be on the up-and-up is a failure to actually reference or link to the specific studies in question. Sometimes this is just a result of lazy reporting, but if the main proponent of an “amazing breakthrough!” won’t give you the specifics, there’s a decent chance that they’re just making stuff up or trying to hide how crappy the studies were. If something seems too good to be true and is less than forthcoming with details, see what the experts have to say. One of the wonderful things about the growth in blogging is that scientists love to debunk charlatans and usually do so quickly and viciously.

Scary (but vague) buzzwords. A hallmark of bad science is its affinity for attacking nebulous concepts without much actual meaning. If something is touted to combat “toxins” or claims to be a better alternative than “chemicals,” that doesn’t actually tell you much of anything about what it does. “Chemicals” sound like something you should strive to avoid, even if they’re actually completely harmless or even beneficial. Take Dihydrogen Monoxide:

Terrifying, right? Unless you’ve caught on that the above image is a parody of fearmongering and that Dihydrogen Monoxide is better known as H₂O, or water. Are some chemicals dangerous? Absolutely! Should you be afraid of something just because it’s a component of rocket fuel or whatever? Not necessarily. Look for specifics.

“Germs” is another word that’s largely used to misrepresent what a product does. Our lovely editor tipped me off to a product called Halo Germ Defense that you spray in your mouth to supposedly kill the germs you breathe in for up to six hours. And it’s clinically proven! A little digging, however, shows that the claims are misleading and that it likely doesn’t do much, if anything, to keep you from getting sick. In this case I was able to eventually find the link to the touted study to see what the product really does (though they certainly didn’t make it easy to find). When sprayed on the back of the throat, the antimicrobial agent did reduce the quantity of some types of bacteria and yeast on mouth membranes for a period of six hours. However, most of the organisms tested don’t actually spread via coughing or sneezing; rather, they spread via direct contact. There’s also no way of knowing if the antimicrobial actually killed incoming germs or merely some of the ones already present in the mouths of the test subjects (or if they were even exposed to germs during testing and if so, if they remained healthy afterwards). Sprays in the mouth also do very little to protect you from germs breathed in through the nose. And perhaps most damningly, it does absolutely nothing to viruses like the cold or flu. You may assume that it kills cold and flu viruses because it claims to kill “99.9% of Infectious Germs,” but it’s completely ineffective against the diseases that people actually worry about this time of year.

Does it even make sense? Plenty of good science, frankly, doesn’t make a whole lot of sense unless you’re a specialist in the field (and some truly esoteric theories baffle even the experts). That’s not what I’m talking about here. Some claims are so blatantly illogical that it hurts my brain that anyone would believe in them. Take homeopathy, in the specific sense of cures that claim to increase the potency of ingredients by repeatedly diluting them. Anyone who’s ever wound up with a cup of watery soda after the ice melted can tell you that dilution doesn’t increase potency. Why would it work any differently with medication? If one ibuprofen doesn’t knock out your headache, the next time you get one you’ll probably try two or three pills instead of cutting one in half or quarters and expecting that to work better. Any benefits from homeopathic remedies is either pure placebo effect or attributable to other active ingredients that are included at full strength.

“THEY don’t want you to know!” Any story that starts by telling you that they’ve uncovered a secret that the government, “Big Pharma,” or “Corporate overlords” want to keep hidden is probably less than reputable. To be clear, I’m not naive enough to think that any of those entities is entirely honest and scrupulous, but I’m cynical enough to know that if a company can make money off something, they will. If a product is truly a miracle cure for everything that ails you, it won’t be relegated to sad late-night infomercials with websites written in multi-colored Comic Sans. If someone has to convince you of a conspiracy to sell their product, they’re probably the one conspiring (to take your money).

Look at the source. Even the best science can be twisted to say just about anything by people with an agenda. The other day I was looking at the science feed on a newsreader app and was startled to see a headline claiming that astronomers had declared  Phobos to be a hollow spaceship in orbit around Mars. The story turned out to be from UFO Digest and was (unsurprisingly) a complete misrepresentation of the findings. While Phobos does appear to be porous and there have been questions about its origin and composition for decades, no reputable scientist seriously thinks that it’s a spaceship. Any site that takes a particular angle on an issue is unlikely to report findings that contradict its point of view. Sometimes the sites will do you a favor and put the agenda right in their name (for example, Say No to GMOs or Vaccine Lies), while others disguise themselves as legitimate news organizations while still lying through their teeth (cough, Fox News, cough).

Bad science is not merely frustrating, it can be downright dangerous. Ailanthus-altissima has some great ideas about how our schools can do a better job of teaching science literacy and preparing students to separate real advances from lies, and I’ve previously recommended some great science news sites that you can check out. And of course, you can always ask me, and I’ll do my best to figure out what’s going on.