A layperson's guide to evaluating science
How to understand when science is working as it should and when it is not.
I wish this column were not necessary. The fact that it is repudiates a lot of my life’s work. But the only way to fix what is broken in science is to call it out.
I’ve made no secret of the fact that I believe that 21st-century science has largely betrayed the trust of Americans. For me personally, this is a bitter pill to swallow. I have believed since I was old enough to know what science was that it was our best hope for solving humanity’s problems. I’m less than thrilled with how that’s worked out since the turn of the century.
All reputable science is based on adherence to the scientific method. The scientific method has been with us, a few twists and turns aside, since the age of enlightenment. If you want to know how good a scientific idea, concept, or theory happens to be, a good indication of its veracity is to see how well it fares when filtered through the scientific method. In the case of Newtonian gravity, this happens to be extremely well. In the case of gender-affirming treatments for minors, not so much.
The scientific method, as it is practiced in the real world, proceeds roughly along the following lines: 1) Develop a testable hypothesis; 2) Acquire funding (peer review I); 3) Conduct experiments and/or perform research; 4) Analyze data; 5) Publish results (peer review II); 6) Report findings at professional meetings (peer review III).
As you can see, peer review plays no small role in the scientific method. Peer review is a process by which others with qualifications similar to your own, who know reasonably as much about the science of what you are doing as you do, judge your work.
Peer review shows up three times in the practical scientific method: in the acquisition of funding (no bucks, no Buck Rogers); in publishing; and in conference talks. Peer review is essential to science. If the universe consisted of only one person, science would not be possible. Peer review is what makes science self-correcting. At least that is what it is supposed to do.
One of the big problems in science right now is that peer review, in many fields, is either lousy or nonexistent. This problem is more acute in the social and behavioral sciences, where activists have often been successful in hijacking peer review, but is increasingly prevalent in the medical and physical sciences as well.
One of my least favorite terms, vis-à-vis science, is “a new study shows.” A single study, until it has undergone peer review, is an anecdote. Like all anecdotes, it may well turn out, in the fullness of time, to have merit, but that isn’t the sole judgement of the authors of the study or of non-peer readers. It’s the judgement of scientific peers.
The failure of scientists and the media to make this distinction, more than anything else, has precipitated a rapid decline in the public’s trust in science since the turn of the century. Far too many scientists have followed the infamous path paved by Pons and Fleischmann of conducting science via media release instead of peer review.
In the early days of the COVID pandemic, for instance, there were rapidly conducted studies that produced wildly conflicting results based on small sets of data. This was unavoidable, given the nature of the pandemic, but non-optimal. These studies spread like wildfire in the media.
The problem was not that the early data was scarce, as was to be expected; it was that incredibly consequential decisions were being made based on non-peer-reviewed studies as if they were the last word. It would be difficult to argue, in retrospect, that the world was well served by the abandonment of sound science because of panic.
Even when a study is peer-reviewed, the quality of the review matters. There are entire scientific fields in which peer review appears to be broken. Each morning, I start my over-coffee reading with the latest from Retraction Watch, a website that keeps track of retracted papers or papers that have been cited with expressions of concern.
Bearing in mind that each paper published in an academic or scientific journal has already undergone some peer review, the number of retracted papers is stunning. The number of COVID-19 papers alone is in the hundreds. The reasons for retractions and expressions of concern run the gamut from extreme sloppiness and flawed methodology to innocent mistakes to outright fraud. My jaw hits the ground often while I’m reading RW.
The scientific method is supposed to ensure that all scientific work is reviewed by the best and brightest before it is accepted as advancing the body of scientific knowledge. This is not nearly as true in the contemporary world as it should be. Some of this is the result of inattention, some is due to incompetence, some is due to resources, and some, frankly, is laziness. It takes a lot to adequately review another’s scientific work. One must devote, at a bare minimum, some time and resources to this endeavor. And, at some point, one must have the ability and gravitas to call out poor or mistaken work.
These days, the latter is more difficult than it sounds. In a post-meritocracy world, personal attacks on those offering well-grounded criticism are very commonplace.
The guardrails that the scientific method imposes on science are essential because scientists are just like members of all other professions: some are good, some are bad, and most are average. Even great scientists occasionally make faulty assumptions or mistakes, and this isn’t necessarily bad.
One of the greatest experiments in physics, the Michelson-Morley experiment, had a null result. Yet it was a success. It was the failure of the Michelson-Morley experiment to verify the presence of luminiferous ether that reset thinking on how light propagated through the vacuum of space, paving the way for Einstein’s theories of special and general relativity in spacetime.
So how, exactly, is a layperson who is far from an expert in a scientific field supposed to judge the merits of scientific findings? In a word, trust. When science works properly, one outside of any field should have the expectation that the professionals within that field have done their jobs when it comes to validating any work. If people in the fields of science, academia, and the media were doing their jobs, it would be easier for a layperson to trust science.
Unfortunately, that’s not our current situation. Far from it, in fact. The media flat-out sucks when it comes to covering science. Even when they lack a social agenda to color their coverage of science, they just aren’t particularly good at their jobs. Academia has become infested with legions of administrators whose jobs are not to educate, perform research, or provide university service but to justify their existence with nonsense like obtrusive levels of bean counting and DEI. And anymore, more than a few scientists are, frankly, terrible.
If I may draw a parallel, popular music reached a zenith in the 1970’s. Back then, popular music was infused with incredibly talented players, engineers, and producers who drew inspiration from multiple genres of music, including the blues, jazz, funk, soul, and classical. Those folks were solid professionals. Some of the pop recordings made in the ‘70s have a harmonic and melodic sophistication that hasn’t made its way onto a contemporary playlist in decades.
These days, though there is still incredible musical talent to be found in the world, there aren’t that many great recordings. In my life as an audio engineer and musician, we have an expression that addresses this: “Before there was Pro Tools, there were pros.”
These days, a critical mass of people have learned to cut corners in music because simplicity is easier to produce and sells better than sophistication. You can win a Grammy with a recording made in your bedroom. Can you still find great contemporary music? Yes, you can. But you are going to have to look for it. And the arbiters of musical taste, like the Grammys or Rolling Stone magazine, aren’t going to be much help.
These days, in science, we’ve similarly lost our way. We are willing to tolerate completely stupid arguments, like the scientific aspirations of Bigfoot, but we want to censor completely reasonable arguments for things like the lab leak hypothesis concerning the origins of COVID-19. The former is en vogue because it sends a thrill down the legs of those who are obsessed with the evidence-free paradigm, ways of knowing. The latter, because it runs afoul of a preferred narrative of the current powers that be.
I am hardly ever in favor of shutting down any qualified scientific point of view. As annoying as it sometimes is to have to argue with those on the fringes of science, you must be willing to listen to qualified peers who have different ideas. That’s what makes science better. As we’ve found recently, those allegedly on the fringes of scientific thinking were often shown, in the fullness of time, to be more correct than what was once perceived as mainstream.
So what should we do about all of this? To tell you the truth, I’m about out of moves. No one individual can possibly acquire the expertise required to independently judge the merits of the entire vast body of scientific knowledge. In physics, we tend to offer strong opinions on only our narrow fields of expertise. At some point, one must find experts to trust.
But that doesn’t mean that you cannot develop a nose for ferreting out scientific bullshit. When you come across a “study,” find out who funded it and if it was peer-reviewed. Those two things will tell you a lot. Be aware of the fact that the onus in science is on proving that your facts are true. It’s not on others to show that they are untrue (though that is part of the process). Be aware of the fact that extraordinary claims require extraordinary proof.
And finally, everyone has some motivation for doing anything, including science. Figure out what the motivation is, and you are a lot closer to discerning the truth in science that you may not otherwise fully understand.
Associated Press and Idaho Press Club-winning columnist Martin Hackworth of Pocatello is a physicist, writer, and retired Idaho State University faculty member who now spends his time with family, riding bicycles and motorcycles, and arranging and playing music. Follow him on Twitter @MartinHackworth, on Facebook at facebook.com/martin.hackworth, and on Substack at martinhackworthsubstack.com.