Thinking about Science: Fitting Models
Not those kinds of models. And not that kind of fitting. This is serious y’all.
Many people mistakenly believe that science is true or truth. It can’t be. Our collective standards for science expect statements to be falsifiable. As such, we can only know what isn’t true. The goal of science is to get less wrong over time; to stand the test of time by failing to be falsified. Thus, the best we can achieve is “failing to reject a hypothesis.”
Roughly speaking, scientific models attempt to explain and predict. Data are collected through observations and measurements, and a model is fit to those data. A sense of the model’s explanatory power is assessed, and if it passes muster by exceeding some conventions for “good” models, predictions are made and tested. If the model shows predictive power, again understood through scientific norms, then it begins to be accepted for further investigation and discussion. As time passes and as more observations and measurements are taken, the model’s explanatory and predictive power are tested and retested. Scientific models that survive this process, in whole or in part, gain wider acceptance and get distributed as scientific “facts.” That’s how it’s supposed to work anyway*.
*Notice how much the scientific method depends on norms and conventions? Hmmm.
Researchers often attempt to defend models by explaining away data that don’t fit. On the face of it, there’s nothing wrong with that because it’s part of the testing and retesting process that produces the fittest models. There are several ways to defend a model against contradictory findings. These include, but are not limited to the following:
Argue that the countervailing data are outliers.
Attack the methodology.
Replicate findings that support the model.
Ignore the contradictory findings.
Cast aspersions on the undermining research(ers).
Inhibit funding for the undermining research(ers).
Some of these ways of defending the existing model should be part of the scientific debate. And, all of these ways can be used to stifle scientific debate - something discussed ad nauseam all over the Interwebs. Instead, I’m going to discuss a more distressing development.
Well. Not a development really. Essentially, we haven’t come that far since the days of Galileo. I’m going to use evolution as an illustrative example. But I’m not going to debate evolution. I’m going to argue that it now transcends scientific fact, having reached the level of dogma. And I’m not the first to make this argument - Ben Stein made a whole movie about it back in the day, called Expelled. You should see what is said about it - proving the point I’m about to make.
Consider the following religious context. Someone claims that X is such a way and only God knows why. We’ll find out eventually, they say. And what response follows? Derision? Mockery? Pity? Some might argue that the claims about X stem from the age of mythology, and humans simply have the tendency to explain mystery with mysticism, preferring dogma to evidence that doesn’t “fit.” Maybe. But the reason is irrelevant for the present discussion. (I’ll be writing about our obsession with “reasons” soon - hopefully.)
Here’s a challenge. Go find nature documentary clips in which surprising creatures or creature features are discussed. It won’t take long before you observe the following type of statement:
This creature feature is surprising! It appears to confer no evolutionary advantage, and might even present an extra burden to the species. But we’ll find the evolutionary advantage eventually.
Do you see the parallels to the religious context? There’s a model being challenged by data that don’t “fit.” And rather than engaging in scientific debate, an appeal to faith (in the model) is made. In the religious context, that can be ok. Because faith. But in a scientific context?
Here’s the problem in a nutshell: rather than fitting a model to the data, too many scientists are fitting the data to the model. They’ve become the new priesthood leading the inquisition against the non-conformists. And it’s not just in the discussion of evolution. It’s invasive species. It’s climate. It’s COVID. It’s infectious disease (whatever that means). And more.
To be clear, this isn’t about evolution, and I’m not trying to debate the truth value of any scientific model. By construct and norms, they’re guaranteed to be wrong, and they’ll be updated accordingly. We can only fail to reject them, aiming for them to get more right over time. Debating their truth is literally a waste of time. Instead, debate their explanatory and predictive powers, and be transparent about any data that don’t “fit.”
Or - revamp the nature of science.
There are objective ways of knowing whether a model is on the right track - testing generalizability based on a prior predictions of new data can be very helpful in ground proofing models. It's not rocket science, but it's powerfully convincing. And quantitative vs. argumentative. Nested learning in wrappers optimizing data representation, feature selection, parameter optimization, model selection, all in one, perfectly testable - and better than curve-fitting and least-squaresizing everything! Also allows non-linearity - arbitrary but testable.
See this article (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2719747/) about scientific consensus. Especially the quotes of Michael Crichton, like this one: “In science consensus is irrelevant. What are relevant are reproducible results. The greatest scientists in history are great precisely because they broke with the consensus. There is no such thing as consensus science. If it’s consensus, it isn’t science. If it's science, it isn't consensus. Period.” Fitting data to a consensus model is not science. If the data are reproducible, and don’t fit the model, then a scientist should propose a new model.