Book review: How Social Science Got Better: Overcoming Bias with More Evidence, Diversity, and Self-Reflection, by Matt Grossmann.
It’s easy for me to become disenchanted with social science when so much of what I read about it is selected from the most pessimistic and controversial reports.
With this book, Grossmann helped me to correct my biased view of the field. While plenty of valid criticisms have been made about social science, many of the complaints lobbed against it are little more than straw men.
Grossmann offers a sweeping overview of the progress that the field has made over the past few decades. His tone is optimistic and hearkens back to Steven Pinker’s Better Angels of our Nature, while maintaining a rigorous (but dry) style akin to the less controversial sections of Robin Hanson’s Age of Em. Throughout the book, Grossmann aims to outdo even Wikipedia in his use of a neutral point of view.
Politics and biases have marred efforts to understand some important topics. Science helps mitigate these biases. By science, I don’t mean the features that typically come to mind first when thinking of science in politicized contexts. Grossmann highlights the importance of empiricism and reason in scientific inquiry, rather than authority.
Grossmann reaffirmed some of my hopes that most people who call themselves scientists use enough empiricism, and enough scientific heuristics, to improve on human knowledge, in spite of biases.
Most of the book focuses on the quality of the scientific literature, rather than whether the average person has become more knowledgeable thanks to science over the past few decades.
Good Trends
“Bayesian and frequentist assumptions, while hardly reconciled, are more regularly compared”.
P-hacking is not getting worse. Researchers are focusing less on statistical significance, and more on evaluating the sensitivity to assumptions.
Researchers have been moving away from rational choice models, grand theories, and Marxism. They’re replacing those with increased empiricism.
Rather than consider one gold standard of experiments, Kubinek says we should consider three silver standards, judging each by how much they reduce uncertainty around our causal knowledge; experiments, qualitative process training, and observational analysis could all play a role depending on the current state of research.
High level disagreements between, e.g. Charles Murray and his critics have been largely replaced by more specific and more civil disagreements over narrower issues, with more precise claims on each side reducing the misunderstandings and more closely approaching the kinds of disputes that get resolved empirically.
The Replication Non-Crisis
Replication failures have uncovered systemic problems with wide areas of science.
Far from dooming those fields, they represent both a symptom of increasing rigor, and a cause of additional effort to improve the quality of research.
One really nice replication failure (which didn’t get enough publicity for me to notice at the time) was The Grievance Studies Affair. Researchers tried to replicate the Sokal hoax. Their falsified (and mostly silly) gender studies papers got a very mixed reaction. Four were published in low impact factor journals. They were exposed as frauds, more than a year after the first one was published, but before the project planned to reveal the hoax.
That’s evidence that even in areas where I expect ideology to often trump science, there are many somewhat competent people who are limiting the harm done by ideology.
Fact Checks
I didn’t check Grossmann’s claims very carefully. My impression is that they’re mostly quite good. One exception stood out: he mentions “overpoliced communities”. He cites a paper that rants about problems with police and with research about police, but which doesn’t seem to document overpolicing. More likely Grossmann cited the paper for its claims about research, and carelessly assumed that communities were overpoliced.
The research that I’ve found which bears directly on overpolicing says the US seems underpoliced.
To repeat: this seems like an exception. I expect it would be hard to find more similarly bad claims, even though the book contains an unusually high number of factual claims.
Another reference that I checked was about why peer review became seen as crucial for scientific legitimacy. Grossmann cites Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States as saying peer review was “adopted largely to justify public scientific support”.
The paper documents that the codification of peer review was quite political. Casual refereeing was moderately common before 1970. There was a sudden shift to formal, mandatory peer review in the 1970s in the US. That coincided closely with a decline in the Cold War related consensus behind government funding of science. The paper provides evidence that some rather arbitrary political considerations were key forces behind the elevation of a minor part of scientific publishing to a sacred ritual. Grossmann’s summary of this is mostly true, but the details sound more petty and short-sighted than Grossmann led me to expect.
Concluding Thoughts
The book is not exciting, eloquent, or controversial enough to become as influential as Pinker.
It’s likely somewhat biased toward only showing the positive trends in social science. But that’s what I needed, to remind me that, for many areas where I’m not actively researching topics, my other information sources have been strongly selected for a focus on what has gone wrong recently.
I highly recommend reading the first few chapters. I’m unsure whether to recommend reading the whole book.
[Thanks to ChatGPT for lightly editing this review.]