Just published in LSE Impact, on the “deep research” summary-and-citation tools from both the AI giants and the likes of Elsevier:
At the core of all these tools is a series of verbs: to surface, to rank, to summarise, and to recommend. In this sense the AI products are already acting as knowledge arbitrators, by picking winners and losers according to what they make visible. There are two big and interlocking problems with this role: the models are trained on the scholarly past, and their filtering logic is inscrutable. As a result, they may smuggle in the many biases that mark the history of scholarship, around gender, geography, and other lines of difference.
In this context it is useful to revive an old concept in the sociology of science. According to the Matthew effect, named by Robert Merton decades ago, prominent and well-cited scholars tend to receive still more prominence and citations. The flip side is that less-cited scholars tend to slip into further obscurity over time. These dynamics of cumulative advantage have, in practice, served to amplify the knowledge system’s patterned inequalities, with, for example, diminished citations for female scientists, a development aptly labeled the Matilda effect by Margaret Rossiter in 1993.
The deployment of generative AI tools in scholarship, as their use picks up, may produce an even worse Matthew effect, on the scale of Elsevier and with no paper trail. The problem is analogous to the well-documented bias-smuggling with existing generative models; image tools trained on, for example, mostly white and male photos reproduce the skew in their prompt-generated outputs. With our bias-laden scholarship as training data (“120 years of research”, in Web of Science’s boast), the research models may spit out results that, in effect, double-down on inequality.
What is worse is that we won’t really know, due to the models’ black-boxed character. Thus, the tools may act as laundering machines—context-erasing abstractions that disguise their probabilistic “reasoning”. Existing biases, like male academics’ propensity for self-citation, may win a fresh coat of algorithmic legitimacy. Or consider centre-periphery dynamics along North-South and native-English-speaking lines: Gaps once traceable to geopolitical history, including the legacy of European colonialism, may be buried still deeper. The models, in short, could serve as privilege multipliers. Early evidence suggests that’s exactly what’s happening.
The piece is here.