I’ve had some interesting comments on the “Matthew Effect in AI Summary” piece I published last month at LSE Impact, and thought to share a couple. One was posted by Nicola Willand:
It absolutely happened to me! In 2023 I (female) took the doi of my most cited paper, published during my PhD, and put it into Research Rabbit for a quick bibliometric analysis. In the results it put a (male) Prof of my university as the first author. He was not even in my supervisory team and definitely had nothing to do with the paper, but I had published with him (with me as first author) post-PhD. I was named as the second author. I then thought if I export it into Endnote, Endnote will correct its mistake. It did not. The Prof was still first author. If anyone then cited this incorrectly detailed paper, the Prof’s citations would increase undeservedly and unfairly. I was shocked and unreasonably angry with the Prof 🙂 I repeated the exercise in 2024 with the same incorrect attribution. I have been too afraid to repeat the exercise since but I warn my students about this and use it to demonstrate bias. Thank you for giving it a name 🙂
Another commenter, Aaron Tay, politely challenged the notion that Research Rabbit in 2023 [and 2024] is AI, in the LLM sense. Tay offered a lengthy comment on my piece, pushing back thoughtfully (my replies in brackets):
I think while it is hard to deny LLMs encode streotypes and biases from training data, I think whether AI search tools e.g RAG enabled, Deep Research lead to a Matthew effect (or worse effect) compared to current tools in use is a empirical matter. [agreed—JP]
At end of the day, the question is about retrevial (if you can’t retrieve, you can’t generate/summarise). Because academic search engines (AI or not) are complicated mix of technology and techniques and the degree in which LLMs impact the retrieval differs greatly depending on the tool, you cannot jump to the conclusion they will definitely lead to the Mathhew effect or any other bias exhibited by just the LLM alone. [also agreed—JP]
For example, there are at least 2 studies that have studied this directly and they actually conclude the AI search tools in their study had the opposite effect and led to citing highly cited papers LESS. (see https://allenai.org/blog/asta-citations and https://www.science.org/doi/full/10.1126/science.adw3000). [I see only one study, the Science one, but Tay summarizes it fairly—JP]
As someone who has some experience testing and using such AI search tools, I can tell you the fact that these tools and search deeply and surface papers that are spot on relevant but lesser known and would have been missed if you used Google Scholar or a conventional database. So this could be part of the explaination… [fair—JP]
Finally, I would like to make the point that for many AI search engines, LLMs plays a relatively minor role in retrieval. For example, I was recently looking at results for a study for a different matter and I noticed the tool Consensus.app would randomly have totally irrelevant papers ranked high. Looking at the offenders (extreme high citation outliers) and a look at the documentation (Consensus’s 2nd stage ranking is described as weighting quality, defined as high citation, recency and quality journal) explained what was happening. [very interesting, and could be true of the big-AI and publisher Deep Researches, but they haven’t said]
Notice this analysis has nothing to do with LLMs. In fact, each of the “AI search engines” I studied had very different characteristics, and I am highly skeptical you can tarr all of them with one brush e.g. they all lead to bias X. [agreed again—JP]
I really appreciate both comments, including Tay’s. I have probably come across his name, but wasn’t aware of his blog, Aaron Tay’s Musings about Librarianship. It’s smart and informed—consider his latest post, which uses early Google Scholar complaints measured against its current usefulness to urge that we all keep an open mind about academic AI search tools. I also recommend this summary post from a week earlier. He catalogs his year of testing of the tools, an impressive array indeed.
Tay is coming from a pragmatic perspective—are these tools good for academic users, librarians included? He’s a booster, in that respect, with some justified expectation that the tools will get better. While I agree with his tar-all-with-one-brush worry, the issues I’m addressing in the Matthew effect piece aren’t about usefulness—they’re about the credit system and the potential for patterned injustice. What does he make of the four studies I cited (here, here, here, and here) that find early evidence for a Matthew effect? I’ll write to ask :).