Since 1987

History

TextGroup began in the fall of 1987 with six people — two faculty and four graduate students — meeting weekly to talk about reading and language. These are some of the moments since.

2024–2026

The Need in LLMs for a Cognitive Benchmark

In Spring 2024, TextGroup linguist Andreas Schramm presented on how deeply time is embedded in language and on his planned study with Mike Mensink and Hannah Riddle combining Jeannette Gundel’s Givenness Hierarchy with his research on linguistic aspect. At that meeting, UMN Computer Science professor Dongyeop Kang proposed a collaborative benchmark effort — which would become known as CogBench — involving his doctoral candidate Karin de Langis and several other members from TextGroup and the Computer Science Department.

The vision behind CogBench (Cognitive Benchmark) was to move beyond standard NLP evaluations and instead measure LLMs against established psycholinguistic and cognitive tasks drawn from decades of human research. Rather than treating LLMs as black boxes to be probed only with language benchmarks, CogBench would treat them as cognitive systems whose processing could be compared directly with human readers — a framing that opened the door to a genuinely interdisciplinary collaboration between cognitive scientists, linguists, and NLP researchers.

The ensuing initial effort was highly productive and resulted in the paper How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs. It consisted of comparing Andreas Schramm’s cognitive psycholinguistic data from human readers with output from seven LLMs. It was a deeply gratifying experience during which both sides learned a lot about each other’s areas of expertise. We communicated multiple times over the summer of 2024 between the US and Europe to get LLMs to understand and follow human instructions — at times enlisting the help of the 180-mph German bullet train ICE to propel our research forward. An extended version with four human populations (undergraduate and graduate native-English students; low- and high-advanced nonnative English students) was well-received at the conference of the American Association of Applied Linguists in the spring of 2026.

The second part of the CogBench project was conducted with the help of additional TextGroup members Püren Öncel, Andrew Elfenbein, and Laura Allen. Two papers came out of this effort. One tested LLM comprehension of the set of stories from Ed O’Brien’s lab — including the infamous story with Mary, the cheeseburger-eating vegetarian (Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?). The other investigated the performance of LLMs on a set of classic executive-functioning tasks (Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs).

The Mary study revealed that even state-of-the-art LLMs often fail to notice the kind of glaring narrative inconsistencies that human readers catch effortlessly, suggesting that these models lean on surface-level coherence more than on genuine updating of a world model. The executive-functioning study, in turn, surfaced a striking asymmetry: LLMs exhibit remarkably strong working memory but comparatively weak cognitive control, struggling to suppress prepotent responses or flexibly shift between rules on classic tasks such as Stroop and Wisconsin Card Sorting. Together, the three CogBench papers point toward a consistent picture of today’s LLMs as systems with rich representations but limited metacognitive regulation — and toward cognitive science as a productive source of benchmarks for the next generation of models.

We look forward to future collaborations between TextGroup and the Computer Science Department!

SPRING 2018

AndyFest

In early 2018, Andrew Elfenbein published The Gist of Reading, a book dedicated to TextGroup. To celebrate, the group secretly organized a one-day event in his honor — disguised as “SidFest 2.0: Electric Boogaloo.”

Acknowledgments from The Gist of Reading, dedicating the book to TextGroup members

The ruse worked — Andy never suspected a thing. He was likely thrown off the scent by the meme-based advertising campaign:

Meme collage advertising SidFest 2.0 to disguise AndyFest

Officially the “2nd Annual Minnesota Meeting on Text and Discourse Comprehension,” the schedule featured talks connecting to Andy’s interests — from rhymes as memory cues in poetry to text comprehension from first principles.

Program highlights:
Brooke Lea, Andrew Elfenbein, David Rapp & Chelsea Voskuilen · Rhymes as memory cues in poetry
Andreas Schramm, Verena Haser, Michael Mensink & Jonas Reifenrath · Cognitive processing of time in language
Catherine Bohn-Gettler · Process vs. product instructions for science texts
Mija Van Der Wege · Marking irony in text-based communications
Panayiota Kendeou · The assessment of reading comprehension
Sashank Varma · Text comprehension from first principles
Andrew Elfenbein · Poetry as speech act

During the Welcome, the group revealed to Andy the real reason for the meeting:

Andy Elfenbein reacting to the AndyFest surprise reveal

Paul van den Broek joined via Skype and told the story of founding TextGroup with Randy Fletcher — and how Andy had found Paul’s class and then TextGroup as his own research turned in that direction.

FALL 2014

SidFest

In Fall 2014, Sid Horton decamped from Chicago (Northwestern) to spend his sabbatical in Minneapolis. His presence was felt in many ways, and at the end of the semester, TextGroup decided to hold a one-day event to honor his time among them.

Officially advertised as the “1st Annual Minnesota Meeting on Text and Discourse Comprehension,” it was really “SidFest.” Faculty gave 30-minute talks — long enough to develop an idea in depth. The venue was the large and beautiful Room 325 in the Education Sciences Building; breakfast was pastries and coffee; and lunch was a group walk to Dinkytown for Greek food.

Program highlights:
Brooke Lea · Narrative time shifts and reactivation
Panayiota Kendeou · Knowledge revision and source credibility
Andreas Schramm · Lexical aspect in native and non-native speakers
Joseph Magliano · Noticing differences in adapted narratives
Kate Bohn-Gettler · Standards of coherence vs. standards of relevance
Andrew Elfenbein · After reading
Mija Van Der Wege · Closeness-communication bias and group membership
Sid Horton · Metaphor and character intimacy in narrative
Randy Fletcher · Levels of representation and metacomprehension
“Joe Magliano, visiting from Northern Illinois University, could…not…believe…the intellectual excitement and camaraderie of TextGroup! My lasting memory will be of him smiling and shaking his head that such an event could happen.”
SPRING 1987

The Founding of TextGroup

The idea of TextGroup originated when Randy Fletcher visited Paul van den Broek at the University of Kentucky in Spring 1987. Van den Broek was on the faculty in the UK Department of Psychology but was moving to the Department of Educational Psychology at the University of Minnesota in Fall 1987. Randy visited to work on an internal Small Grant to conduct joint research in collaboration with Jay Samuels. During the visit, they decided to start a joint research group with their graduate students and others interested in issues broadly related to reading and language processing — with weekly meetings to discuss articles, ongoing research, and so on.

That started in the fall of 1987 — and 39 years later, it is still running! At the start there were only six: Randy, Paul, and four graduate students. Besides the academic “stuff,” the group held annual potluck dinners/BBQs and even traditional Dutch Saint Nicholas celebrations at year’s end.

“The idea of TextGroup originated when Randy Fletcher visited me at the University of Kentucky. I believe it was Spring 1987… That started in the fall of 1987 — and 39 years later, it is still running!”
— Paul van den Broek
SPRING 2013

The Replication Crisis of ’13

In Spring 2013, TextGroup took a holiday from Text and Discourse to see the revolution firsthand. We spent the semester immersed in the Replication Crisis — not as voyeurs but as journalists intent on discovering if, under all of the bluster and righteousness of the revolutionaries, there was something important there.

On Christmas Eve 2012, Sashank Varma emailed the group proposing a radical new model: instead of reading papers about text comprehension, they would spend a semester exploring the crisis of confidence in psychological science. The proposed syllabus spanned 15 topics — from false positives and the decline effect to outright fraud and proposed solutions.

“I think it would be a lot of fun to read some of these articles next semester. We could meet every week or two over lunch/coffee or over dinner/drinks and chat about the flawed science of our peers. (Those charlatans!)”
— Sashank Varma, December 23, 2012

It was a grand experiment. The group met weekly in a spare room in Elliott Hall. The question of “What constitutes good science?” attracted attention well beyond the usual circle. Chad Marsolek brought strong ideas every week. Brooke Lea drew on his experience as a statistics textbook author. Katrina Schleisman, a philosophically inclined PhD student, was an active participant. People from other departments who had never before attended TextGroup came to join the conversation.

A highlight was when Mija Van Der Wege invited her colleague Julie Neiworth to attend. Julie did comparative work with monkeys and was part of a network of researchers tasked with replicating Marc Hauser’s experiments to see which held up and which were fabricated. For one glorious day, monkeys reigned in TextGroup!