Cognitive science experiments on language models¶
Learning experiments from cognitive science, run in context on any Hugging Face language model, and with the same trials on LSTMs, exemplar and Bayesian learners.
uv add cognarium
Paradigms¶
Each section says what the paradigm does, how it maps onto the human task, the work it builds on, and what the model version can't capture. The paradigms run with the same trials and scoring on four kinds of learner.
Learning and transmission¶
- Iterated learningEach generation learns from the last generation's answers, for any category system or function. After Kirby, Cornish and Smith (2008) and Griffiths and Kalish (2007).
- Category learningLearn the Shepard structures, any category system or a function from corrective feedback. After Shepard, Hovland and Jenkins (1961) and Nosofsky et al. (1994).
- Artificial grammarStudy strings from a finite-state grammar, then judge new ones, or compare them by probability. After Reber (1967) and Knowlton and Squire (1996).
Learners¶
- Hugging Face modelsAny causal language model, learning in context from its prompt, with logits, hidden states and the logit lens recorded.
- PyTorch modulesThe bundled LSTM or your own PyTorch module, learning in context or by training its weights.
- Exemplar learnerLabels by similarity to the examples it has stored, as in the Generalized Context Model (Nosofsky, 1986).
- Bayesian learnerInfers a labelling from a space of hypotheses and samples, averages or takes the most probable (Griffiths and Kalish, 2007).
Run one¶
These four lines run artificial grammar learning on GPT-2. The run builds the trials,
queries the model, scores the effects and saves everything to
runs/artificial_grammar/, so running it again reads the saved run instead.
Read the quickstart for what the score table holds.
import cognarium as cg
subject = cg.load("gpt2", device="auto")
results = cg.run(cg.paradigms.ArtificialGrammar(), subject)
results.score()