Quickstart

This page runs artificial grammar learning on GPT-2, reads the scores, and compares the model's item means with a table of human data.

Run a paradigm

import cognarium as cg

subject = cg.load("gpt2", device="auto")
results = cg.run(cg.paradigms.ArtificialGrammar(), subject)

The model studies strings from a finite-state grammar in its prompt, then judges new strings as grammatical or not. The run writes everything to runs/artificial_grammar/<key>/, where the key is a digest of the settings. Running the same call again reads the saved run and makes no model calls.

Read the scores

results.score()

Each row is an effect the paradigm declares. difference is mean_a minus mean_b, with a bootstrap interval over items, a permutation test's p_value, and detected when the interval excludes zero. measure names what was compared, and proxy_for names the human behaviour a proxy measure stands in for. Here the effect compares the probability of answering "grammatical" between grammatical and ungrammatical strings.

Every row of the experiment is a pandas table:

table = results.to_pandas()
table[["item", "status", "answer", "response"]].head()

Ask your own questions

contrast compares any measure between two levels of any column, and metrics summarises the run:

results.contrast("answer_probability", by="status", levels=("grammatical", "ungrammatical"))
results.metrics("accuracy", "entropy")

Compare with people

compare_to_human averages a measure per item and correlates it with human item means. The values below are invented, only to show the shape of the table, so replace them with your own data.

import pandas as pd

items = sorted(table["item"].unique())
human = pd.DataFrame({"item": items, "value": [0.5 + 0.05 * (position % 5) for position in range(len(items))]})
results.compare_to_human(human, "answer_probability")

The result gives the correlation with an interval over items, the number of shared items, and, when you pass ceiling=, a noise ceiling from cg.metrics.split_half_ceiling(ratings) beside it.

Next

• Change settings such as temperature, instructions or n_participants on the paradigm: see concepts.
• Swap in your own stimuli with stimuli=, or write a paradigm: see custom paradigms.
• Record hidden states and read the logit lens: see recording.