API reference

Running experiments

cognarium.load(model, *, tokenizer=None, device='auto', dtype='auto', batch_size=32, revision=None, pretrained=True, adapter=None, layer_map=None, layers=None, readout=None, chat=False)

Turns a model into a subject that can run experiments.

Parameters:

Name	Type	Description	Default
model	str | Module | Subject	A Hugging Face model id or local path, a model object, or a subject, which is returned unchanged.	required
tokenizer	Any	The tokenizer. Required for model objects.	None
device	str	Where the model runs.	'auto'
dtype	str	The model's precision.	'auto'
batch_size	int	How many sequences go through the model at once.	32
revision	str | None	The Hugging Face revision to load.	None
pretrained	bool	Whether a custom module's weights are already trained.	True
adapter	Callable[..., Tensor] | None	How to get logits out of a custom module.	None
layer_map	LayerMap | None	Where a Hugging Face model keeps its blocks, final norm and head, for a model that can't be recognised or detected.	None
layers	Mapping[str, str] | None	Which submodules of a custom module to record, as names mapped to dotted paths.	None
readout	str | None	The dotted path of a custom module's output head, which the logit lens reads states through.	None
chat	bool	Whether a Hugging Face model gets each prompt as one user message through its chat template, as instruction-tuned models expect.	False

Returns:

Type	Description
Subject	The subject.

Raises:

Type	Description
TypeError	If the model isn't a supported type, a model object comes without a tokenizer, or a setting doesn't apply to this kind of model.
ValueError	If chat is on and the tokenizer has no chat template.

cognarium.run(paradigm, subject, *, learning='auto', record=(), layers=None, pool='last', state_dtype='auto', out=None, resume=True, progress=True)

Runs an experiment, or continues it if it was stopped.

Every step is saved as soon as it finishes. Running the same call again picks up after the last saved step, and a finished run makes no further model calls.

Parameters:

Name	Type	Description	Default
paradigm	Paradigm	The experiment to run.	required
subject	Subject	Who takes part.	required
learning	str	How the subject learns. "auto" picks the subject's natural mode.	'auto'
record	Sequence[str]	What to save beyond the results table: "logits", "hidden_states" or both. With logits, a subject that has tokens also gets the columns logit_margin, logit_difference, target_rank, vocab_entropy, top_token_id and option_first_tokens.	()
layers	Sequence[str] | None	The layers to record, by name. Defaults to every layer the subject has.	None
pool	str | None	How to reduce each layer's time dimension: "last" keeps the last real token, "mean" averages over real tokens, and None keeps every token.	'last'
state_dtype	str	The precision to store states in. "auto" takes the subject's own default, which is float16 for neural subjects.	'auto'
out	str | Path | None	The run folder. Defaults to runs/<paradigm name>/<run key>/.	None
resume	bool	Whether an existing folder may be continued.	True
progress	bool	Whether to show a progress bar.	True

Returns:

Type	Description
Results	The results of the whole run.

Raises:

Type	Description
MissingAbilityError	If the subject can't take part, before any model call.
ValueError	If record names something that can't be recorded, layers, pool or state_dtype has a value that doesn't exist, or one is set when record leaves out "hidden_states".
RunConflictError	If the folder holds a different run, or resume is False and the folder already holds a run. Nothing is ever deleted.

cognarium.Results

The results of a run: one row per query, plus how they were made.

Parameters:

Name	Type	Description	Default
path	Path	The run folder.	required
frame	DataFrame	The results table.	required
manifest	dict[str, Any]	The run's settings, seeds and versions.	required
subject	Subject | None	The subject that made these results, if it's still around.	None
paradigm	Paradigm | None	The paradigm that made these results, if it's still around.	None

Attributes:

Name	Type	Description
path		The run folder.
manifest		The run's settings, seeds and versions.
subject		The subject that made these results, or None.
paradigm		The paradigm that made these results, or None.

hidden property

The run's hidden states, read lazily from their shards.

Raises:

Type	Description
MissingAbilityError	If this run didn't record hidden states.

load(path, *, subject=None, paradigm=None) classmethod

Reopens a run from its folder.

Parameters:

Name	Type	Description	Default
path	str | Path	The run folder.	required
subject	Subject | None	The subject that made these results, if you have it.	None
paradigm	Paradigm | None	The paradigm that made these results, which score needs.	None

Returns:

Type	Description
Results	The results of every finished step. The table is empty if no step has
Results	finished yet.

to_pandas()

Gives the results table.

Returns:

Type	Description
DataFrame	A copy of the table, with one row per query.

logits(step)

Loads the full-vocabulary logits of one step.

Parameters:

Name	Type	Description	Default
step	int	The step.	required

Returns:

Type	Description
Tensor	Raw logits of shape (n_queries, vocab).

Raises:

Type	Description
MissingAbilityError	If this step's logits weren't recorded.

lens(subject=None, *, layers=None, batch_size=256)

Reads each layer's states as if they had reached the output head.

This is the logit lens. It runs each recorded state through the subject's final norm and output head, on the subject's device. Options are represented by their first token, so measures of multi-token options are approximate.

Parameters:

Name	Type	Description	Default
subject	Subject | None	The subject that made this run. Only needed after Results.load.	None
layers	Sequence[str] | None	The layers to read. Defaults to every recorded layer.	None
batch_size	int	How many states go through the head at once.	256

Returns:

Type	Description
DataFrame	One row per query per layer, with the columns step, query, which is the
DataFrame	row's position in its step, item, the paradigm's grouping columns, layer,
DataFrame	top_token, top_log_prob, entropy, which is in bits over the whole
DataFrame	vocabulary, option_log_probs, which are normalised over the options,
DataFrame	logit_margin, target_rank and logit_difference.

Raises:

Type	Description
MissingAbilityError	If this run didn't record hidden states, its learners were discarded because it trained weights, no subject is available, or the subject has no output head or tokens.
ValueError	If the states weren't recorded with pool="last", or subject isn't the model this run used.

metrics(*names, by=None, **options)

Computes metrics from the results table.

Parameters:

Name	Type	Description	Default
*names	str	The metrics to compute. From the responses and answers: accuracy, balanced_accuracy, precision, recall, f1, mcc, d_prime, criterion or a_prime. From the probabilities: answer_probability, option_probability, entropy, surprisal, auroc, auprc, brier, ece, cross_entropy or perplexity. From iterated learning: fidelity, chain_entropy or kl. With recorded logits: logit_margin, logit_difference, target_rank or vocab_entropy. From naming systems, where a system is one chain's answers in a generation of iterated learning, or else one participant's answers in a group, one label per item: ib_complexity, ib_accuracy, efficiency_loss and ib_alignment, which need a cg.tasks.Mapping with numeric features, and alignment and co_naming_mse, which compare with reference. With numeric labels, as in function learning: mse and correlation against reference. These score each system, then average over the systems in a group.	()
by	Sequence[str] | None	A column or columns to group by, in place of the paradigm's own. With ["generation", "chain"] the naming measures give one value per chain.	None
**options	Any	Passed to the metrics that accept them. raw=True makes a probability metric use the distribution before temperature and top-p. positive names the option treated as the positive class. It defaults to the macro average for precision, recall and f1, and to the first option for the detection metrics. option names the option whose probability option_probability averages. n_bins configures ece, and reference and smoothing configure kl. For alignment and co_naming_mse, reference is one label per item or an items x labels table of probabilities. For mse and correlation, it is one number per item or a function of an item's features. auprc and ece see every option's probability, so auprc is the macro average over the options and ece scores the chosen option, even with two options.	{}

Returns:

Type	Description
DataFrame	A table with the grouping columns, then metric, value and
DataFrame	reference, which names what kl compared against.

Raises:

Type	Description
ValueError	If no names are given, a name is unknown, or a metric is undefined for a group, which the message names.
TypeError	If an option isn't accepted by any requested metric.
MissingAbilityError	If this run lacks the columns a metric needs.

contrast(measure, by=None, *, levels=None, against=None, where=None, ci_by='item', n_resamples=2000, level=0.95, seed=0, **options)

Compares two levels of a condition, or one level with a fixed value.

Parameters:

Name	Type	Description	Default
measure	str	surprisal, answer_surprisal, entropy, accuracy, answer_probability, answer_margin, option_probability, expected_option, or any numeric column of the results table.	required
by	str | None	The column naming the levels.	None
levels	Sequence[Any] | None	The levels to compare. level_a minus level_b is reported.	None
against	float | None	Compares with this value rather than a second level.	None
where	Mapping[str, Any] | None	Keeps only the rows whose columns equal these values.	None
ci_by	str	The column whose groups the interval resamples.	'item'
n_resamples	int	How many resamples the interval and the p value each use.	2000
level	float	The interval's coverage, such as 0.95.	0.95
seed	int	Seeds the resampling.	0
**options	Any	Passed to the measure, such as option for option_probability. raw=True uses the distribution before temperature, top-k and top-p.	{}

Returns:

Type	Description
DataFrame	One row, with the columns of cognarium.metrics.contrast_table, then
DataFrame	measure and proxy_for, which names the human behaviour a proxy measure
DataFrame	stands in for, and is None for any other measure.

Raises:

Type	Description
ValueError	If the measure is unknown, no rows match where, a value isn't finite, participants are identical and ci_by is participant, or the comparison can't be made.
MissingAbilityError	If a measure needs answers that some rows lack.

compare_to_human(human, measure, *, on='item', value='value', where=None, method='pearson', ceiling=None, **options)

Correlates the model's item means of a measure with people's.

The measure is averaged over every row of each on level, and so is the human table's value. Only levels on both sides are compared, and a warning counts the ones dropped.

Parameters:

Name	Type	Description	Default
human	DataFrame	People's data, with one or more rows per level of on.	required
measure	str	A measure that contrast takes, or a numeric column.	required
on	str	The column both tables share, such as item.	'item'
value	str	The human table's column of values.	'value'
where	Mapping[str, Any] | None	Keeps only the result rows whose columns equal these values.	None
method	str	pearson or spearman.	'pearson'
ceiling	float | None	A noise ceiling to report beside the correlation, such as cognarium.metrics.split_half_ceiling(ratings)["ceiling"].	None
**options	Any	Passed to the measure, such as option for option_probability.	{}

Returns:

Type	Description
DataFrame	One row with measure, correlation, its interval over items as ci_low
DataFrame	and ci_high, n, the number of shared items, ceiling, and proxy_for,
DataFrame	which names the human behaviour a proxy measure stands in for and is None
DataFrame	for any other measure.

Raises:

Type	Description
ValueError	If a column is missing, the human values aren't numbers, fewer than 3 items are shared, or either side is constant.

score()

Reports the effects the paradigm declares.

Returns:

Type	Description
DataFrame	One row per effect, with the primary effect first.

Raises:

Type	Description
MissingAbilityError	If this run was reopened without its paradigm, or the paradigm declares no effects.

Building paradigms

cognarium.Paradigm

An experiment, written as a list of trials or as a function of past results.

Most paradigms only need trials. Each participant then gets those trials in their own order, and context says which past trials stay in view. A paradigm that needs more control overrides step and config instead.

A paradigm keeps no running state: step depends only on its arguments. A stopped run can therefore continue from its last saved step.

Parameters:

Name	Type	Description	Default
stimuli	Any	Rows that replace the paradigm's built-in items, for a paradigm that names its item_columns: a path to a .csv, .json, .jsonl or .parquet file, a cg.data.Dataset, a table, or a list of mappings.	None
n_participants	int	How many virtual participants take part. Each one draws its own trials and order.	1
personas	Sequence[str] | None	Text placed before the instructions, with one per participant, repeating if there are fewer than participants. Only subjects that read text can take these.	None
instructions	str | None	Text placed before each prompt.	None
examples	Sequence[Any]	Demonstrations shown before the trials. Observations or (stimulus, label) pairs.	()
temperature	float	Divides log probabilities before a response is sampled.	1.0
top_p	float	Keeps the smallest set of options covering this much probability.	1.0
top_k	int | None	Keeps only this many of the most likely options.	None
max_new_tokens	int	How many tokens a free response may use.	32
seed	int	Seeds everything random in the experiment.	0
**parameters	Any	Anything else the paradigm uses, which becomes an attribute.	{}

Attributes:

Name	Type	Description
name	str	Identifies the paradigm in results. Defaults to the class name in snake case.
requires_learning	bool	Whether subjects must be able to learn.
single_label_training	bool	Whether every label a subject learns from is the same one, as when only grammatical strings are shown. A learner that scores options only by the examples it holds of each would then give every item that label.
group_by	tuple[str, ...]	The columns metrics are reported by. Defaults to the trials' condition labels.
noise	float	How likely a transmitted label is to flip.
modalities	frozenset[str]	The stimulus fields this paradigm provides. Defaults to those every trial has.
settings	Settings	The experiment settings.
context	ContextPolicy	Which past trials stay in a subject's context.
ordered	bool	Whether participants see trials in the order trials gives. Otherwise each participant's trials are shuffled within their blocks.
response_mode	str	"choice" picks an option, and "generate" answers freely.
template	PromptTemplate	How stimuli and labels are written for subjects that read text.
effects	tuple[Effect, ...]	What score reports. The first is the paradigm's primary effect.
item_columns	tuple[str, ...]	The columns each row of stimuli must fill.
stimuli_label	str | None	Describes the stimuli the paradigm makes itself, which results report unless stimuli replaces them.
rows	list[dict[str, Any]] | None	The rows given as stimuli, or None.
personas	tuple[str, ...] | None	The personas, if any.
n_participants	int	How many virtual participants take part.
max_new_tokens	int	How many tokens a free response may use.
parameters		The paradigm's other parameters.

Raises:

Type	Description
FileNotFoundError	If stimuli names a file that doesn't exist.
ValueError	If a count is below 1, the list of personas is empty, a parameter would replace one of the paradigm's methods, or stimuli can't fill item_columns.

group_by property

The trials' condition labels, which metrics are reported by.

modalities property writable

The stimulus fields every trial has.

trials(rng)

Gives one participant's trials.

Parameters:

Name	Type	Description	Default
rng	Generator	Seeded for this participant, for anything random in the trials.	required

Returns:

Type	Description
Iterable[Trial]	The trials, in any order unless the class sets ordered.

Raises:

Type	Description
NotImplementedError	If the paradigm defines neither trials nor step.

default_items(rng)

Gives the rows this paradigm uses when no stimuli are passed.

Parameters:

Name	Type	Description	Default
rng	Generator	Seeded for this participant, for anything random in the rows.	required

Returns:

Type	Description
list[dict[str, Any]]	One mapping per item, holding at least item_columns.

Raises:

Type	Description
NotImplementedError	If the paradigm has no built-in items.

item_rows(rng)

Gives the rows to build trials from.

Parameters:

Name	Type	Description	Default
rng	Generator	Seeded for this participant, for anything random in the rows.	required

Returns:

Type	Description
list[dict[str, Any]]	The rows passed as stimuli, or else the built-in ones.

sequences()

Gives each participant's trials in the order they see them.

Returns:

Type	Description
list[list[Trial]]	One list per participant, which depends only on the seed.

Raises:

Type	Description
ValueError	If a participant has no trials, a condition label would collide with a result column, or a label or item mixes types.

step(index, previous)

Builds the next batch of queries.

Parameters:

Name	Type	Description	Default
index	int	The step to build.	required
previous	DataFrame | None	The results table of the step before it, if any.	required

Returns:

Type	Description
Step | None	The step, or None when the experiment is over.

Raises:

Type	Description
ValueError	If a later step has no previous results, or a policy with feedback meets a trial with no answer.

preview(query=0, step=0)

Writes out one query as a subject that reads text sees it.

Parameters:

Name	Type	Description	Default
query	int	Which query of the step, in the order the step holds them.	0
step	int	Which step. Steps after the first only work for paradigms whose prompts never depend on earlier responses.	0

Returns:

Type	Description
str	The full prompt, then a last line listing the options it scores.

Raises:

Type	Description
ValueError	If the step doesn't exist or needs earlier responses.
IndexError	If the step has no such query.

config()

Describes the paradigm in JSON-safe form.

Returns:

Type	Description
dict[str, Any]	Everything that changes the results, which together identify a run. The
dict[str, Any]	digest of the trials identifies them even when a parameter that made them
dict[str, Any]	can't be written as JSON.

score(results)

Reports this paradigm's effects.

Parameters:

Name	Type	Description	Default
results	Results	The results of a run of this paradigm.	required

Returns:

Type	Description
DataFrame	One row per effect, with the primary effect first. See Results.contrast
DataFrame	for the columns.

Raises:

Type	Description
MissingAbilityError	If the paradigm declares no effects.

annotate(frame)

Adds columns that depend on a finished step's responses.

Parameters:

Name	Type	Description	Default
frame	DataFrame	The results table of one step.	required

Returns:

Type	Description
DataFrame	The table with any added columns.

texts()

Lists the text a subject could see, for building a vocabulary.

Returns:

Type	Description
list[str]	The personas, the instructions, each trial with each option, and the
list[str]	examples.

hypothesis_space()

Gives the hypotheses a Bayesian learner considers on this paradigm.

Returns:

Type	Description
HypothesisSpace	The hypothesis space.

Raises:

Type	Description
MissingAbilityError	If the paradigm doesn't define one.

cognarium.Effect dataclass

One comparison a paradigm reports, declared as data.

Attributes:

Name	Type	Description
name	str	Identifies the effect in the scores.
measure	str	A measure with one value per row, such as answer_surprisal, or a numeric result column.
by	str | None	The column naming the levels.
levels	tuple[Any, ...] | None	The levels to compare. level_a minus level_b is reported, so a human-like effect comes out positive.
against	float | None	Compares with this value instead of a second level, such as 0.5 for chance.
where	Mapping[str, Any]	Keeps only the rows whose columns equal these values.
options	Mapping[str, Any]	Passed to the measure.
proxy_for	str | None	The human behaviour the measure stands in for, when it isn't the one its measure usually stands in for. An answer margin, for instance, stands in for reaction time by default and for the error rate in an errors effect.

cognarium.Trial dataclass

One stimulus with the responses a subject can give to it.

Attributes:

Name	Type	Description
stimulus	Stimulus	What the subject is shown.
options	tuple[str, ...]	The possible responses. Any sequence is accepted and stored as a tuple. No options means the subject answers freely.
answer	str | None	The correct response, if the trial has one.
condition	Mapping[str, Any]	Labels for the experimental condition.
item	int | str | None	An identifier for this item across chains or participants.
block	int	Which block of trials this belongs to. A new block starts with a fresh context.

Raises:

Type	Description
ValueError	If there is only one option, repeated options, or an answer that isn't an option.

free property

Whether the subject answers freely rather than choosing an option.

cognarium.Stimulus dataclass

Something shown to a subject, in one or more modalities.

Attributes:

Name	Type	Description
text	str | None	Text for subjects that read.
image	Any	An image for subjects that see.
audio	Any	A waveform for subjects that hear.
features	Tensor | None	A numeric vector, which lets non-neural learners read the same trial a language model reads as text.

Raises:

Type	Description
ValueError	If every field is empty.

modalities property

The names of the fields this stimulus has set.

cognarium.PromptTemplate dataclass

How stimuli and labels are written out for subjects that read text.

Attributes:

Name	Type	Description
stimulus	str	A format string for the stimulus, with a {text} field.
response	str	The cue the response follows. When it is empty, the response continues the stimulus directly, as when scoring the next word of a sentence.
separator	str	What joins the parts of a prompt.
complete	bool	Whether every option is scored with the separator after it, as a whole answer. Then an option that would usually go on, such as a string that stops halfway, scores lower than one that ends where it should.

render_query(stimulus)

Writes a stimulus up to the point where a response is due.

Parameters:

Name	Type	Description	Default
stimulus	Stimulus	The stimulus to write out.	required

Returns:

Type	Description
str	The rendered text, ending with the response cue if there is one.

Raises:

Type	Description
ValueError	If the stimulus has no text.

render_observation(observation)

Writes a labelled stimulus, including its label.

Parameters:

Name	Type	Description	Default
observation	Observation	The observation to write out.	required

Returns:

Type	Description
str	The rendered text, ending with the label.

cognarium.context

What a subject remembers from one trial to the next.

ContextPolicy dataclass

Which past trials stay in a subject's context.

Attributes:

Name	Type	Description
keep	int | None	How many past trials stay in context. None keeps every one, and 0 keeps none.
feedback	bool	Whether past trials show the correct answer rather than the subject's own response.

window(k, *, feedback=False)

Keeps only the most recent trials in context.

Parameters:

Name	Type	Description	Default
k	int	How many past trials to keep.	required
feedback	bool	Whether past trials show the correct answer rather than the subject's own response.	False

Returns:

Type	Description
ContextPolicy	The policy.

Raises:

Type	Description
ValueError	If k is below 1.

Paradigms

cognarium.paradigms.IteratedLearning

Bases: Paradigm

Each generation learns from the last generation's responses.

Each chain starts from the task's first labelling for it. Each generation sees a random subset of the previous generation's labels, labels every item, and passes those labels on. A trial's answer is its chain's first label for the item, so accuracy measures agreement with where the chain started.

Maps onto iterated learning experiments, where chains of people drift toward what they find easiest to learn, such as structured languages (Kirby, Cornish and Smith, 2008) and positive linear functions (Kalish, Griffiths and Lewandowsky, 2007). For a Bayesian learner that samples from its posterior, the chains converge to its prior (Griffiths and Kalish, 2007). What it can't capture: every generation is the same model with a fresh context, so the chains show one model's in-context biases rather than a population of different people.

Parameters:

Name	Type	Description	Default
task	ShepardCategories | Mapping	The task being transmitted, such as cg.tasks.ShepardCategories or cg.tasks.Mapping.	required
generations	int	How many learners follow each other in a chain.	10
chains	int	How many independent chains to run.	50
n_prev_examples	int	How many items each generation sees labelled by the previous one, drawn without repeats. Showing every item lets a subject that reads its context simply copy it.	4
noise	float	How likely each transmitted label is to flip to another option.	0.0
memory	int	How many of its own latest answers in the same generation a chain still sees, after the transmitted examples. With 0, every item is named independently and a generation is one step. Otherwise each chain names the items one at a time, in a new random order every generation, so a generation takes one step per item. Imel and Zaslavsky (2026) used 10, which kept chains from collapsing onto a single label.	0
instructions	str | None	Text placed before each prompt.	None
examples	Sequence[Any]	Demonstrations shown every generation, before the transmitted data. Observations or (stimulus, label) pairs. They can bias a chain, and results record that they were used.	()
temperature	float	Divides log probabilities before a response is sampled.	1.0
top_p	float	Keeps the smallest set of options covering this much probability.	1.0
top_k	int | None	Keeps only this many of the most likely options.	None
seed	int	Seeds everything random in the experiment.	0

Raises:

Type	Description
ValueError	If a count or probability is out of range, memory is negative, or an example's label isn't one of the task's options.

step(index, previous)

Builds the next step: one generation, or with memory one item per chain.

Parameters:

Name	Type	Description	Default
index	int	The step to build.	required
previous	DataFrame | None	The results table of the step before it.	required

Returns:

Type	Description
Step | None	The step's queries, or None after the last generation.

Raises:

Type	Description
ValueError	If a later step has no previous results, or they miss a chain.

config()

Describes the experiment in JSON-safe form.

Returns:

Type	Description
dict[str, Any]	Everything that changes the results, with settings at the top level.

annotate(frame)

Adds the group each chain's labelling falls into.

Parameters:

Name	Type	Description	Default
frame	DataFrame	The results table of one step.	required

Returns:

Type	Description
DataFrame	The table with a group column. With memory, a chain's labelling is
DataFrame	complete only on its last item of a generation, so only that row gets a
DataFrame	group and the others get None.

texts()

Lists every line of text a subject could see.

Returns:

Type	Description
list[str]	The instructions, each stimulus with each option, and the examples.

hypothesis_space()

Gives the hypotheses a Bayesian learner considers on the task.

Returns:

Type	Description
HypothesisSpace	The task's hypothesis space.

cognarium.paradigms.CategoryLearning

Bases: _StudyThenTest

Learn a category structure, or a function, from corrective feedback.

Every block shows each item repeats times, in a new order, and asks for its label. With feedback="trial", each trial's context holds every earlier trial with its correct label, so the subject learns as it goes, as a person does who sees the correct answer after each response. With feedback="block", every trial in a block shares one context, holding the earlier blocks only, which is faster but leaves the first block at chance. Each participant is a separate learner with their own orders.

The task says what is learned: a Shepard structure, or a cg.tasks.Mapping, which holds any items and labels. The mapping's start is the labelling taught, and without one a random labelling is drawn from the run's seed. A function read on a scale, from Mapping.from_function, makes this function learning from feedback, scored with the mse and correlation metrics by block.

Maps onto supervised category learning, such as the Shepard, Hovland and Jenkins (1961) structures, where people learn type I fastest, type II next, types III to V more slowly and type VI slowest (Nosofsky et al., 1994). What it can't capture: a person's context holds their own responses and they forget, while the model reads every earlier item with its correct label and nothing else, so there is no forgetting and no effect of its own errors. The Shepard objects are the eight abstract ones of the task, not human-normed.

Parameters:

Name	Type	Description	Default
task	ShepardCategories | Mapping	What to learn. Its starting labelling is the correct answer.	required
blocks	int	How many blocks to run.	6
repeats	int	How many times each item appears in a block. Nosofsky et al. (1994) showed each of the eight objects twice per block.	1
feedback	str	"trial" to show the correct label after every trial, or "block" to show a block's labels only once it is over.	'trial'
**settings	Any	What every paradigm takes, such as n_participants.	{}

Raises:

Type	Description
ValueError	If there are fewer than two blocks, repeats is below 1, or feedback is unknown.

effects property

The probability of the correct label in the last block against the first.

trials(rng)

Tests every item repeats times per block, against its correct label.

Parameters:

Name	Type	Description	Default
rng	Generator	Unused, since each block's order is shuffled afterwards.	required

Yields:

Type	Description
Iterable[Trial]	One trial per item, repeat and block.

step(index, previous)

Builds one block's tests for every participant.

Parameters:

Name	Type	Description	Default
index	int	The position of the block to build.	required
previous	DataFrame | None	Ignored, since what is shown never depends on responses.	required

Returns:

Type	Description
Step | None	The block's queries, or None after the last block.

study(participant, block)

Gives every earlier block's items with their correct labels.

Parameters:

Name	Type	Description	Default
participant	int	The participant.	required
block	int	The block about to be tested.	required

Returns:

Type	Description
tuple[Observation, ...]	The labelled items, in the order the participant was tested on them.

config()

Describes the paradigm in JSON-safe form.

Returns:

Type	Description
dict[str, Any]	What every paradigm reports, and the task.

hypothesis_space()

Gives the hypotheses a Bayesian learner considers on the task.

Returns:

Type	Description
HypothesisSpace	The task's hypothesis space.

cognarium.paradigms.ArtificialGrammar

Bases: _StudyThenTest

Study strings from a finite-state grammar, then judge new ones.

The grammar is Reber's (1967), over the letters T, P, S, X and V, with strings of 3 to 8 letters. Each participant studies their own strings, then is tested on new grammatical strings and as many ungrammatical ones. Each ungrammatical string changes one letter of a tested grammatical string so that it leaves the grammar, and of the changes possible, the one whose chunk strength is closest to the original's is used, so that grammaticality and familiarity come apart as in Knowlton and Squire (1996). A string's chunk strength is how often, on average, its letter pairs and triples occurred in the participant's study strings. Letters are written apart, as in Letters: T X S, so a subject that reads words sees letters. A string's features count its letter pairs, including its first and last letters, and strings that would share features are never used.

There are two tests. With test="judgment", the subject studies each string labelled grammatical, then judges each test string grammatical or ungrammatical, and each test string is marked high or low in chunk strength, splitting its status in half. With test="probability", the subject studies the bare strings, then each trial compares a grammatical string with its ungrammatical change by the probability of each as a whole string, with no judgment asked. Only a language model gives these probabilities, so this test needs a subject that reads text. Results gain a chunk_strength column, and under the probability test a chunk_difference column, the grammatical string's chunk strength minus its change's.

Maps onto artificial grammar learning, where people trained on grammatical strings later judge new ones above chance without being able to say the rules, and endorse familiar chunks whatever their grammaticality (Knowlton and Squire, 1996). The probability test is the model's familiarity with each string, which a judgment can misreport (Hu and Levy, 2023). What it can't capture: people aren't told the strings follow rules, while under the judgment test every study string carries the same label, and the measures are probabilities rather than endorsement rates. The strings come from the grammar, not human-normed.

Parameters:

Name	Type	Description	Default
n_train	int	How many strings each participant studies.	16
n_test	int	How many grammatical strings are tested, and as many ungrammatical ones.	8
noise	float	How often a Bayesian learner assumes a label it sees is wrong. The labels themselves never change.	0.2
test	str	"judgment" or "probability".	'judgment'
**settings	Any	What every paradigm takes, such as n_participants.	{}

Raises:

Type	Description
ValueError	If a count is below 1, there aren't enough grammatical strings for both counts, noise is outside [0, 1), or test is unknown.

effects property

The effects this test scores.

Under the judgment test, how far grammaticality and chunk strength each raise the probability of answering grammatical. Under the probability test, the probability of the grammatical string of each pair, against chance.

trials(rng)

Tests new grammatical strings against their one-letter ungrammatical changes.

Parameters:

Name	Type	Description	Default
rng	Generator	Seeded for this participant. Its first draws choose what is studied.	required

Yields:

Type	Description
Iterable[Trial]	Under the judgment test, one trial per tested string, with its status
Iterable[Trial]	and chunk. Under the probability test, one trial per grammatical string,
Iterable[Trial]	whose options are it and its change, in random order.

study(participant, block)

Gives the participant's study strings.

Parameters:

Name	Type	Description	Default
participant	int	The participant.	required
block	int	The block about to be tested. There is only one.	required

Returns:

Type	Description
Observation	The study strings, in the order they are shown: each labelled grammatical
...	under the judgment test, and bare under the probability test.

annotate(frame)

Adds each test string's chunk strength.

Parameters:

Name	Type	Description	Default
frame	DataFrame	The results table of the test.	required

Returns:

Type	Description
DataFrame	The table with chunk_strength, and under the probability test
DataFrame	chunk_difference.

hypothesis_space()

Gives the grammar under every relabelling of its letters.

Returns:

Type	Description
HypothesisSpace	120 hypotheses, one per permutation of the five letters, over every string
HypothesisSpace	any participant sees. The unpermuted grammar is in group true and the
HypothesisSpace	others in other.

Tasks

cognarium.tasks.ShepardCategories

Sort 8 objects with 3 binary features into two categories.

Objects come in itertools.product([0, 1], repeat=3) order, over size, colour and shape, and read like "small red circle". Hypothesis i gives item j the label at bit j of i, with 0 meaning A and 1 meaning B.

Parameters:

Name	Type	Description	Default
type	str	The category structure that seeds a chain, from "I" to "VI".	'VI'

Raises:

Type	Description
ValueError	If the type is unknown.

start(chain, seed)

Gives a chain's first labelling.

Parameters:

Name	Type	Description	Default
chain	int	The chain.	required
seed	int	The run's seed.	required

Returns:

Type	Description
tuple[str, ...]	The seed structure, which every chain starts from.

group_of(labels)

Names the group a full labelling of the 8 objects belongs to.

Parameters:

Name	Type	Description	Default
labels	Sequence[str]	One label per object, in object order.	required

Returns:

Type	Description
str	The Shepard type, or "unbalanced" for unequal category sizes.

hypothesis_space()

Gives all 256 labellings of the 8 objects as hypotheses.

Returns:

Type	Description
HypothesisSpace	The hypothesis space, with the features of each object.

type_prior(weights)

Builds a prior over labellings from a weight per group.

Parameters:

Name	Type	Description	Default
weights	Mapping[str, float]	Weight per group name. Missing groups get zero, and weights are normalised to sum to 1.	required

Returns:

Type	Description
Tensor	A float64 prior of length 256, split evenly within each group.

Raises:

Type	Description
ValueError	If a name is unknown or no weight is positive.

config()

Describes the task in JSON-safe form.

Returns:

Type	Description
dict[str, Any]	The task's name and seed type.

cognarium.tasks.Mapping

Items mapped to labels from a fixed set, which a paradigm transmits or teaches.

A colour naming system is a mapping, with one feature vector per colour and made-up words as labels. So is a function read on a scale, with one x value per item and the points of the scale as labels. Items and their labels are written Features: [0.2, 0.7, 0.1] -> Label: wug. Iterated learning passes a mapping down chains of learners, and category learning teaches it from feedback.

Parameters:

Name	Type	Description	Default
features	Sequence[Any]	One entry per item: a sequence of numbers or words, or a single value, which is written without brackets.	required
labels	Sequence[str]	The fixed set of labels a subject chooses from.	required
start	Sequence[str] | None	One label per item: the first generation's labels in iterated learning, and the labels taught in category learning. None draws a random labelling from the run's seed, one per chain in iterated learning and one for the whole run in category learning.	None
template	PromptTemplate | None	How an item and its label are written.	None
points	Sequence[Sequence[float]] | None	Where each item sits for the naming measures, one sequence of numbers per item, when that differs from what the prompt shows. Colours written as RGB triples, for example, are compared in CIELAB, where distance follows perceived difference. Defaults to the features.	None
sigma	float | None	How far each item's meaning spreads over the others, for the naming measures, in the units of points. Defaults to the median distance from an item to its nearest neighbour. Only numeric features or points have one.	None
prior	Sequence[float] | None	How often each item comes up, for the naming measures. Defaults to uniform.	None
group_by	Callable[[tuple[str, ...]], str] | None	Turns one labelling, a label per item in item order, into a group name for chain_entropy and kl. By default a labelling's group is the number of labels it uses.	None
name	str	Names the task in results.	'mapping'

Raises:

Type	Description
ValueError	If there are no items, items of numbers differ in length, there are fewer than two labels or a repeated one, start doesn't give one known label per item, points doesn't give one finite vector of the same length per item, sigma isn't positive, or prior doesn't give one weight per item with a positive total and none negative.

from_function(x, function, scale, **settings) classmethod

Builds a function-learning task, as in Kalish, Griffiths and Lewandowsky (2007).

Parameters:

Name	Type	Description	Default
x	Iterable[float]	The input values, one item each.	required
function	Callable[[float], float]	Gives the first generation's answer for each input.	required
scale	Iterable[float]	The answers a subject chooses from, in order.	required
**settings	Any	Passed to Mapping, except start, which this sets.	{}

Returns:

Type	Description
Mapping	A mapping from each x to a point of the scale, starting from the point
Mapping	nearest to function(x).

start(chain, seed)

Gives a chain's first labelling.

Parameters:

Name	Type	Description	Default
chain	int	The chain.	required
seed	int	The run's seed.	required

Returns:

Type	Description
tuple[str, ...]	The given start, or a random label for each item drawn for this chain.

group_of(labels)

Names the group a full labelling belongs to.

Parameters:

Name	Type	Description	Default
labels	Sequence[str]	One label per item, in item order.	required

Returns:

Type	Description
str	What group_by gives, or how many labels the labelling uses.

hypothesis_space()

Refuses, because a mapping doesn't list its possible labellings.

Raises:

Type	Description
MissingAbilityError	Always.

meaning_space()

Gives what the naming measures need about the items.

Returns:

Type	Description
Tensor	The points, or else the features, shape (n_items, n_dimensions), in
float	float64, the meaning width, and the prior over items.

Raises:

Type	Description
MissingAbilityError	If there are no points and the features aren't numbers.

config()

Describes the task in JSON-safe form.

Returns:

Type	Description
dict[str, Any]	The items, labels, start, wording and meaning settings. A group_by rule
dict[str, Any]	appears by its name, so renaming it is the way to mark a changed rule.

Subjects

cognarium.subjects.ExemplarLearner

Bases: Subject

Categorizes by similarity to stored labelled exemplars.

Similarity is exp(-c * sum_d(w_d * |x_d - e_d|)). An option's score is gamma * log(criterion + summed similarity to its exemplars). With no criterion an option with no exemplars scores -inf, so a chain whose labels all agree stays that way unless guess is set.

A criterion lets an option with no exemplars still be chosen, when the item is unlike everything stored. This plays the part of the criterion in the GCM's old/new recognition rule (Nosofsky, 1988), and it is what a learner needs when every example it sees carries the same label, as in artificial grammar learning.

Parameters:

Name	Type	Description	Default
c	float	How quickly similarity falls with distance.	2.0
attention	Sequence[float] | None	Weight per feature, normalised to sum to 1. Uniform by default.	None
gamma	float	Sharpens or flattens responses.	1.0
guess	float	Probability of responding uniformly at random.	0.0
criterion	float	Similarity added to every option before the log. 0 leaves the model as Nosofsky (1986) wrote it.	0.0
device	str	Where to compute.	'cpu'

Raises:

Type	Description
ValueError	If a parameter is out of range.

respond(queries, *, capture_logits=False, capture_states=None)

Scores each query's options from the exemplars in its context.

Parameters:

Name	Type	Description	Default
queries	Sequence[Query]	The queries to answer. Their context holds the exemplars.	required
capture_logits	bool	Ignored, since scores already are the raw values.	False
capture_states	StateCapture | None	Where to send each query's similarity to each exemplar, as similarity. Its pool is ignored, since there is no time dimension.	None

Returns:

Type	Description
Responses	One row of scores per query.

Raises:

Type	Description
ValueError	If a stimulus has no features, or an exemplar's label isn't one of the options.

state_layers()

Names what this learner can record.

Returns:

Type	Description
tuple[str, ...]	similarity, which holds each query's similarity to each exemplar.

bind(paradigm, learning)

Checks that the paradigm's examples can serve as exemplars.

Parameters:

Name	Type	Description	Default
paradigm	Paradigm	The paradigm about to run.	required
learning	str | None	The learning mode the run will use.	required

Returns:

Type	Description
ExemplarLearner	This learner.

Raises:

Type	Description
MissingAbilityError	If the paradigm trains on a single label and there is no criterion, so every item would get that label.
ValueError	If an example has no features.

config()

Describes the model's parameters.

Returns:

Type	Description
dict[str, Any]	The class name and every parameter. The criterion appears only when it
dict[str, Any]	isn't 0, so runs from before it existed keep their keys.

runtime()

Describes where the model runs.

Returns:

Type	Description
dict[str, Any]	The device.

cognarium.subjects.BayesianLearner

Bases: Subject

Updates a prior over hypotheses from labelled observations.

The decision rule matters under iterated learning. Learners who sample a hypothesis from the posterior make the prior the chain's stationary distribution (Griffiths and Kalish, 2007), while map learners exaggerate it. sample and map commit to one hypothesis per chain, whereas average responds to each item from the posterior predictive. Scores are always the posterior predictive.

Parameters:

Name	Type	Description	Default
hypotheses	HypothesisSpace | None	The hypothesis space. By default it comes from the paradigm.	None
prior	Tensor | None	Prior probability per hypothesis. Uniform by default.	None
likelihood	Callable[[Tensor, int, float], Tensor] | None	Maps (hypothesis_labels, label_index, noise) to a log likelihood per hypothesis. By default a noisy channel, where a label flips with the paradigm's noise. The prior is only stationary when this matches how the paradigm transmits labels.	None
decision	str	How a response is chosen: "sample", "map" or "average".	'sample'
device	str	Where to compute.	'cpu'

Raises:

Type	Description
ValueError	If the decision rule is unknown.

bind(paradigm, learning)

Takes the hypotheses and noise from the paradigm.

Parameters:

Name	Type	Description	Default
paradigm	Paradigm	The paradigm about to run.	required
learning	str | None	The learning mode the run will use.	required

Returns:

Type	Description
BayesianLearner	A new learner, ready to respond.

Raises:

Type	Description
ValueError	If shaping is combined with a joint decision rule, the prior doesn't fit the hypotheses, or an example isn't one of the task's items.

log_posterior(context)

Computes the posterior over hypotheses given some observations.

Parameters:

Name	Type	Description	Default
context	Sequence[Observation]	The observations.	required

Returns:

Type	Description
Tensor	The log posterior, of length n_hypotheses.

Raises:

Type	Description
RuntimeError	If this learner hasn't been bound to a paradigm.
ValueError	If the observations rule out every hypothesis.

respond(queries, *, capture_logits=False, capture_states=None)

Scores each query with the posterior predictive.

Parameters:

Name	Type	Description	Default
queries	Sequence[Query]	The queries to answer. Their context holds the observations.	required
capture_logits	bool	Ignored, since scores already are the raw values.	False
capture_states	StateCapture | None	Where to send each query's posterior over hypotheses, as posterior. Its pool is ignored, since there is no time dimension.	None

Returns:

Type	Description
Responses	One row of scores per query, with choices set unless the decision
Responses	rule is "average".

Raises:

Type	Description
RuntimeError	If this learner hasn't been bound to a paradigm.

state_layers()

Names what this learner can record.

Returns:

Type	Description
tuple[str, ...]	posterior, which holds each query's posterior over hypotheses.

group_prior()

Gives the prior probability of each hypothesis group.

Returns:

Type	Description
dict[str, float]	Probability per group name.

config()

Describes what determines this learner's behaviour.

Returns:

Type	Description
dict[str, Any]	The decision rule, noise, likelihood name, and digests of the prior
dict[str, Any]	and hypotheses.

runtime()

Describes where the learner runs.

Returns:

Type	Description
dict[str, Any]	The device.

Recording

cognarium.HiddenStates

The hidden states of a run, read lazily from their shards.

Opening reads no shard. trials has one row per recorded query, in order, and every method that takes where returns its rows in that order.

Parameters:

Name	Type	Description	Default
path	Path	The folder that holds the states.	required
manifest	dict[str, Any]	The folder's manifest.	required
trials	DataFrame	The table of trials, with state_shard and state_offset columns.	required

Attributes:

Name	Type	Description
path		The folder that holds the states.
manifest		The folder's manifest.
trials		The table of trials.
layers		The names of the recorded layers.
pool		How the time dimension was reduced: "last", "mean" or None.
dtype		The stored precision, such as "float16".

open(path) classmethod

Opens a saved store, or the states of a run folder.

Parameters:

Name	Type	Description	Default
path	str | Path	The folder.	required

Returns:

Type	Description
HiddenStates	The states. No shard is read yet.

Raises:

Type	Description
MissingAbilityError	If the folder is a run that didn't record states.

to_tensor(layer, *, where=None)

Loads a layer's states into one tensor.

Parameters:

Name	Type	Description	Default
layer	str	The layer.	required
where	Any	A boolean mask over trials, as a Series or an array.	None

Returns:

Type	Description
Tensor	One row per selected trial, in the stored dtype.

Raises:

Type	Description
ValueError	If the layer is unknown, where isn't a mask of the right length or selects nothing, rows differ in shape, or every token was kept, in which case sequences is the method to use.

to_numpy(layer, *, where=None)

Loads a layer's states into one array.

Parameters:

Name	Type	Description	Default
layer	str	The layer.	required
where	Any	A boolean mask over trials, as a Series or an array.	None

Returns:

Type	Description
ndarray	One row per selected trial. NumPy has no bfloat16, so that becomes float32.

Raises:

Type	Description
ValueError	In the same cases as to_tensor.

sequences(layer, *, where=None)

Loads a layer's states with every token, for a store with no pooling.

Parameters:

Name	Type	Description	Default
layer	str	The layer.	required
where	Any	A boolean mask over trials, as a Series or an array.	None

Returns:

Type	Description
list[Tensor]	One tensor of shape (tokens, ...) per selected trial.

Raises:

Type	Description
ValueError	If the layer is unknown, where isn't a mask of the right length or selects nothing, or the store was pooled, in which case to_tensor is the method to use.

compare(metric='linear_cka', *, layers=None, where=None)

Compares every layer with every other layer.

Parameters:

Name	Type	Description	Default
metric	str | Callable[[Tensor, Tensor], float]	A name from metrics.GEOMETRY, or a function of two matrices.	'linear_cka'
layers	Sequence[str] | None	The layers to compare. Defaults to all of them.	None
where	Any	A boolean mask over trials, as a Series or an array.	None

Returns:

Type	Description
DataFrame	A square table labelled by layer.

Raises:

Type	Description
ValueError	In the same cases as to_tensor, or if metric is unknown.

save(path)

Copies these states to a standalone folder that can be shared.

The folder holds manifest.json, trials.parquet and states/. It appears only once it's complete, so a failed save leaves nothing behind.

Parameters:

Name	Type	Description	Default
path	str | Path	The folder to create. It may exist if it's empty.	required

Returns:

Type	Description
HiddenStates	The saved store.

Raises:

Type	Description
RunConflictError	If the folder holds anything already. Nothing is ever deleted.

push_to_hub(repo_id, *, private=True, token=None)

Uploads a store to the Hugging Face Hub as a dataset.

Only the store's own files go up, so a run's other files stay local.

Parameters:

Name	Type	Description	Default
repo_id	str	The dataset repository, which is created if needed.	required
private	bool	Whether the repository must be private.	True
token	str | None	The Hub token. By default, the one you are logged in with.	None

Raises:

Type	Description
ValueError	If private is set and the repository already exists as a public one, since uploading would publish these states.

cognarium.LayerMap dataclass

Dotted paths to the parts of a model that recording needs.

Attributes:

Name	Type	Description
layers	str	The list of blocks, such as "transformer.h".
norm	str | None	The norm applied after the last block, or None if there is none.
head	str	The module that maps states to logits.

Data

cognarium.data.convert

Datasets read from CSV, JSON and Parquet, which convert to arrays or trials.

Dataset

A table whose columns have roles in an experiment.

Parameters:

Name	Type	Description	Default
table	DataFrame	The rows.	required
text	str | None	The column holding text stimuli.	None
label	str | None	The column holding correct answers.	None
features	str | Sequence[str] | None	The numeric columns forming a feature vector. A single column may hold lists.	None
condition	Sequence[str]	The columns holding condition labels.	()
item	str | None	The column identifying each item.	None
options	Sequence[str] | None	The response options. Defaults to the sorted labels.	None
vocabularies	Mapping[str, Sequence[Any]] | None	Fixed orders for label and condition values. Values that aren't given here are sorted, so their numbering doesn't depend on row order.	None
source	str | None	Where the rows came from.	None

Attributes:

Name	Type	Description
text		The column holding text stimuli.
label		The column holding correct answers.
features		The numeric columns forming a feature vector.
condition		The columns holding condition labels.
item		The column identifying each item.
options		The response options, if they were given.
vocabularies		The order of the label values, under label, and of each condition column's values, under its name.
source		Where the rows came from.

Raises:

Type	Description
ValueError	If there are no rows, a named column doesn't exist, or a cell is blank, isn't a number or isn't in its vocabulary. The message names the row and column.

columns property

The names of the table's columns.

to_pandas()

Gives the rows as a table.

Returns:

Type	Description
DataFrame	A copy of the table.

to_numpy()

Converts the columns with roles to arrays.

Returns:

Type	Description
dict[str, ndarray]	text as objects, label and each condition column as positions in their
dict[str, ndarray]	vocabularies, features as a float matrix, and item, each if the dataset
dict[str, ndarray]	has that role.

to_tensor(*, tokenizer=None, max_length=None)

Converts the columns with roles to tensors.

Parameters:

Name	Type	Description	Default
tokenizer	Any	Turns text into right-padded input_ids and attention_mask. It needs encode(text) and pad_token_id. Without one, text is left out.	None
max_length	int | None	The most tokens a text may have.	None

Returns:

Type	Description
dict[str, Tensor]	label and each condition column as positions in their vocabularies,
dict[str, Tensor]	features in float32, and any tokens.

Raises:

Type	Description
ValueError	If there is a tokenizer but no text, or a text is over max_length.

to_parquet(path)

Saves the rows, with their roles and vocabularies in the file's metadata.

Parameters:

Name	Type	Description	Default
path	str | Path	Where to write the file.	required

from_parquet(path) classmethod

Opens a Parquet file.

Parameters:

Name	Type	Description	Default
path	str | Path	The file. One written by to_parquet keeps its roles.	required

Returns:

Type	Description
Dataset	The dataset.

to_trials(*, options=None)

Converts each row to a trial.

Parameters:

Name	Type	Description	Default
options	Sequence[str] | None	The response options. Defaults to the dataset's options, then to its sorted labels.	None

Returns:

Type	Description
list[Trial]	One trial per row.

Raises:

Type	Description
ValueError	If the dataset has neither text nor features, there are no options, or a label isn't one of the options.

from_csv(path, *, text=None, label=None, features=None, condition=(), item=None, options=None, **read_options)

Reads a CSV file.

Only empty cells count as missing, so words like "null" and "NA" stay text.

Parameters:

Name	Type	Description	Default
path	str | Path	The file, read as UTF-8 unless encoding says otherwise.	required
text	str | None	The column holding text stimuli.	None
label	str | None	The column holding correct answers.	None
features	str | Sequence[str] | None	The numeric columns forming a feature vector.	None
condition	Sequence[str]	The columns holding condition labels.	()
item	str | None	The column identifying each item.	None
options	Sequence[str] | None	The response options. Defaults to the sorted labels.	None
**read_options	Any	Passed to pandas.read_csv.	{}

Returns:

Type	Description
Dataset	The dataset.

Raises:

Type	Description
FileNotFoundError	If the file doesn't exist.
ValueError	If the file is empty or not UTF-8, or see Dataset.

from_json(path, *, records=None, text=None, label=None, features=None, condition=(), item=None, options=None)

Reads a JSON file: an array of rows, an object of columns, or one row per line.

Parameters:

Name	Type	Description	Default
path	str | Path	The file, read as UTF-8. A .jsonl or .ndjson file holds one row per line.	required
records	str | None	A dotted path to the list of rows, for a file that nests it.	None
text	str | None	The column holding text stimuli. This and the other columns may be dotted paths into each row, such as stim.sentence. The column then takes the name after the last dot, unless that's already taken.	None
label	str | None	The column holding correct answers.	None
features	str | Sequence[str] | None	The numeric columns forming a feature vector. A single column may hold lists.	None
condition	Sequence[str]	The columns holding condition labels.	()
item	str | None	The column identifying each item.	None
options	Sequence[str] | None	The response options. Defaults to the sorted labels.	None

Returns:

Type	Description
Dataset	The dataset.

Raises:

Type	Description
FileNotFoundError	If the file doesn't exist.
ValueError	If the file isn't UTF-8 or valid JSON, records doesn't lead to a list, or see Dataset.

read_rows(source, columns)

Reads stimulus rows from anything a paradigm's stimuli= accepts.

Parameters:

Name	Type	Description	Default
source	Any	A path to a .csv, .json, .jsonl or .parquet file, a Dataset, a table, or a list of mappings.	required
columns	Sequence[str]	The columns every row must fill.	required

Returns:

Type	Description
tuple[list[dict[str, Any]], str]	The rows, and a label for where they came from.

Raises:

Type	Description
FileNotFoundError	If a file doesn't exist.
ValueError	If source is of another kind, has no rows, or lacks a column or a value, which the message names.

cognarium.data.registry

The datasets cognarium can fetch, what each one's terms allow, and loading them.

Entries come in four tiers. Tier A datasets are openly licensed: they download and print their citation. Tier B datasets have restrictive terms or none stated: they show those terms and load only with accept_terms=True. Tier C datasets are gated behind a login or a licence agreement: their entry says how to get a copy, and loads it from path=. Tier D datasets aren't offered until their source and licence are confirmed.

Every file is fetched from its authors' own URL, which is pinned with a checksum and size, and cognarium never rehosts one. An entry stays disabled until its URL and checksum have been verified.

Entry dataclass

One dataset in the registry.

Attributes:

Name	Type	Description
name	str	The name load takes.
tier	str	A for openly licensed, B for restrictive or no stated terms, C for gated behind a login or agreement, and D for not offered.
title	str	The dataset's usual name.
licence	str	The licence, as its authors state it, or NO_LICENCE.
citation	str	The paper to cite when using it.
url	str | None	The authors' URL for the file, or hf://datasets/<owner>/<repo>/<file>@<commit> for a file on the Hugging Face Hub at a pinned commit.
sha256	str | None	The file's SHA-256 checksum, in hex.
size	int | None	The file's size in bytes.
loader	Callable[..., Dataset] | None	Turns the file into a Dataset. It takes the file's path and the options load passes on.
verified	str | None	The date the URL and checksum were checked. An entry without one stays disabled.
instructions	str | None	How to get a copy of a tier C dataset.
note	str | None	Why the entry is disabled, or what its terms say beyond the licence's name.
file_name	str | None	The name the download is saved under, for a URL that doesn't end in one.

status property

ready, local copy for a usable tier C entry, disabled or not offered.

available(*, include_disabled=False)

Lists the datasets load can give.

Parameters:

Name	Type	Description	Default
include_disabled	bool	Whether to list entries that are disabled, such as ones whose download hasn't been verified. Tier D entries are never listed.	False

Returns:

Type	Description
DataFrame	One row per dataset, with name, tier, title, licence, size in bytes and
DataFrame	status, which is ready, local copy or disabled.

load(name, *, accept_terms=False, path=None, cache_dir=None, **options)

Loads a dataset from the registry, downloading it the first time.

The download is checked against the entry's checksum, kept in the cache, and checked again on later calls, which make no network request. Each call prints the citation. The dataset's source names the entry and its checksum, so a run's manifest records exactly which file its stimuli came from.

Parameters:

Name	Type	Description	Default
name	str	The dataset, as available() lists it.	required
accept_terms	bool	Confirms that you have read a tier B dataset's terms and will keep to them. A tier B dataset doesn't load without it.	False
path	str | Path | None	A copy of the file you already have, which is read instead of downloading. A tier C dataset needs it.	None
cache_dir	str | Path | None	Where downloads are kept. See fetch.	None
**options	Any	Passed to the entry's loader.	{}

Returns:

Type	Description
Dataset	The dataset.

Raises:

Type	Description
ValueError	If the name is unknown, the dataset isn't offered or is disabled, its terms weren't accepted, a tier C dataset has no path, or the download doesn't match its checksum. The message says why, and what to do.
FileNotFoundError	If path doesn't exist.
ConnectionError	If the download fails and there's no copy in the cache.

Metrics

cognarium.metrics.information

Information theory metrics, working on log probabilities.

entropy(log_probs, *, dim=-1, base=2.0)

Computes the entropy of a distribution.

Parameters:

Name	Type	Description	Default
log_probs	Tensor	Log probabilities. Impossible outcomes (-inf) contribute nothing.	required
dim	int	The dimension holding the outcomes.	-1
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	The entropy, with dim removed.

kl_divergence(log_p, log_q, *, dim=-1, base=2.0)

Computes the KL divergence of p from q.

Parameters:

Name	Type	Description	Default
log_p	Tensor	Log probabilities of p.	required
log_q	Tensor	Log probabilities of q.	required
dim	int	The dimension holding the outcomes.	-1
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	The divergence, with dim removed. It is infinite when p gives
Tensor	probability to an outcome q rules out.

surprisal(log_prob, *, base=2.0)

Computes how surprising an outcome is.

Parameters:

Name	Type	Description	Default
log_prob	Tensor	The outcome's log probability.	required
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	The surprisal.

cross_entropy(log_p, log_q, *, dim=-1, base=2.0)

Computes the cross-entropy of q relative to p.

Parameters:

Name	Type	Description	Default
log_p	Tensor	Log probabilities of p.	required
log_q	Tensor	Log probabilities of q.	required
dim	int	The dimension holding the outcomes.	-1
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	The cross-entropy, with dim removed. It is infinite when p gives
Tensor	probability to an outcome q rules out.

js_divergence(log_p, log_q, *, dim=-1, base=2.0)

Computes the Jensen-Shannon divergence, which is symmetric and always finite.

Parameters:

Name	Type	Description	Default
log_p	Tensor	Log probabilities of p.	required
log_q	Tensor	Log probabilities of q.	required
dim	int	The dimension holding the outcomes.	-1
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	The divergence, with dim removed. In bits it lies between 0 and 1.

joint_entropy(log_joint, *, base=2.0)

Computes the entropy of two variables together.

Parameters:

Name	Type	Description	Default
log_joint	Tensor	Log probabilities of each pair, shape (n_x, n_y).	required
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	H(X, Y).

conditional_entropy(log_joint, *, base=2.0)

Computes how uncertain Y remains once X is known.

Parameters:

Name	Type	Description	Default
log_joint	Tensor	Log probabilities of each pair, shape (n_x, n_y).	required
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	H(Y | X).

mutual_information(log_joint, *, base=2.0)

Computes how much knowing X says about Y.

Parameters:

Name	Type	Description	Default
log_joint	Tensor	Log probabilities of each pair, shape (n_x, n_y).	required
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
Tensor	I(X; Y), which is never negative.

perplexity(log_probs)

Computes the perplexity of a sequence of observed outcomes.

Parameters:

Name	Type	Description	Default
log_probs	Tensor	The natural log probability given to each outcome that occurred.	required

Returns:

Type	Description
Tensor	The exponential of the mean negative log probability. A model that spreads
Tensor	its belief evenly over n outcomes has a perplexity of n.

ksg_mutual_information(x, y, *, k=3, base=2.0)

Estimates mutual information between continuous variables.

This is estimator 1 of Kraskov, Stoegbauer and Grassberger (2004), which is meant for continuous values such as hidden states. Its memory use grows with the square of the sample count.

Parameters:

Name	Type	Description	Default
x	Tensor	Samples of X, shape (n, ...).	required
y	Tensor	Samples of Y, shape (n, ...).	required
k	int	How many neighbours to use. A larger k has less variance and more bias.	3
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
float	The estimate. Sampling error can make it slightly negative.

Raises:

Type	Description
ValueError	If x and y differ in rows, there are no more samples than k, or a sample is repeated k or more times.

information_bottleneck(log_joint, betas, *, n_clusters=None, n_iterations=500, tolerance=1e-08, seed=0, base=2.0)

Traces the information bottleneck curve of a joint distribution.

For each beta this runs the iterative algorithm of Tishby, Pereira and Bialek (1999), which compresses X into clusters T that keep what X says about Y. The algorithm finds a local optimum, so different seeds can give different points.

Parameters:

Name	Type	Description	Default
log_joint	Tensor	Log probabilities of each pair, shape (n_x, n_y).	required
betas	Sequence[float]	How much to favour relevance over compression, one point per value.	required
n_clusters	int | None	How many clusters T may use. Defaults to n_x.	None
n_iterations	int	The most updates to run for each beta.	500
tolerance	float	Stops early when the encoder moves less than this.	1e-08
seed	int	Seeds each beta's starting encoder.	0
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
DataFrame	A table with one row per beta and the columns beta, complexity, which
DataFrame	is I(T; X), and relevance, which is I(T; Y).

cognarium.metrics.classification

Classification metrics, always computed on CPU.

Label metrics share one confusion matrix. A macro average covers the classes present in the predictions or the targets, as in scikit-learn.

accuracy(predictions, targets)

Computes the share of correct predictions.

Parameters:

Name	Type	Description	Default
predictions	Tensor	Predicted class indices, shape (n,).	required
targets	Tensor	True class indices, shape (n,).	required

Returns:

Type	Description
float	The accuracy.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length.

balanced_accuracy(predictions, targets)

Computes recall averaged over the classes that occur in the targets.

Parameters:

Name	Type	Description	Default
predictions	Tensor	Predicted class indices, shape (n,).	required
targets	Tensor	True class indices, shape (n,).	required

Returns:

Type	Description
float	The balanced accuracy.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length.

precision(predictions, targets, *, positive=None)

Computes how often a predicted class is right.

Parameters:

Name	Type	Description	Default
predictions	Tensor	Predicted class indices, shape (n,).	required
targets	Tensor	True class indices, shape (n,).	required
positive	int | None	The class to score. By default, the macro average.	None

Returns:

Type	Description
float	The precision. A class that is never predicted scores 0.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length.

recall(predictions, targets, *, positive=None)

Computes how often a true class is found.

Parameters:

Name	Type	Description	Default
predictions	Tensor	Predicted class indices, shape (n,).	required
targets	Tensor	True class indices, shape (n,).	required
positive	int | None	The class to score. By default, the macro average.	None

Returns:

Type	Description
float	The recall. A class that never occurs scores 0.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length.

f1(predictions, targets, *, positive=None)

Computes the harmonic mean of precision and recall.

Parameters:

Name	Type	Description	Default
predictions	Tensor	Predicted class indices, shape (n,).	required
targets	Tensor	True class indices, shape (n,).	required
positive	int | None	The class to score. By default, the macro average.	None

Returns:

Type	Description
float	The F1 score. A class with no true positives scores 0.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length.

mcc(predictions, targets)

Computes the Matthews correlation coefficient.

Parameters:

Name	Type	Description	Default
predictions	Tensor	Predicted class indices, shape (n,).	required
targets	Tensor	True class indices, shape (n,).	required

Returns:

Type	Description
float	The coefficient between -1 and 1. It is 0 when either side holds only one
float	class, since no correlation can be measured then.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length.

auroc(probabilities, targets)

Computes the area under the ROC curve.

Parameters:

Name	Type	Description	Default
probabilities	Tensor	The probability of class 1, shape (n,), or of each class, shape (n, k).	required
targets	Tensor	True class indices, shape (n,).	required

Returns:

Type	Description
float	The area. With more than two classes, the macro average of one-vs-rest.

Raises:

Type	Description
ValueError	If a class has no examples, since the area is undefined then, or the probabilities are out of range.

auprc(probabilities, targets)

Computes the area under the precision-recall curve, as average precision.

Parameters:

Name	Type	Description	Default
probabilities	Tensor	The probability of class 1, shape (n,), or of each class, shape (n, k).	required
targets	Tensor	True class indices, shape (n,).	required

Returns:

Type	Description
float	The area. Shape (n,) scores class 1 alone. Shape (n, k) is the macro average
float	of one-vs-rest, even with two classes, so it doesn't depend on which class is
float	called positive. The two can differ on the same data.

Raises:

Type	Description
ValueError	If a class has no examples, since the area is undefined then, or the probabilities are out of range.

brier(probabilities, targets)

Computes the mean squared error of predicted probabilities.

Parameters:

Name	Type	Description	Default
probabilities	Tensor	The probability of class 1, shape (n,), or of each class, shape (n, k).	required
targets	Tensor	True class indices, shape (n,).	required

Returns:

Type	Description
float	The mean summed squared error, halved for two classes so that it matches the
float	usual binary definition, as in scikit-learn.

Raises:

Type	Description
ValueError	If the probabilities are out of range.

ece(probabilities, targets, *, n_bins=15)

Computes the expected calibration error.

Parameters:

Name	Type	Description	Default
probabilities	Tensor	The probability of class 1, shape (n,), or of each class, shape (n, k).	required
targets	Tensor	True class indices, shape (n,).	required
n_bins	int	How many equal-width confidence bins to use.	15

Returns:

Type	Description
float	The mean gap between confidence and accuracy, weighted by bin size. Shape (n,)
float	takes the probability of class 1 as the confidence. Shape (n, k) takes the
float	probability of the chosen class, even with two classes, as is usual for this
float	metric. The two can differ on the same data.

Raises:

Type	Description
ValueError	If the probabilities are out of range.

cognarium.metrics.detection

Signal detection measures, computed from the four outcome counts.

d_prime(hits, misses, false_alarms, correct_rejections, *, correction='loglinear')

Computes sensitivity, the separation of signal and noise in z units.

Parameters:

Name	Type	Description	Default
hits	int	Signal trials answered as signal.	required
misses	int	Signal trials answered as noise.	required
false_alarms	int	Noise trials answered as signal.	required
correct_rejections	int	Noise trials answered as noise.	required
correction	str | None	"loglinear" adds 0.5 to each count and 1 to each total, which keeps rates of 0 and 1 finite. None uses the raw rates.	'loglinear'

Returns:

Type	Description
float	d'. Zero means no sensitivity.

Raises:

Type	Description
ValueError	If a count is negative, signal or noise has no trials, or a rate of 0 or 1 is used with no correction.

criterion(hits, misses, false_alarms, correct_rejections, *, correction='loglinear')

Computes response bias.

Parameters:

Name	Type	Description	Default
hits	int	Signal trials answered as signal.	required
misses	int	Signal trials answered as noise.	required
false_alarms	int	Noise trials answered as signal.	required
correct_rejections	int	Noise trials answered as noise.	required
correction	str | None	"loglinear" adds 0.5 to each count and 1 to each total, which keeps rates of 0 and 1 finite. None uses the raw rates.	'loglinear'

Returns:

Type	Description
float	c. Negative values lean towards answering signal, positive towards noise.

Raises:

Type	Description
ValueError	If a count is negative, signal or noise has no trials, or a rate of 0 or 1 is used with no correction.

a_prime(hits, misses, false_alarms, correct_rejections, *, correction='loglinear')

Computes the non-parametric sensitivity of Pollack and Norris (1964).

Parameters:

Name	Type	Description	Default
hits	int	Signal trials answered as signal.	required
misses	int	Signal trials answered as noise.	required
false_alarms	int	Noise trials answered as signal.	required
correct_rejections	int	Noise trials answered as noise.	required
correction	str | None	"loglinear" adds 0.5 to each count and 1 to each total. None uses the raw rates, which A' can handle.	'loglinear'

Returns:

Type	Description
float	A', where 0.5 means no sensitivity and 1 means perfect.

Raises:

Type	Description
ValueError	If a count is negative, or signal or noise has no trials.

cognarium.metrics.geometry

Measures of how similar two representations are, and of a single one's shape.

Every function takes matrices with one row per stimulus and flattens any trailing dimensions. The rows of two matrices must line up: row i of each comes from the same stimulus. Tensors on CUDA stay there, and MPS tensors move to CPU, since MPS has no float64.

linear_cka(x, y, *, debiased=False)

Computes linear centred kernel alignment (Kornblith et al., 2019).

CKA ignores rotation and overall scale, so it compares what two representations encode rather than how they lay it out across units.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same number of rows.	required
debiased	bool	Whether to use the unbiased estimator, which stays near 0 for unrelated matrices even when they have many columns for their rows. The plain estimator is biased upwards there.	False

Returns:

Type	Description
float	The similarity, where 1 means identical up to rotation and scale. The
float	debiased form can fall slightly below 0.

Raises:

Type	Description
ValueError	If the inputs are invalid, either has no variance, or debiased is set with fewer than 4 rows.

rbf_cka(x, y, *, sigma_fraction=1.0)

Computes centred kernel alignment with a Gaussian kernel.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same number of rows.	required
sigma_fraction	float	The kernel width, as a fraction of each matrix's median pairwise distance. Smaller values focus on local neighbourhoods.	1.0

Returns:

Type	Description
float	The similarity, where 1 means identical up to rotation and scale.

Raises:

Type	Description
ValueError	If the inputs are invalid, sigma_fraction isn't positive, or the median distance within a matrix is 0.

rsa(x, y, *, metric='correlation', method='spearman')

Computes representational similarity analysis.

This compares the two matrices' pairwise distances between stimuli, so it needs no shared units or width.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same number of rows.	required
metric	str	The distance between stimuli, "correlation" or "euclidean".	'correlation'
method	str	How to compare the distances, "spearman" or "pearson".	'spearman'

Returns:

Type	Description
float	The correlation between the two sets of distances.

Raises:

Type	Description
ValueError	If the inputs are invalid, an option is unknown, a row has no variance under correlation distance, or all of a matrix's distances are equal.

procrustes(x, y, *, scale=True)

Computes the distance left after optimally aligning two representations.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same number of rows. The narrower matrix is padded with columns of zeros.	required
scale	bool	Whether to give both matrices a unit norm first, as in scipy.spatial.procrustes. Otherwise they are only centred, so a difference in scale counts towards the distance.	True

Returns:

Type	Description
float	The sum of squared differences after the best rotation or reflection. It is
float	0 when they match, and at most 1 when scale is set.

Raises:

Type	Description
ValueError	If the inputs are invalid, scale is set and either has no variance, or scale isn't set and the values are too large to square.

mse(x, y)

Computes the mean squared error between two matrices.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same shape.	required

Returns:

Type	Description
float	The mean squared difference across all entries.

Raises:

Type	Description
ValueError	If the inputs are invalid, differ in their columns, or are too large to square.

svcca(x, y, *, variance=0.99)

Computes singular vector canonical correlation (Raghu et al., 2017).

Each matrix is first reduced to the principal components that hold variance of its variance, which removes noisy directions before CCA aligns the two.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same number of rows.	required
variance	float	The share of variance to keep, above 0 and up to 1.	0.99

Returns:

Type	Description
float	The mean canonical correlation, between 0 and 1. CCA fits as many directions
float	as it keeps, so it reads high when rows are few. It warns when the number of
float	rows alone forces a correlation of 1.

Raises:

Type	Description
ValueError	If the inputs are invalid, variance is out of range, or either matrix has no variance.

pwcca(x, y)

Computes projection weighted canonical correlation (Morcos et al., 2018).

Canonical directions are weighted by how much of a matrix's units they explain, so a direction that hardly matters counts for little. Weights come from whichever matrix has fewer independent components, or from x on a tie, so the score isn't symmetric in general.

Parameters:

Name	Type	Description	Default
x	Tensor	One representation, shape (n, ...).	required
y	Tensor	The other representation, with the same number of rows.	required

Returns:

Type	Description
float	The weighted mean canonical correlation, between 0 and 1. CCA fits as many
float	directions as there are components, so it reads high when rows are few. It
float	warns when the number of rows alone forces a correlation of 1.

Raises:

Type	Description
ValueError	If the inputs are invalid, or either matrix has no variance.

participation_ratio(x)

Computes how many dimensions share a representation's variance.

Parameters:

Name	Type	Description	Default
x	Tensor	The representation, shape (n, ...).	required

Returns:

Type	Description
float	The squared sum of the covariance eigenvalues over the sum of their squares.
float	It is 1 when one dimension holds all the variance, and the number of columns
float	when they all hold the same amount.

Raises:

Type	Description
ValueError	If the input is invalid or has no variance.

intrinsic_dimension(x)

Estimates the dimension of the manifold a representation lies on.

This is TwoNN (Facco et al., 2017), which reads the dimension from the ratio of each point's second to first nearest neighbour distance. It discards the largest 10% of ratios, as the paper does. Repeated rows are dropped, since a neighbour at distance 0 has no defined ratio.

Parameters:

Name	Type	Description	Default
x	Tensor	The representation, shape (n, ...).	required

Returns:

Type	Description
float	The estimated dimension.

Raises:

Type	Description
ValueError	If the input is invalid, it has fewer than 3 distinct rows, or neighbour distances differ only by the input's rounding, as on a regular grid.

layer_matrix(states, metric='linear_cka')

Compares every layer with every other layer.

Parameters:

Name	Type	Description	Default
states	Mapping[str, Tensor]	Each layer's states, shape (n, ...), with rows that line up.	required
metric	str | Callable[[Tensor, Tensor], float]	A name from GEOMETRY, or a function of two matrices.	'linear_cka'

Returns:

Type	Description
DataFrame	A square table labelled by layer. Row a, column b holds metric(a, b).

Raises:

Type	Description
ValueError	If metric is an unknown name.

cognarium.metrics.statistics

Effect sizes, resampling tests, contrasts and multiple comparison correction.

cohens_d(x, y)

Computes the standardised difference between two means.

Parameters:

Name	Type	Description	Default
x	Tensor	One group's values.	required
y	Tensor	The other group's values.	required

Returns:

Type	Description
float	The difference of means, x minus y, in pooled standard deviations.

Raises:

Type	Description
ValueError	If a group has fewer than two values, or neither group varies.

hedges_g(x, y)

Computes Cohen's d with the small sample correction.

Parameters:

Name	Type	Description	Default
x	Tensor	One group's values.	required
y	Tensor	The other group's values.	required

Returns:

Type	Description
float	The corrected difference of means, x minus y.

Raises:

Type	Description
ValueError	If a group has fewer than two values, or neither group varies.

bootstrap_ci(values, *, groups=None, statistic=None, n_resamples=10000, level=0.95, seed=0)

Computes a percentile bootstrap confidence interval.

Pass groups when rows aren't independent, such as several rows per chain, item or participant. Resampling whole groups then gives an honest interval, where resampling rows would give one that's too narrow.

Parameters:

Name	Type	Description	Default
values	Tensor	The values, shape (n,).	required
groups	Tensor | None	Which group each value belongs to, shape (n,). Whole groups are resampled.	None
statistic	Callable[[Tensor], Tensor] | None	Maps a 1-D tensor to a scalar tensor. Defaults to the mean.	None
n_resamples	int	How many resamples to draw.	10000
level	float	The interval's coverage, such as 0.95.	0.95
seed	int	Seeds the resampling.	0

Returns:

Type	Description
tuple[float, float]	The lower and upper bounds.

Raises:

Type	Description
ValueError	If values is empty or not finite, groups differs in length, or level isn't between 0 and 1.

permutation_test(x, y, *, paired=False, n_resamples=10000, seed=0)

Tests whether two groups differ in their means, by shuffling.

Parameters:

Name	Type	Description	Default
x	Tensor	One group's values.	required
y	Tensor	The other group's values.	required
paired	bool	Whether x[i] and y[i] belong together. Pairs then swap sides at random, instead of all values being shuffled.	False
n_resamples	int	How many shuffles to draw.	10000
seed	int	Seeds the shuffling.	0

Returns:

Type	Description
float	The two-sided p value. It counts the observed split as one of the shuffles,
float	so it is never exactly 0.

Raises:

Type	Description
ValueError	If paired is set and the lengths differ.

fdr(p_values)

Adjusts p values with the Benjamini-Hochberg procedure.

Parameters:

Name	Type	Description	Default
p_values	Tensor	The p values, in any order.	required

Returns:

Type	Description
Tensor	Adjusted p values in the input's order. Comparing them with a threshold
Tensor	controls the false discovery rate at that threshold.

Raises:

Type	Description
ValueError	If a p value isn't between 0 and 1.

contrast_table(frame, value, *, by=None, levels=None, against=None, group='item', n_resamples=2000, level=0.95, seed=0)

Compares two levels of a condition, or one level with a fixed value.

Rows are first averaged within each group, because several rows of one item aren't independent. When every group appears in both levels the comparison is paired, so differences between items don't widen the interval. Otherwise the two levels are resampled separately.

Parameters:

Name	Type	Description	Default
frame	DataFrame	The table holding the values.	required
value	str	The column to compare, which must be numeric.	required
by	str | None	The column naming the levels.	None
levels	Sequence[Any] | None	The levels to compare, as (level_a, level_b). The difference is level_a minus level_b. With exactly two levels present this defaults to them in reverse sorted order, so True comes before False. With against, give one level to test only its rows.	None
against	float | None	Compares with this value rather than a second level.	None
group	str	The column naming the groups to resample, such as item or participant.	'item'
n_resamples	int	How many resamples the interval and the p value each use.	2000
level	float	The interval's coverage, such as 0.95.	0.95
seed	int	Seeds the resampling.	0

Returns:

Type	Description
DataFrame	One row, with the columns by, level_a, level_b, mean_a, mean_b,
DataFrame	difference, ci_low, ci_high, p_value and detected, which says whether
DataFrame	the interval excludes 0. The interval is a percentile bootstrap and the p value
DataFrame	comes from a permutation test. With against, level_b is "chance" for 0.5 and
DataFrame	otherwise the value as text.

Raises:

Type	Description
ValueError	If a column is missing, there is nothing to compare with, the levels are ambiguous or have no rows, a level has fewer than two groups, or a value isn't finite.

cognarium.metrics.comparison

Comparing a model with people: item correlations, noise ceilings and effect sizes.

item_correlation(model, human, *, method='pearson', n_resamples=2000, level=0.95, seed=0)

Correlates a model's value for each item with people's.

The interval is a percentile bootstrap over items, so it reflects how much the correlation depends on which items happened to be chosen.

Parameters:

Name	Type	Description	Default
model	Sequence[float]	One value per item.	required
human	Sequence[float]	One value per item, in the same order.	required
method	str	pearson, or spearman for the correlation of ranks, with ties given their average rank.	'pearson'
n_resamples	int	How many resamples of the items the interval uses.	2000
level	float	The interval's coverage, such as 0.95.	0.95
seed	int	Seeds the resampling.	0

Returns:

Type	Description
dict[str, float]	correlation, the interval as ci_low and ci_high, and n, the number of
dict[str, float]	items.

Raises:

Type	Description
ValueError	If the method is unknown, the two differ in length, there are fewer than 3 items, a value isn't finite, either side is constant, or level isn't between 0 and 1.

split_half_ceiling(ratings, *, n_splits=1000, method='pearson', level=0.95, seed=0)

Estimates how well any model could correlate with people's item means.

Split s shuffles the participants with numpy.random.default_rng([seed, s]) and halves them, the first half taking the smaller share when the count is odd. The item means of the two halves are correlated, over items both halves rated, and the Spearman-Brown formula 2r / (1 + r) corrects each correlation to the full sample. The ceiling is the mean of the corrected values.

Parameters:

Name	Type	Description	Default
ratings	ndarray | DataFrame	One row per participant and one column per item. Missing ratings are NaN.	required
n_splits	int	How many random splits to average.	1000
method	str	pearson or spearman.	'pearson'
level	float	The share of corrected values between ci_low and ci_high.	0.95
seed	int	Seeds the splits.	0

Returns:

Type	Description
dict[str, float]	ceiling, and ci_low and ci_high, which bound the middle level of the
dict[str, float]	corrected values across splits. They show how much the ceiling depends on the
dict[str, float]	split, and aren't a confidence interval for it.

Raises:

Type	Description
ValueError	If there are fewer than two participants or 3 items, the method is unknown, level isn't between 0 and 1, or no split has two halves that vary over at least 3 shared items.

effect_comparison(model_a, model_b, human_a, human_b, *, paired=False, n_resamples=2000, level=0.95, seed=0)

Compares the size of the same effect in a model and in people.

Each effect is standardised on its own data: Cohen's d with a pooled standard deviation, or with paired, the mean difference over the standard deviation of the differences. A model's values and people's are on different scales, such as surprisal and milliseconds, and standardising puts them on one. The interval resamples the model's data and people's data separately, each within itself.

Parameters:

Name	Type	Description	Default
model_a	Sequence[float]	The model's values in the first condition.	required
model_b	Sequence[float]	The model's values in the second condition.	required
human_a	Sequence[float]	People's values in the first condition.	required
human_b	Sequence[float]	People's values in the second condition.	required
paired	bool	Whether a[i] and b[i] belong together, on each side.	False
n_resamples	int	How many resamples the interval uses.	2000
level	float	The interval's coverage, such as 0.95.	0.95
seed	int	Seeds the resampling.	0

Returns:

Type	Description
dict[str, float]	model_d, human_d, difference, which is model_d minus human_d, and its
dict[str, float]	interval as ci_low and ci_high.

Raises:

Type	Description
ValueError	If a condition has fewer than two values or a value isn't finite, paired conditions differ in length, a side has no variance, or level isn't between 0 and 1.

cognarium.metrics.naming

How efficiently a naming system compresses meanings, after Zaslavsky et al. (2018).

A naming system is an encoder q(w | m): one row per item, giving the probability of each word for it. Each item stands for a meaning, a distribution over the items that peaks on it and spreads to its neighbours. The information bottleneck (IB) trades the system's complexity, I(M; W), against its accuracy, I(W; U), the information its words keep about the items. Imel and Zaslavsky (2026) use these measures to follow naming systems through iterated in-context learning.

Frontier dataclass

The optimal naming systems along the information bottleneck curve.

Attributes:

Name	Type	Description
betas	Tensor	The trade-off values, in increasing order.
complexity	Tensor	I(M; W) of the optimal system at each beta.
accuracy	Tensor	I(W; U) of the optimal system at each beta.
encoders	tuple[Tensor, ...]	The optimal system at each beta, one row per item.

encoder_at(beta)

Gives the optimal system computed at the beta nearest to this one.

Parameters:

Name	Type	Description	Default
beta	float	The trade-off value.	required

Returns:

Type	Description
Tensor	The encoder, one row per item.

to_pandas()

Gives the curve as a table.

Returns:

Type	Description
DataFrame	One row per beta, with beta, complexity and accuracy.

meanings(points, sigma)

Turns each item into a meaning: a Gaussian blur over the items around it.

Parameters:

Name	Type	Description	Default
points	Tensor	Each item's coordinates, shape (n_items, n_features). For colour these are CIELAB coordinates.	required
sigma	float	How far the blur spreads, in the coordinates' units.	required

Returns:

Type	Description
Tensor	m(u), shape (n_items, n_items), where row m is a distribution over items u.

ib_complexity(encoder, prior, *, base=2.0)

Measures I(M; W), how much information a naming system spends on the items.

Parameters:

Name	Type	Description	Default
encoder	Tensor	q(w | m), shape (n_items, n_words), with rows that sum to 1.	required
prior	Tensor	How often each item comes up, shape (n_items,).	required
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
float	The complexity.

ib_accuracy(encoder, prior, meanings, *, base=2.0)

Measures I(W; U), how much a naming system's words say about the items meant.

Parameters:

Name	Type	Description	Default
encoder	Tensor	q(w | m), shape (n_items, n_words), with rows that sum to 1.	required
prior	Tensor	How often each item comes up, shape (n_items,).	required
meanings	Tensor	m(u), shape (n_items, n_items), from meanings.	required
base	float	The logarithm base. 2 gives bits.	2.0

Returns:

Type	Description
float	The accuracy.

ib_frontier(prior, meanings, *, betas=None, n_iterations=200, tolerance=1e-09, base=2.0)

Traces the optimal naming systems by reverse deterministic annealing.

The search starts at the largest beta from a system with one word per item, which is optimal there, and lowers beta step by step, starting each step from the last optimum. Words whose meanings become identical are merged, which leaves the objective unchanged and keeps the search fast.

Parameters:

Name	Type	Description	Default
prior	Tensor	How often each item comes up, shape (n_items,).	required
meanings	Tensor	m(u), shape (n_items, n_items), from meanings.	required
betas	Tensor | None	The trade-off values to trace. Defaults to 300 values spaced evenly in log scale from 1 to 2^12.	None
n_iterations	int	The most updates to run for each beta.	200
tolerance	float	Stops a beta's updates early when the encoder moves less than this.	1e-09
base	float	The logarithm base of the reported complexity and accuracy.	2.0

Returns:

Type	Description
Frontier	The frontier, ordered by increasing beta.

efficiency_loss(encoder, prior, meanings, frontier)

Measures how far a naming system falls from the optimal trade-off.

This is epsilon = min over beta of (F_beta[q] - F*_beta) / beta, with F_beta = I(M; W) - beta I(W; U), as in Zaslavsky et al. (2018).

Parameters:

Name	Type	Description	Default
encoder	Tensor	q(w | m), shape (n_items, n_words), with rows that sum to 1.	required
prior	Tensor	How often each item comes up, shape (n_items,).	required
meanings	Tensor	m(u), shape (n_items, n_items), from meanings.	required
frontier	Frontier	The optimal systems, from ib_frontier with the same prior and meanings.	required

Returns:

Type	Description
tuple[float, float]	The efficiency loss, in the frontier's units, and the beta that attains it.

gnid(first, second, prior)

Measures the generalised normalised information distance between two systems.

It is 1 - I(W1; W2) / max(I(W1; W1'), I(W2; W2')), where each pair of words names the same item, so label names don't matter. 0 means the two systems divide the items alike, and 1 that they share no information.

Parameters:

Name	Type	Description	Default
first	Tensor	One system's q(w | m), shape (n_items, n_words).	required
second	Tensor	The other's, shape (n_items, n_other_words).	required
prior	Tensor	How often each item comes up, shape (n_items,).	required

Returns:

Type	Description
float	The distance, from 0 to 1. Two systems that each give every item one word
float	are 0 apart.

co_naming_mse(first, second)

Compares which items two systems give the same label, whatever the labels are.

Parameters:

Name	Type	Description	Default
first	Tensor	One system's q(w | m), shape (n_items, n_words).	required
second	Tensor	The other's, shape (n_items, n_other_words).	required

Returns:

Type	Description
float	The mean squared difference, over every ordered pair of different items, in
float	the probability that the two items get the same label.