Observed frequencies and equivalent items

Frequency reports

pattern_frequencies reports observed response counts and exact matches to reference rows. It corresponds to the descriptive DAKS::pattern(dataset,n,P) workflow, with explicit weights and numeric row keys.

from knowledgespaces.datasets import load_pisa
from knowledgespaces.metrics import pattern_frequencies
from knowledgespaces.derivation import iita
from knowledgespaces.structures import KnowledgeStructure

pisa = load_pisa()
fit = iita(pisa.data)
candidate = KnowledgeStructure.from_surmise_relation(fit.relation)
report = pattern_frequencies(pisa.data, n=5, reference_patterns=candidate)
print(report.patterns, report.counts, report.probabilities)
print(report.reference_patterns, report.reference_counts)

The first five rows are the most frequent observed patterns. Their probabilities divide by all 340 responses, not the sum of the five reported frequencies. n=None returns every distinct positive-weight row; n_observed_patterns counts them before truncation. Counts sort descending, with ties in ascending numeric row order. Zero-weight rows do not count as observed patterns. Fractional weights are supported and remain analysis weights rather than being rounded into people.

reference_patterns can be a numeric matrix or a KnowledgeStructure. A matrix uses the data’s column order, preserving every requested row and duplicate. A structure is aligned by item label, in canonical state order. Unobserved reference rows receive zero. These are observed exact-match frequencies, not fitted latent-state masses. With response errors, observing a state’s binary pattern does not establish that the person has that state. To estimate latent masses, fit an appropriate probabilistic model instead.

Raw matrices also support finite numeric polytomous scores:

import numpy as np

scores = np.array([[1, 11], [11, 1], [1, 11]])
report = pattern_frequencies(scores, items=["q1", "q2"], n=None)
assert report.counts.tolist() == [2, 1]

Unlike DAKS 2.1-3’s string concatenation, this keeps (1,11) and (11,1) separate. Their concatenated strings both read 111; an R counterexample is retained in the fixtures. Integral score arrays retain their integer representation, including values above 2**53. Numeric ties use numeric ordering rather than locale-specific string ordering. Polytomous frequency reporting does not make BLIM, SLM or IITA polytomous models.

With raw matrices, counts supplies weights and items supplies labels (default item_1, item_2, …). A ResponseMatrix already supplies both; conflicting arguments are rejected. Missing/infinite scores are rejected without imputation. Output arrays are independent, read-only copies. max_memory_bytes preflights estimated sorting and lookup storage; it does not guarantee the exact peak allocation of NumPy’s backend.

Empirical structure at a frequency threshold

from knowledgespaces.derivation import empirical_structure
from knowledgespaces.estimation import ResponseMatrix

data = ResponseMatrix(["a", "b"], np.array([[0, 0], [1, 0], [1, 0], [1, 1]]))
structure = empirical_structure(data, threshold=2)
assert structure.states == {frozenset(), frozenset({"a"}), frozenset({"a", "b"})}

The selector matches the question of kstMatrix::kmgenerate: retain observed patterns whose frequency is at least the cutoff, then include empty/full states. threshold=None or zero sets the cutoff to ceil(N / 2**Q); N is the total weight, not the number of compressed rows. Positive fractional thresholds and weights are supported. The implementation counts only observed patterns and does not allocate all 2**Q possibilities. It does not correct response errors or guarantee union closure, learning-space axioms or a consistent latent estimator. Raw input must be binary, even in zero-weight rows.

Reducing a quasi-order

A surmise relation can contain mutually prerequisite items. They form classes under \(R\cap R^{-1}\), taken after transitive closure. The induced relation on these classes is a partial order: Falmagne and Doignon (2011), Learning Spaces, §1.6.6. Representatives are identifiers for whole classes; no information about their members is discarded.

from knowledgespaces.structures import SurmiseRelation
from knowledgespaces.viz import plot_relation

relation = SurmiseRelation(
    list("abcd"), [("a", "b"), ("b", "a"), ("b", "c")],
)
quotient, classes = relation.quotient()
assert classes == {"a": frozenset("ab"), "c": frozenset("c"), "d": frozenset("d")}
figure = plot_relation(relation, collapse_equivalent=True)

item_equivalence_classes() returns all classes, including singletons. quotient() returns a transitively closed partial order and a dictionary from each lexicographically first class member to its full class. Both methods work on supplied generating pairs by taking their transitive closure. They use the relation directly, without enumerating knowledge states. Expanding each downset of the quotient by replacing representatives with whole classes recovers a unique original downset.

plot_relation(..., collapse_equivalent=True) draws that quotient, with all class members shown in each node. Without the option, the existing partial-order graph behavior is retained and cycles raise a helpful error. The returned object is a Matplotlib figure; the class mapping remains available independently through quotient(). Plot options and supplied axes work in either mode. Prerequisite arrows point toward the dependent class. This is an item-class graph, distinct from plot_hasse(structure), whose nodes are knowledge states. The latter uses covers of state inclusion, which can span more than one item when items are equivalent.

DAKS’s numerical reduction was compared on all 29 quasiorders on three items and a five-item example with two nontrivial classes. The comparison captured the relation that the installed DAKS::hasse passes to plot; Rgraphviz rendering was not executed. Independent path/state oracles and tests of the actual Python graph’s nodes and arrows provide separate checks. Visual positions or pixel equality with R are not claimed.

Run python cookbook/13_frequencies_and_quotients.py for a complete example.