Comparing Structures

The knowledgespaces.metrics module provides distance measures and agreement indices for comparing two knowledge structures or surmise relations.

For empirical distances, DI/DA, gamma and VC against observed responses, see Empirical validation and BLIM diagnostics.

Distance between structures

from knowledgespaces.metrics import symmetric_difference, hausdorff, directional_distances

# Symmetric difference: states in one but not the other
d = symmetric_difference(ks_human, ks_ai)

# Hausdorff distance: max min-Hamming-distance between states
h = hausdorff(ks_human, ks_ai)

# Directional distances (asymmetric)
dd = directional_distances(ks_human, ks_ai)
print(dd.forward_mean)   # Human→AI: how far are Human states from AI?
print(dd.backward_mean)  # AI→Human: how far are AI states from Human?

Interpretation of directional distances:

forward (H→A)	backward (A→H)	Meaning
Low	Low	Similar structures
Low	High	AI adds states beyond Human
High	Low	Human has exclusive states
High	High	Divergent structures

Agreement on prerequisites

from knowledgespaces.metrics import cohens_kappa, graph_edit_distance

# Cohen's kappa on prerequisite relations
k = cohens_kappa(rel_human, rel_ai)  # [-1, 1], 1 = perfect agreement

# Graph edit distance (edge differences)
added, removed, total = graph_edit_distance(rel_human, rel_ai)

Domain validation

All metric functions require matching domains:

# This raises ValueError:
symmetric_difference(ks_with_domain_abc, ks_with_domain_xyz)

Cohen’s kappa is undefined when both relations classify all off-diagonal pairs in one category: expected chance agreement is 1, giving 0/0. cohens_kappa returns nan in this case (including two empty relations), as it does for domains with fewer than two items. Perfect raw agreement should not be reported as an estimable kappa in that degenerate case.