Research Study
Taxonomy cluster study
If we let embeddings and HDBSCAN choose how many failure classes there are across the 121 categories ingested from OWASP, MAST, DAPLab, Microsoft AIRT, Swiss Cheese, and Shah 2026, do we converge on SAICA's manual 11, or something different? Two algorithmic methods were tried.Text clustering over MiniLM sentence embeddings of thelabel + description prose landed onKtext = 26 — finer than SAICA and weakly agreeing with it (ARI 0.08, NMI 0.38).Response clustering over the (category × tool) bipartite reachability matrix landed onKbipartite = 7, coarser than SAICA's 11 and substantially more aligned with it (ARI 0.52, NMI 0.74).
UMAP — categories grouped into cluster regions
Each shaded polygon is one cluster's convex hull, colored by partition and labeled with the cluster's proposed name. Dots are the 121 source categories at their UMAP-2D coordinates. Light-gray dots are HDBSCAN noise (cluster = -1) — they get no hull. Switch the coloring to see howthe same 121 points get carved up differently by text clustering (many thin hulls), bipartite clustering (few broad hulls), and SAICA's manual partition (11 modes). Click any dot for its full description and fellow cluster members.
Alluvial — how categories flow between partitions
Three stacked columns, each a vertical inventory of clusters at a different K: text (26 thin slivers) → bipartite (7 tall blocks) → SAICA modes (11). Ribbon thickness = number of categories that belong to both the left and right bar; ribbons inherit their source bar's color. Read it left-to-right: many text slivers fan into a handful of bipartite blocks (concentration), and those blocks map tightly onto SAICA's modes (alignment). Hover any bar to light up the ribbons that touch it; hover a ribbon to see its size.
Key takeaways
- Text clustering fragments: 26 clusters, most with only 2–4 members; a huge noise bucket (~48 categories) absorbs whatever the prose-similarity signal could not commit to.
- Bipartite clustering concentrates: just 7 clusters, each tall, carving the corpus into tool-response families.
- Fan-in, left → middle: every bipartite bar is fed by many text slivers — text K=26 is too fine for the supervision structure.
- Tight mapping, middle → right: each bipartite bar lines up with 1–2 SAICA modes (ARI 0.52, NMI 0.74), confirming that "what can stop it" recovers SAICA's manual partition.
- Text → SAICA is scattered: the prose alone is not enough; supervision-tool reachability is the signal that matches the expert taxonomy.
Clusters for the current coloring
Rows correspond to clusters of the currently-selected partition. Noise points (HDBSCAN id = -1) are listed last.
| Cluster | Size | SAICA top match | Coverage | Top 3 members |
|---|
Further reading
- Full narrative report:
research/TAXONOMY_CLUSTER_STUDY.md - Raw artifacts:
research/taxonomy_clustering/ - Machine-readable input:
/taxonomy_study.json