1Introduction
Conversational AI systems generate millions of chat sessions daily. Understanding what users actually do in these conversations — debugging code, drafting emails, role-play, medical questions — matters for product decisions, safety monitoring, and research on usage patterns, as demonstrated at scale by Anthropic's Clio system (Tamkin et al., 2024) and the analyses accompanying LMSYS-Chat-1M (Zheng et al., 2024) and WildChat (Zhao et al., 2024). The standard recipe is unsupervised: embed each conversation, optionally reduce dimensionality, cluster, and name the clusters. Each stage offers many choices, and published systems differ at every one of them, yet controlled comparisons over the joint pipeline on session-level chat transcripts are scarce. Benchmarks such as MTEB (Muennighoff et al., 2023) evaluate embeddings for clustering, but on short single texts with fixed downstream algorithms and internal metrics computed in the clustering space.
This page asks a practical question: which combination of session encoding, embedding, reduction, clustering, and evaluation should a practitioner use to discover topics in AI chat transcripts — and how should they know when it has failed? We answer it with a five-phase study whose design choices — reporting internal metrics in the original embedding space, weighting judge scores by cluster size, and stress-testing on real corpora rather than only benchmarks — each turn out to change the conclusions materially.
Our contributions:
- A controlled 288-run grid over 4 datasets × 2 encodings × 4 embeddings × 3 reductions × 3 clusterers, with external validation on labeled dialogue corpora.
- An LLM-as-judge protocol (label-blind coherence + intruder test) applied to 35 labeled runs, including all three major public AI-chat corpora at up to 10K sessions.
- Identification and quantification of degenerate collapse as the dominant real-world failure mode, its invisibility to unweighted metrics, its sample-level instability, and a cheap guard plus a re-encoding fallback that empirically eliminates it.
- A stability analysis separating reproducible cluster cores from unstable noise boundaries.
- An open framework, all 335 run records, per-cluster LLM labels, and the interactive explorer embedded below.
2Related Work
Topic modeling and text clustering. Classical topic models (LDA) have largely been superseded for short informal text by embedding-based pipelines. BERTopic (Grootendorst, 2022) popularized the UMAP→HDBSCAN→c-TF-IDF recipe that our study treats as a primary candidate. Sentence embeddings for clustering trace to Sentence-BERT (Reimers & Gurevych, 2019); the MTEB benchmark (Muennighoff et al., 2023) ranks embedding models on clustering tasks via V-measure. Our study differs in clustering full multi-turn sessions and in crossing embeddings with downstream algorithm choices rather than fixing them.
Analyzing AI chat corpora. Zheng et al. (2024) cluster LMSYS-Chat-1M prompts; Zhao et al. (2024) analyze WildChat topics; Clio (Tamkin et al., 2024) summarizes each conversation with an LLM before embedding and clustering — the summarize-then-embed strategy we benchmark in §5.5 — and names clusters with an LLM, as we do.
Cluster evaluation. Internal indices include silhouette (Rousseeuw, 1987), Calinski–Harabasz, Davies–Bouldin, and the density-based DBCV (Moulavi et al., 2014); external agreement uses ARI (Hubert & Arabie, 1985) and NMI. Our intruder test adapts the word-intrusion protocol of Chang et al. (2009) to whole conversations with an LLM judge, providing an objective, label-free measure of cluster distinctness. We mitigate LLM-judge self-preference by generating labels and judging with different models.
3Methodology
3.1 Datasets
| Dataset | Sessions | Type | Labels |
|---|---|---|---|
| MultiWOZ 2.2 | 8.4K | task dialogue | service domains |
| Banking77 | 10K | intents (1-turn) | 77 intents |
| OASST1 | 10K trees | open assistant | — |
| HH-RLHF | 170K | open assistant | — |
| LMSYS-Chat-1M | 1M | real AI chat | — |
| WildChat-1M | 1M | real AI chat | — |
| Chatbot Arena | 33K | real AI chat | — |
Table 1. Datasets. Labeled corpora anchor external metrics; the three gated real-AI-chat corpora are the deployment target. Runs sample 2,000 English sessions (10,000 for scale checks) with fixed seeds. Each conversation is flattened to a single USER:/ASSISTANT: transcript.
3.2 Session encodings
truncate: first 4,000 characters of the transcript. user_turns: user messages only. summarize: a 1–3 sentence intent summary generated by gemini-2.5-flash (concurrent, disk-cached, with refusal fallback), after Clio.
3.3 Embeddings
Two open-weight SentenceTransformers — all-MiniLM-L6-v2 (384d) and all-mpnet-base-v2 (768d) — and two commercial Google models — gemini-embedding-001 and the multimodal gemini-embedding-2 (both 3072d; the latter requires one Content object per input, as a bare string batch is silently pooled into a single embedding). All vectors are L2-normalized; API calls run concurrently with exponential-backoff retry and are disk-cached.
3.4 Dimensionality reduction
None, PCA-50, or UMAP-25 (n_neighbors 15, min_dist 0, cosine) before clustering; t-SNE and UMAP are used for 2-D visualization only. Internal metrics computed in a reduced space are inflated (silhouette 0.78 in UMAP space vs. 0.07 in the original space for the same partition), so all reported internal metrics are computed in the original embedding space.
3.5 Clustering
KMeans (k selected by silhouette sweep over 8–60 on a subsample), HDBSCAN (min_cluster_size = max(15, n/200), EOM selection; noise label −1), and agglomerative clustering (cosine, average linkage).
3.6 LLM labeling and judging
For each cluster, the 8 members nearest the centroid plus 4 random members are shown to gemini-2.5-flash, which returns a 2–6 word topic, a summary, and flagged outliers (structured outputs). A separate, stronger judge (gemini-3.5-flash) then (a) rates a label-blind random sample's coherence 1–5 and (b) performs 20 intruder trials per run: shown five members of one cluster and one secretly injected member of another, it must identify the intruder (chance = 1/6).
3.7 Metrics and aggregation
Internal: silhouette (cosine), Calinski–Harabasz, Davies–Bouldin, DBCV — original space, noise excluded, noise fraction reported separately. External: ARI, NMI, V-measure. LLM: mean judge coherence — reported both per-cluster and size-weighted — and intruder accuracy. Pipelines are compared by Borda rank aggregation (per-dataset ranks per metric, external metrics double weight); §5.4 shows why size-weighting is not optional. Degeneracy is flagged when the largest cluster holds >50% of assigned sessions.
4Experiments
Phase 1–2 (benchmark grid, 288 runs): {OASST1, HH-RLHF, MultiWOZ, Banking77} × {truncate, user_turns} × 4 embeddings × {none, PCA-50, UMAP-25} × {KMeans, HDBSCAN, agglomerative}, 2,000 sessions per run. Phase 3 (LLM-judged, 16 runs): the top pipelines with full labeling, judging, and intruder testing. Phase 4 (real corpora, 9 + 2 scale runs): the three finalists on LMSYS-Chat-1M, WildChat, and Chatbot Arena at 2K, and the recommended pipeline at 10K on LMSYS and Arena. Phase 5 (encoding ablation, 8 runs): summarize-then-embed vs. truncate. Stability (12 runs): 3 clustering seeds on fixed data and 3 overlapping sample draws from 4K pools, compared by pairwise ARI on shared sessions. Total commercial API cost was under $25.
5Results
5.1 Embedding quality dominates
Gemini embeddings dominate open-weight models on both labeled datasets: best ARI 0.80/0.81 (gemini-2) and 0.78/0.79 (gemini-001) vs. 0.53/0.61 for MPNet on Banking77/MultiWOZ; a 3-seed replication (§5.6) confirms the family gap but finds the two Gemini models statistically indistinguishable from each other. On Banking77, gemini-2 pipelines recover ≈60 clusters against 77 true intents at NMI up to 0.95. With strong embeddings the reduction step barely matters for accuracy (Banking77 ARI: raw 0.795, PCA 0.801, UMAP+HDBSCAN 0.790) but matters enormously for cost: clustering raw 3072-d vectors made runs up to 42× slower, so reduction is justified on efficiency grounds alone. Encoding interacts with embedding strength: user_turns helps weak local models, while full transcripts win with long-context Gemini models.
5.2 Internal metrics weakly track external truth
No internal index is a safe proxy for label agreement — DBCV even disagrees with ARI about ranking clusterers. We therefore treat internal metrics as tie-breakers and rely on external and LLM-based measures.
5.3 Degenerate collapse on real chat corpora
On curated benchmarks every finalist looked healthy. On real corpora, UMAP+HDBSCAN collapsed into a single cluster holding 65–98% of sessions for at least one corpus under each embedding. Intruder accuracy stays high during collapse — the small surviving clusters are genuinely distinct — so distinctness metrics cannot detect it. KMeans never degenerates (max share 5–17%); its failure mode is graceful mediocrity rather than catastrophe. Judge coherence on real corpora tops out at 2.5–3.3 vs. ≈5 on task-oriented benchmarks: results reported only on curated data substantially overstate real-world quality.
5.4 Aggregation can hide degeneracy
The unreduced agglomerative pipeline won both the Borda ranking and the per-cluster judge average (4.21/5) while assigning 99% of OASST1 and HH-RLHF sessions to a single cluster: many tiny perfect clusters outvoted one enormous incoherent one. Size-weighting drops it from first to last (3.01) and crowns gemini-001 + UMAP + HDBSCAN (3.95, intruder 1.00). Cluster-level scores — including LLM-judge scores — must be weighted by cluster mass before any ranking is trusted.
5.5 Scale, encoding ablation, and stability
Scale stabilizes. At 10K sessions the recommended pipeline produced healthy, face-valid structure on both LMSYS (41 clusters) and Arena (38), with better judge coherence than at 2K and ≈2 minutes of clustering compute. Several large LMSYS clusters could not be labeled because the labeling model refuses adult/role-play content — production systems need an explicit refusal path.
Summarize-then-embed (Table 2) eliminated degeneracy entirely for gemini-001 (max share ≤15% on all corpora, including WildChat where truncate collapsed) at equal coherence, but backfired for gemini-2 (3 of 4 corpora degenerate) and costs external accuracy on fine-grained labels. Collapse is an embedding-geometry × HDBSCAN interaction, not a transcript-length artifact; changing the encoding relocates rather than removes it.
| LMSYS | Arena | WildChat | MultiWOZ ARI | |
|---|---|---|---|---|
| g001 truncate | 2.51 / .16 | 2.48 / .15 | 1.72 / .82 | 0.79 |
| g001 summarize | 2.52 / .15 | 2.06 / .13 | 2.40 / .13 | 0.72 |
| g2 truncate | 1.28 / .93 | 1.02 / .98 | 2.85 / .26 | 0.81 |
| g2 summarize | 1.23 / .93 | 1.24 / .87 | 2.12 / .65 | 0.52 |
Table 2. Encoding ablation with UMAP+HDBSCAN: size-weighted judge coherence / largest-cluster share (real corpora) and MultiWOZ ARI. Red cells are degenerate (share > 0.5).
Stability. Cluster cores are highly reproducible when runs are healthy: core ARI 0.96–0.97 across seeds and 0.89 across LMSYS sample draws. The unstable components are the noise boundary (all-points ARI 0.37–0.74) and, on WildChat, collapse itself: 2 of 3 sample draws degenerated while all seed replicates on a different draw were healthy. Degeneracy is a per-draw coin flip, so the guard must run on every clustering batch.
5.6 Post-hoc ablations: leaf selection and seed replication
Leaf selection fixes the collapse for free — with trade-offs. Re-clustering every degenerate configuration with HDBSCAN cluster_selection_method="leaf" (identical UMAP spaces) de-collapsed all 12 tested configs, including both collapsed WildChat sample draws (max share 0.78–0.98 → 0.07–0.12). Costs: noise rises to 36–54% (vs. 23–34% for the summarize-then-embed fallback), and ARI splits by taxonomy granularity — Banking77 (77 fine intents) improves 0.70→0.87 while MultiWOZ (coarse domains) drops 0.79→0.65. The guard in §6 is therefore tiered: leaf first, re-encoding when coverage or a coarse taxonomy matters.
| Config (UMAP+HDBSCAN) | EOM share | Leaf share | EOM ARI | Leaf ARI |
|---|---|---|---|---|
| g2 / LMSYS | 0.93 | 0.09 | — | — |
| g2 / Arena | 0.98 | 0.12 | — | — |
| g001 / WildChat | 0.82 | 0.09 | — | — |
| g001 / Banking77 | 0.07 | 0.03 | 0.70 | 0.87 |
| g001 / MultiWOZ | 0.10 | 0.12 | 0.79 | 0.65 |
Table 3. Leaf-selection ablation (selected rows; full table in the repository). Leaf de-collapses every degenerate config; ARI moves with taxonomy granularity.
3-seed replication of the key cells. Replicating {MultiWOZ, Banking77} × 4 embeddings × {UMAP+KMeans, UMAP+HDBSCAN} over three full-pipeline seeds: the Gemini-vs-local family gap holds in every cell's mean (e.g., Banking77 HDBSCAN 0.74/0.80 vs. 0.50/0.51), but gemini-2 > gemini-001 held in only 1/3 seeds in 3 of 4 cells — the two are best treated as tied, and the single-seed maxima in §5.1 carry selection inflation. Notably, g001+HDBSCAN is the most stable combination (ARI std 0.014–0.031, vs. up to 0.276 for gemini-2 and 0.14–0.23 for Gemini+KMeans, whose silhouette-swept k is seed-sensitive) — supporting it as the default on stability grounds rather than mean accuracy. The KMeans-vs-HDBSCAN ordering is dataset-dependent, consistent with the granularity account.
6Discussion: a Guarded Pipeline
Guard (mandatory, per batch): compute the largest cluster's share of assigned sessions; if it exceeds 0.5, first re-cluster with HDBSCAN leaf selection (free; de-collapsed all 12 tested configs, and is the best choice outright for fine-grained taxonomies); when coverage matters (leaf pushes 36–54% of sessions to noise) or the target taxonomy is coarse, re-encode with summarize-then-embed (still gemini-001), or fall back to KMeans when every session must be assigned.
Variants: gemini-embedding-2 when matching a human taxonomy is the goal (best ARI, but never with the summarize encoding); MPNet + user_turns when no API budget exists.
Methodologically, three practices changed our conclusions and should generalize: compute internal metrics in the original space; weight all cluster-level scores by cluster mass; and evaluate on the deployment distribution, not only benchmarks — every qualitative surprise in this study (collapse, refusals, the coherence ceiling) appeared only on real corpora.
7Limitations
Judge and embeddings share a model family for the Gemini pipelines; although labeler and judge differ, a same-family preference cannot be fully excluded and a cross-family (e.g., Claude) judge is future work. Judge scores across encodings are not perfectly comparable, since the judge reads summaries for summarize-encoded runs; degeneracy and ARI carry that comparison. English-only sessions, 2K–10K samples, and 2K–10K samples limit scope. The full grid remains single-seed, though the key cells are replicated over 3 seeds (§5.6) and stability is assessed separately; min-samples sweeps remain untested. Coherence 1–5 is coarse; human evaluation was limited to spot checks of ≈60 cluster labels. BERTopic's full pipeline and a chunk-mean-pool encoding were not benchmarked.
8Conclusion
We presented, to our knowledge, the most comprehensive controlled study of session-level AI-chat clustering to date: 335 runs across seven datasets with internal, external, and LLM-judge evaluation. The headline results — embeddings dominate, internal metrics mislead, real corpora collapse pipelines that benchmarks crown, aggregation hides the collapse, and cores (but not noise) replicate — compose into a simple guarded pipeline that is accurate on benchmarks, robust on real data, cheap (≈$1–2 per 10K sessions), and fast (minutes).