Research report · July 2026

Routing Commerce Search Queries: Simple Lookup vs. LLM Overview

A binary classifier that decides, per query, whether an e-commerce search should return the ordinary product grid (SIMPLE — “men’s shirts”) or additionally trigger an LLM-generated shopping overview (COMPLEX — “shirts for a beach wedding in Hawaii in summer”). Built and evaluated on 6,636 ensemble-labeled queries from five public datasets, with leave-one-dataset-out generalization tests, a targeted commerce-complex challenge slice, and an explicit latency/cost frontier.

▶ Try the live demo — the exact int8 export of the recommended model, served from this site. Code on GitHub.

Recommendation. Ship the fine-tuned DistilBERT trained with commerce-complex enrichment, operated at threshold τ = 0.25. It is within noise of the best LLM-free accuracy on the IID test set, doubles recall on the commerce-complex slice the feature actually targets, and costs ~3 ms per query — 200× faster than putting an LLM in the routing hot path (649 ms p50, ~$0.02 per 1k queries).

1Problem & taxonomy

Search overviews are expensive (LLM latency + tokens) and, on straightforward queries, worse than useless — they push the product grid below the fold. The router’s job is to spend that cost only where synthesis adds value. We use a binary taxonomy:

• SIMPLE — a conventional keyword/facet backend fully serves the intent: product, brand, model, category, or any stack of directly filterable attributes. “nike air max 90 size 11”, “stainless steel dishwasher under $700”
• COMPLEX — satisfying the query needs inference or synthesis: occasion/persona translation, advice and superlatives, comparisons, compatibility, how-to, gift discovery, symptom→product. “best vacuum for pet hair”, “will a 265/70r17 tire fit a stock tacoma”

The full annotation rubric (with tie-breakers, e.g. attribute-stacking stays SIMPLE; bare superlatives are COMPLEX; canonical occasion-categories like “wedding dress” stay SIMPLE) is reproduced below and shipped as src/rubric.py.

Label SIMPLE when a conventional product-search backend fully serves the intent:
the query names a product, brand, model/SKU, or category, possibly with directly
filterable attributes (size, color, material, gender, price cap). The user knows
what they want; showing a ranked product grid is the complete answer.

Label COMPLEX when an LLM overview genuinely helps: satisfying the query requires
inference, synthesis, or explanation that a product grid alone does not provide.
Signals (any one suffices): occasion/context translation · advice/recommendation
intent incl. superlatives · multi-constraint trade-offs or persona fit ·
comparisons · compatibility/fitment · question-form or how-to ·
gift discovery · problem/symptom queries.

Tie-breakers: attribute-stacking alone stays SIMPLE no matter how many attributes;
a bare superlative ("best coffee maker") is COMPLEX; a price cap alone stays SIMPLE,
price + use case is COMPLEX; canonical occasion-categories ("wedding dress",
"running shoes") stay SIMPLE; nonsense/navigational fragments default to SIMPLE.

2Datasets

Five public query datasets, four of them commerce product-search logs and one general-web set (MS MARCO) kept deliberately as a domain-shift probe. From each we sampled, cleaned (length 3–150 chars, ≥90% ASCII, cross-dataset dedupe), and pooled 6,661 unique queries.

3Ground truth by LLM ensemble

No public dataset carries this label, so ground truth was produced by an ensemble of Gemini 2.5 Flash annotators: two independent prompt variants — A: rubric-verbatim, label-only; B: a re-framed “router” task with per-query rationale — each labeling all 6,661 queries at temperature 0, plus an adjudicator variant with a larger thinking budget that ruled only on A/B disagreements.

• Raw A/B agreement 97.0%, Cohen’s κ = 0.915 (“almost perfect” band); 201 disagreements (3%) went to adjudication. Final gold set: 6,636 queries (75.9% SIMPLE / 24.1% COMPLEX).
• Per-dataset agreement stayed ≥93% everywhere (lowest on MS MARCO’s ambiguous web queries; ≥97% on all commerce sets).
• A 70-query stratified manual audit (25 SIMPLE, 25 COMPLEX, 20 adjudicated) agreed with ~95% of gold labels; the misses were genuinely borderline (“tacky ugly christmas sweater” labeled COMPLEX).
• Pipeline properties: batched JSON-mode calls, resumable vote logs, majority aggregation. Total labeling cost for the project ≈ $2 (~17.5k votes).

4Classifier candidates

Splits: 70/10/20 train/val/test, stratified by label × dataset, fixed seed; all models share the identical test set (n = 1,328).

5Main results

Three observations. First, query complexity is largely a surface-lexical property: a 0.3 ms sparse linear model recovers ~95% accuracy, and even a length threshold gets 79%. Second, fine-tuning buys its margin mostly on complex recall (.916 vs .844 for TF-IDF at comparable precision) — exactly the metric the product cares about. Third, the LLM’s +1 pt over DistilBERT costs 200× the latency; in a hot path serving every search, that trade is unjustifiable.

6Generalization: leave-one-dataset-out

Each trainable model was retrained five times with one dataset fully held out. Accuracy stays high on held-out commerce datasets, but macro-F1 reveals the cost: recall of the rare COMPLEX class collapses on unseen domains.

The headline finding of this study: learned complexity signals do not transfer across phrasing domains. A model that has never seen question-form informational queries does not discover that “how much does X cost” is COMPLEX — no matter the architecture. Training-data diversity matters more than model capacity; the cheap insurance is a rule-OR-model union (route COMPLEX if either fires), which preserves the heuristic’s domain-independent floor.

7The commerce-complex gap

Only ~160 of the pool’s COMPLEX labels come from commerce logs — yet “shirts for a beach wedding”-style queries are the whole point of the feature. Two interventions target this gap:

1. Enrichment: 2,000 extra ESCI/Home Depot queries, sampled with a bias toward complexity signals (≥5 tokens or advice/occasion keywords), ensemble-labeled the same way (κ = 0.73 — these are harder, more borderline queries). Half joined training; half form a held-out challenge slice (1,000 queries, 65 COMPLEX).
2. Operating point: the router’s threshold is a product decision, not 0.5 by default. An unnecessary overview costs interface real estate and tokens; a missed one loses the feature’s value. We sweep τ on held-out data.

Net effect of both interventions: commerce-complex recall .292 → .600 (+31 pts) for −1.1 pt IID accuracy. The heuristic still catches slightly fewer of these than the shipped configuration while triggering falsely 3× as often.

8Error analysis

The recommended model makes 66 errors on the 1,328-query test set. Reading all of them, three clusters emerge:

• Label-noise / genuinely ambiguous (~40%). Book and media titles typed as queries (“a very stable genius”, “this is marketing…”), fragments (“time to read”), navigational lookups with question words. Both the model and any human would hedge; several gold labels are defensible either way.
• Missed inference-y queries (~35%). “diy garage door install”, “sore throat relief”, “l shaped desk for two people” — symptom/problem/persona intent phrased without any lexical trigger. This is the residual hard core that the threshold fix only partly reaches (“tv under 500 for bright room gaming” still scores p=.04).
• Over-triggers on gift/for phrasing (~25%). “birthday gifts for hanique” (a specific person’s name → really a category browse), “numbers for mailboxes for outside” — “for” as an attribute, not a persona.

9Threats to validity

1. LLM-labeled ground truth. Gold labels are Gemini-ensemble output. The κ=0.915 agreement is between two prompts of the same model family — it bounds prompt-sensitivity, not true label quality. The 70-query human audit (~95% agreement) mitigates but does not eliminate this; the Gemini few-shot “ceiling” in §5 is inflated for the same reason.
2. Domain–label confound. 90% of COMPLEX labels come from MS MARCO (Figure 1), so IID metrics partly reward domain recognition. The LODO study (§6) and the challenge slice (§7) were designed specifically to expose this — and did.
3. Challenge-slice noise. The enriched sample’s κ is 0.73 (borderline queries by construction); its recall numbers carry more label noise than the main test set.
4. English-only, US-centric, and latency measured on one machine (Apple M-series, batch = 1).

10Deployment recommendation

1. Router: distilbert_ft_enriched at τ = 0.25, in-process (~3 ms p50, no extra infra). Where 3 ms is too much, tfidf_logreg at 0.3 ms keeps 95.3% accuracy but gives up complex recall.
2. Union with the heuristic for domain robustness (§6): route COMPLEX if either the model or the question-word/comparison rules fire. The rules cost microseconds and cover phrasing families the training data may lack.
3. Overview is additive, not substitutive: at τ=.25 the router triggers on ~27% of queries; the product grid should always render, with the overview streaming in above it — misroutes then degrade gracefully in both directions.
4. Retrain quarterly from clicked-overview engagement signals; the labeling pipeline is resumable and costs ~$2 per 10k queries, so refreshing gold data is essentially free.
5. Keep the LLM out of the routing hot path. Its +1 pt accuracy costs 200× latency; the same LLM budget is better spent generating the overviews themselves.

11Reproducibility

All code is public at github.com/verma7/commerce-query-router (data and model weights gitignored, pipeline fully scripted): src/rubric.py (taxonomy), src/label.py + aggregate_labels.py (ensemble labeling, resumable), train.py / train_transformer.py / train_enriched.py (models, --lodo), gemini_classify.py (LLM baseline), evaluate.py (bootstrap metrics), predict.py (shipping Router), demo/server.py (interactive demo at localhost:8642). Raw-dataset sources and licenses: data/raw/MANIFEST.json. Splits, seeds, and all reported numbers (results/*.json) are committed. Total external cost of the study: ≈ $2 in Gemini 2.5 Flash calls.

Report generated 2026-07-23. Datasets: Amazon ESCI (Apache-2.0), Home Depot (Kaggle mirror), MS MARCO (non-commercial research terms), WANDS (MIT), TREC Product Search 2023. Labels are LLM-ensemble judgments, not human gold.