Fix #172 for real: subquery filter-after-ANN (planner now uses the HNSW index at scale)

[mkreyman]

Summary

The merged #172 fix (089d592) was deployed and still 500'd in production — I verified this against the live system rather than trusting the green CI:

• fly logs: ERROR 57014 (query_canceled) "canceling statement" in suggest_links/3, firing after the deploy.
• EXPLAIN on the real ~76k-row corpus: Seq Scan on articles + Sort, cost ~57,000 — the HNSW index was never chosen. The prior EXPLAIN test only proved index eligibility (a forced plan with sort disabled), never that the planner picks the index at scale. That gap is exactly what the skeptical-BA reviewer flagged and I closed Verify follow-up (#168/#170): suggested_links STILL 500s in prod — unbounded full-corpus scan (scale), not param interpolation #172 without prod verification. Owned.

Root cause (EXPLAIN-verified on prod): pgvector's HNSW index only accelerates a pure ORDER BY embedding <=> $const LIMIT k. The already-linked anti-join (a JOIN) and the > threshold distance filter, in the same query, make the planner abandon the index and full-scan. (search_semantic has neither — that's why it's fast.) Confirmed by isolating each: anti-join alone → Seq Scan (cost 105k); threshold alone → Seq Scan; pure ANN → Index Scan using articles_embedding_hnsw_idx (cost 742).

Fix

Split into a subquery (the standard pgvector filter-after-ANN pattern, already used by do_distant_pairs):

• INNER: pure top-pool nearest by cosine — only index-safe filters (tenant / status / not-null / not-self / visibility, each EXPLAIN-verified on prod to keep the index).
• OUTER: the anti-join + threshold + final limit, over just pool rows.

Verified on the real prod corpus, including the agent visibility filter (the actual failing path): EXPLAIN ANALYZE = 73 ms, Index Scan using articles_embedding_hnsw_idx, no Seq Scan on articles.

pool = max(limit×5, 100) capped 500, floored at limit.

Enhanced review (team round 1 already in branch history; focused adversarial round 2 here)

Security + engineer (Opus), every claim verified against code/prod:

• Sound: tenant isolation, visibility (filter forced onto the inner subquery), bidirectional anti-join, ordering, DoS bounds — all disproven as attacks.
• Recall change (documented): exclusions now apply to the candidate pool, not the whole corpus, so a densely-linked hub may return fewer than limit. The alternative (anti-join inside the index scan) is prod-disproven (Seq Scan, cost 105k) — the ceiling is intrinsic to the indexed path. Documented on suggest_links/3 + the OpenAPI description.
• Pool floored at limit (operator-misconfig hardening); EXPLAIN guard strengthened to require an actual Index Scan using the HNSW index.

Test that would have caught this

The EXPLAIN guard now asserts the corpus is reached via an Index Scan using articles_embedding(_hnsw)?_idx with no Seq Scan on articles (an outer sort over the small pool is expected) — the invariant that both #170 and the first #172 attempt violated. (Caveat: small-DB CI still can't reproduce the planner's cost-based choice at 76k rows — see the architectural note below.)

Notes for the operator

• Migration drift: prod's index is articles_embedding_hnsw_idx, but migration 20260410022906 creates articles_embedding_idx. Both are HNSW cosine; the query uses whichever exists. Worth reconciling.
• Verification gap: this class of bug (planner choice at scale) is invisible to the test suite. The durable fix is a scale-representative test/staging env + surfacing the real SQLSTATE instead of a generic 500 — proposed as a follow-up architectural track.

Gate: compile --warnings-as-errors, format, credo --strict, dialyzer, 2532 tests — all green.

Fixes #172