(diversity) Integrate determinant diversity in tokp full precision search and disk search by narendatha · Pull Request #1011 · microsoft/DiskANN

narendatha · 2026-05-04T12:30:38Z

Integrate determinant-diversity into topk full-precision search and disk search

Summary

Wires the determinant-diversity post-processing algorithm as an optional SearchPlugin for two search paths:

Async graph-index (full-precision topk via inmem::FullAccessor)
Disk-index (via DiskIndexSearcher post-processor)

Determinant-diversity reranks search results to increase geometric diversity among the top-k, at a configurable quality/speed tradeoff.

Motivation

Standard nearest neighbor search often returns highly similar/redundant results. For some scenarios like RAG search, it's more valuable to retrieve diverse, complementary documents that cover different aspects of the query. This PR implements a principled approach based on determinant maximization of the Gram matrix of query-scaled vectors.

Algorithm

The algorithm scales each candidate vector by (similarity_to_query)^power and then greedily selects vectors to maximize the determinant of the Gram matrix (equivalent to maximizing the volume of the parallelepiped spanned by the vectors).

Two variants are implemented:

eta = 0: Pure greedy orthogonalization (pivoted QR decomposition)
eta > 0: Ridge-regularized version that balances diversity with numerical stability

Both variants run in O(n k d) time where n = candidates, k = results to return, d = dimension.

Performance Characteristics

Tested on OpenAI embeddings dataset (3072 dimensions, 10K sample):

Metric	Normal Search	Det_Div Search	Notes
QPS	~150	~130	~13% overhead for diversity optimization
Recall@10	98-99%	87-88%	Expected trade-off for diversity

The recall reduction is expected - Det_Div search intentionally selects diverse vectors rather than the absolute nearest neighbors.

Diversity Improvement

We measure diversity using log-determinant of the Gram matrix of scaled vectors (higher = vectors span more volume = more diverse).

RAG search consistently improves diversity (all improvements >= 0)
Average improvement of +2.9 to +3.3 in log_det corresponds to 10-27x more volume spanned by selected vectors

Parameter Tuning Guide

`power` (default: 2.0)

Controls the trade-off between relevance and diversity:

Higher values (e.g., 3.0+): Strongly prefer relevance - vectors closer to the query get much higher weight, resulting in less diversity but higher recall
Lower values (e.g., 1.0): Prefer diversity over relevance - similarity weighting is more uniform, allowing more diverse vectors to be selected
Value of 0: Pure diversity maximization with no relevance weighting

`eta` (default: 0.01)

Ridge regularization parameter that controls numerical robustness and relevance preference:

Higher values (e.g., 0.1+): More robust numerically, but biases toward selecting high-relevance vectors over diverse ones
Lower values (e.g., 0.001): Closer to pure determinant maximization, maximizes diversity but may be less stable
Value of 0: Falls back to pure greedy orthogonalization (no ridge regularization)

Recommended Settings

Use Case	`power`	`eta`
Balanced (default)	2.0	0.01
High diversity	1.0	0.001
High relevance	3.0	0.1

Changes

Core algorithm (`diskann-providers`)

Added determinant_diversity_post_process() in diskann-providers/src/model/graph/provider/async_/determinant_diversity_post_process.rs
Two selection strategies controlled by eta:
- eta == 0: greedy orthogonalization
- eta > 0: eta-weighted residual selection

Async graph-index path (`diskann-benchmark`)

Added FullPrecisionVectorAccessor shim trait + impl for inmem::FullAccessor<f32, ...> so the post-processor can fetch real candidate vectors
Implemented DeterminantDiversity plugin that satisfies glue::SearchPostProcess using for<'a, 'b> SearchStrategy HRTB bound (credit: @hildebrandmw)

Disk-index path (`diskann-disk`, `diskann-benchmark`)

Added DeterminantDiversityAndFilter processor to DiskSearchPostProcessor enum
Extended searcher.search() to accept an optional SearchPostProcessorKind::DeterminantDiversity { power, eta }
Extended DiskSearchPhase JSON input to accept post_processor
Result-counting uses result_count from search stats to correctly bound post-processed results

Input/config

New TopkPostProcessor serde enum in diskann-benchmark/src/inputs/post_processor.rs with validation (power > 0, eta >= 0)
Two new example configs: async-determinant-diversity.json, disk-index-determinant-diversity.json

Benchmark JSON examples

// async graph-index
"post_processor": { "type": "determinant-diversity", "power": 1.0, "eta": 0.0 }

// disk-index
"post_processor": { "type": "determinant-diversity", "power": 1.0, "eta": 0.5 }

Notes

Determinant-diversity is only wired for full-precision topk on the async path; quantized (PQ/scalar) paths are intentionally excluded
Spherical, range, and filtered-search plugin variants are unchanged; this post-processor is topk-only

…ugins

Replace kind()-based string equality checks with explicit is_match() and get() phase-shape helpers on plugin structs. This avoids fragile ordering assumptions and makes each plugin responsible for recognising its own phase shape.

Co-authored-by: Copilot <copilot@github.com>

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/backend/index/product.rs # diskann-benchmark/src/backend/index/scalar.rs # diskann-benchmark/src/backend/index/search/plugins.rs # diskann-benchmark/src/backend/index/spherical.rs # diskann-benchmark/src/inputs/graph_index.rs # diskann-benchmark/src/inputs/mod.rs

Co-authored-by: Copilot <copilot@github.com>

…ojection issue This commit explores approaches to wire real candidate vectors into async determinant-diversity post-processing. Current state: IN COMPILATION ERROR (intentional for analysis) Attempted approaches: 1. Initial shim-trait FullPrecisionVectorAccessor with async get_full_precision_vector() - Resulted in 'implementation not general enough' at search_with() call 2. Removed explicit for<'a> post_processor::DeterminantDiversity bound - Still fails - the constraint is inherent in search_with() signature itself Root cause analysis: - search_with() requires: PP: for<'a> SearchPostProcess<S::SearchAccessor<'a>, T, O> - This means post-processor must work for ANY accessor lifetime 'a - But query = queries.row(query_idx) is borrowed for specific loop iteration lifetime - These are fundamentally incompatible - a borrowed value can't satisfy for<'a> generically Compiler errors (3 total): - 'not general enough': implementation needed for or<'a> but found specific '0 - 'does not live long enough': queries lifetime too short for 'static requirement Files modified: - diskann-benchmark/src/backend/index/benchmarks.rs: * Removed explicit for<'a> post_processor::DeterminantDiversity constraint * Narrowed plugin impl to FullPrecisionProvider<f32> - diskann-benchmark/src/backend/index/post_processor/determinant_diversity.rs: * Added shim trait FullPrecisionVectorAccessor * Async method get_full_precision_vector(&mut self, id) -> impl Future<...> Next steps to investigate: - Move determinant-diversity outside search_with() as post-processing reranking - This avoids HRTB entirely by applying after candidates are returned - Benchmark impact: measure recall/QPS with external reranking vs baseline Related context: - Disk index determinant-diversity works correctly (uses real vectors, shows 51-53% QPS cost) - Shared algorithm fixed (distance-to-similarity scoring direction) - Branch already merged with origin/main

Co-authored-by: Copilot <copilot@github.com>

…educe duplication - Use for<'a, 'b> SearchStrategy bound (user-provided fix) to break HRTB lifetime projection issue in the search_with post-processor constraint - Wire FullPrecisionVectorAccessor shim trait so async det-div post-processor fetches real candidate vectors instead of placeholder distances - Populate QPS/latency metrics in async det-div benchmark path (previously all 'missing') - Extract run_topk_timed helper to eliminate ~100 lines of duplicated loop/timing/recall machinery from DeterminantDiversity::run - Update async-determinant-diversity.json example tag (async-index-build -> graph-index-build) - Fix clippy::manual_async_fn in FullPrecisionVectorAccessor shim trait

codecov-commenter · 2026-05-06T05:22:20Z

Codecov Report

❌ Patch coverage is 73.98082% with 217 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.35%. Comparing base (3aa44ac) to head (ae4981d).

Files with missing lines	Patch %	Lines
diskann-benchmark/src/inputs/graph_index.rs	8.88%	41 Missing ⚠️
...odel/graph/provider/async_/inmem/full_precision.rs	0.00%	39 Missing ⚠️
diskann-benchmark/src/index/benchmarks.rs	16.66%	35 Missing ⚠️
diskann-benchmark/src/inputs/post_processor.rs	0.00%	27 Missing ⚠️
diskann-benchmark/src/index/search/plugins.rs	39.47%	23 Missing ⚠️
diskann-benchmark-core/src/search/graph/knn.rs	37.93%	18 Missing ⚠️
diskann-disk/src/search/provider/disk_provider.rs	93.36%	13 Missing ⚠️
.../src/model/graph/provider/determinant_diversity.rs	97.05%	12 Missing ⚠️
diskann-benchmark/src/inputs/disk.rs	0.00%	8 Missing ⚠️
diskann-tools/src/utils/search_disk_index.rs	0.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (73.98%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1011      +/-   ##
==========================================
- Coverage   89.46%   89.35%   -0.12%     
==========================================
  Files         487      489       +2     
  Lines       92170    92975     +805     
==========================================
+ Hits        82460    83075     +615     
- Misses       9710     9900     +190

Flag	Coverage Δ
miri	`89.35% <73.98%> (-0.12%)`	⬇️
unittests	`89.00% <73.98%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark/src/index/inmem/spherical.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/index/search/knn.rs	`77.77% <ø> (ø)`
diskann-benchmark/src/inputs/mod.rs	`81.25% <ø> (ø)`
diskann-disk/src/build/builder/core.rs	`95.35% <100.00%> (+<0.01%)`	⬆️
diskann-tools/src/utils/search_disk_index.rs	`0.00% <0.00%> (ø)`
diskann-benchmark/src/inputs/disk.rs	`1.50% <0.00%> (-0.07%)`	⬇️
.../src/model/graph/provider/determinant_diversity.rs	`97.05% <97.05%> (ø)`
diskann-disk/src/search/provider/disk_provider.rs	`93.59% <93.36%> (-0.09%)`	⬇️
diskann-benchmark-core/src/search/graph/knn.rs	`86.25% <37.93%> (-8.83%)`	⬇️
diskann-benchmark/src/index/search/plugins.rs	`48.14% <39.47%> (-25.44%)`	⬇️
... and 4 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Use for<'a, 'b> SearchStrategy bound to resolve HRTB lifetime mismatch - Wire FullPrecisionVectorAccessor shim trait for async det-div to fetch real vectors - Implement DeterminantDiversity post-processor for async graph-index path - Extract run_topk_timed helper to eliminate ~100 lines of code duplication - Wire post_processor parameter to disk-index search pipeline - Update search parameter handling and result counting for post-processed results - Add TopkPostProcessor input type and necessary imports - Populate QPS/latency metrics in async det-div benchmark path

…rosoft/DiskANN into u/narendatha/det_div_plugins

Co-authored-by: Copilot <copilot@github.com>

…rix-based interface - Updated diskann-providers determinant_diversity to accept MutMatrixView instead of Vec<(Id, f32, Vec<f32>)> - Maintains identical algorithmic behavior and recall across all test datasets (SIFT Small, Enron) - Improves memory layout efficiency with contiguous distance/ID arrays - Updated all call sites: diskann-benchmark inmem and diskann-disk providers - Verified with comprehensive A/B benchmarks: * Recall parity: 100% match vs archived baselines * SIFT Small (256 vectors): -45% QPS (small dataset regression) * Enron (46,750 vectors): -5% to +13% QPS (larger dataset improvement) - All tests passing, no panics or runtime errors

Replaces the previous post-processor sub-enum pattern with a dedicated TopkDeterminantDiversity variant on the SearchPhase enum, following the established pattern for Range, BetaFilter, and MultiHopFilter variants. - Add TopkDeterminantDiversityPhase struct with validate() that reuses DeterminantDiversityParams::new for power/eta validation - Add TopkDeterminantDiversity variants to SearchPhase and SearchPhaseKind - Wire plugins::DeterminantDiversity to use the new variant - Remove now-redundant post_processor field from TopkSearchPhase - Update example config (async-determinant-diversity.json) to new format - DiskSearchPhase intentionally unchanged (separate code path) Addresses PR #1011 review comment r3390077824.

…r dispatch Replace the two Search impls on KNN (one for () default, one for explicit PP) and the Option<PP> + expect(...) panic with a single AsPostProcessor trait. Defaulted resolves to the strategy's default post-processor; Forwarded<PP> carries an explicit one. KNN::new returns KNN<.., Defaulted>; KNN::with_postprocessor returns KNN<.., Forwarded<PP>>. The ConfiguredPostProcessor marker trait is removed.

…bility Remove the brittle hardcoded checks in JSON-input validation that bailed when topk-determinant-diversity was paired with non-Float32 / PQ / SQ / spherical / streaming graph builds. The plugin registry already provides the same compatibility guarantee at run time (only Float32 FullPrecision registers DeterminantDiversity), and produces a generic 'Unsupported search phase' error listing the kinds the backend actually supports.

Adds test_disk_search_determinant_diversity exercising the new SearchPostProcessorKind::DeterminantDiversity path through DiskIndexSearcher::search end-to-end. The test asserts: result count matches k, all selected ids are a subset of the L=20 candidate pool, no duplicates, top-1 matches the nearest neighbor (greedy invariant), pure-greedy (eta=0) preserves the candidate-pool subset invariant, and the vector_filter is honored by DeterminantDiversityAndFilter.

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/inputs/graph_index.rs

…plugins # Conflicts: # diskann-benchmark/src/inputs/graph_index.rs

…plugins # Conflicts: # diskann-benchmark/src/disk_index/search.rs # diskann-benchmark/src/index/benchmarks.rs # diskann-benchmark/src/index/search/plugins.rs

hildebrandmw

I think it's there! Thanks Naren!

…plugins # Conflicts: # diskann-benchmark-core/src/search/graph/knn.rs

arkrishn94

It's looking good Naren, there's just the one comment on the error handling.

…e variant

…skann-providers

suri-kumkaran

Thanks Naren for all the hard work.

Mark Hildebrand and others added 12 commits April 28, 2026 14:47

Make benchmarks stateful.

dcdc377

Checkpoint.

d60ac07

Checkopint.

6cc92bd

Oh boy.

f050e3d

Merge remote-tracking branch 'origin/main' into mhildebr/benchmark-pl…

2d03d11

…ugins

Fix delimiters.

aa78836

Revert formatting.

ccf891a

Make plugins slightly more flexible.

2e060ea

Merge branch 'main' into mhildebr/benchmark-plugins

7a85df8

Add determinant-diversity plugin support on search-plugin architecture

fc8008a

Integrate determinant-diversity via disk search_with post-processor

298d1b8

Restrict determinant-diversity to async full-precision topk

fbd34fd

narendatha changed the base branch from main to mhildebr/benchmark-plugins May 4, 2026 12:41

Keep single async determinant-diversity example JSON

851174e

Base automatically changed from mhildebr/benchmark-plugins to main May 5, 2026 00:45

narendatha and others added 10 commits May 5, 2026 12:41

Use SearchPhaseKind::as_str in benchmark plugin kinds

3a6aa1a

remove serde defaults

ccfe4d7

Co-authored-by: Copilot <copilot@github.com>

minor merge fix

86883b7

hook up actual algorithm, replace placeholder.

554bc7f

Co-authored-by: Copilot <copilot@github.com>

apply mark's beautiful fix for lifetime issue

b73abc8

Co-authored-by: Copilot <copilot@github.com>

Fix CI clippy-features spherical plugin errors and apply formatting

701ce8e

narendatha and others added 4 commits May 6, 2026 16:57

Merge branch 'u/narendatha/det_div_plugins' of https://github.com/mic…

6f47ba3

…rosoft/DiskANN into u/narendatha/det_div_plugins

imrpove code coverage

8acbba2

minor fix

a48e255

Co-authored-by: Copilot <copilot@github.com>

narendatha added 9 commits June 12, 2026 14:27

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

b416b95

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/inputs/graph_index.rs

fix test failure

11c989e

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

855fbd2

…plugins # Conflicts: # diskann-benchmark/src/inputs/graph_index.rs

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

1807bf0

…plugins # Conflicts: # diskann-benchmark/src/disk_index/search.rs # diskann-benchmark/src/index/benchmarks.rs # diskann-benchmark/src/index/search/plugins.rs

hildebrandmw approved these changes Jun 17, 2026

View reviewed changes

Comment thread diskann-providers/src/model/graph/provider/determinant_diversity.rs Outdated

narendatha enabled auto-merge (squash) June 18, 2026 11:52

narendatha disabled auto-merge June 18, 2026 11:52

narendatha added 2 commits June 18, 2026 17:39

Move DeterminantDiversityParams into determinant_diversity module

1c8591d

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

0c0a1b1

…plugins # Conflicts: # diskann-benchmark-core/src/search/graph/knn.rs

suri-kumkaran reviewed Jun 18, 2026

View reviewed changes

arkrishn94 reviewed Jun 18, 2026

View reviewed changes

Comment thread diskann-providers/src/model/graph/provider/determinant_diversity.rs

arkrishn94 approved these changes Jun 18, 2026

View reviewed changes

narendatha added 4 commits June 19, 2026 14:41

Remove redundant ordered_ids vec in determinant-diversity post-process

05a6b37

Make search() take SearchPostProcessorKind directly with explicit Non…

008f62b

…e variant

cargo fmt

3165ce4

Move DeterminantDiversity post-processor from diskann-benchmark to di…

aaca369

…skann-providers

suri-kumkaran reviewed Jun 19, 2026

View reviewed changes

Comment thread diskann-providers/src/model/graph/provider/determinant_diversity.rs Outdated

Use thiserror derive for DeterminantDiversityError

a6652ae

suri-kumkaran approved these changes Jun 19, 2026

View reviewed changes

Comment thread diskann-providers/src/model/graph/provider/determinant_diversity.rs Outdated

Comment thread diskann-disk/src/search/provider/disk_provider.rs Outdated

narendatha added 2 commits June 19, 2026 21:24

Add From<DeterminantDiversityError> for ANNError and use ? at callsites

26784b7

minor comment fix

ae4981d

narendatha enabled auto-merge (squash) June 19, 2026 16:05

suri-kumkaran approved these changes Jun 20, 2026

View reviewed changes

narendatha merged commit b5ebac2 into main Jun 20, 2026
24 checks passed

narendatha deleted the u/narendatha/det_div_plugins branch June 20, 2026 09:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(diversity) Integrate determinant diversity in tokp full precision search and disk search#1011

(diversity) Integrate determinant diversity in tokp full precision search and disk search#1011
narendatha merged 70 commits into
mainfrom
u/narendatha/det_div_plugins

narendatha commented May 4, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 6, 2026 •

edited

Loading

Uh oh!

hildebrandmw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arkrishn94 left a comment

Uh oh!

Uh oh!

suri-kumkaran left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

narendatha commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integrate determinant-diversity into topk full-precision search and disk search

Summary

Motivation

Algorithm

Performance Characteristics

Diversity Improvement

Parameter Tuning Guide

power (default: 2.0)

eta (default: 0.01)

Recommended Settings

Changes

Core algorithm (diskann-providers)

Async graph-index path (diskann-benchmark)

Disk-index path (diskann-disk, diskann-benchmark)

Input/config

Benchmark JSON examples

Notes

Uh oh!

codecov-commenter commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arkrishn94 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

suri-kumkaran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

narendatha commented May 4, 2026 •

edited

Loading

`power` (default: 2.0)

`eta` (default: 0.01)

Core algorithm (`diskann-providers`)

Async graph-index path (`diskann-benchmark`)

Disk-index path (`diskann-disk`, `diskann-benchmark`)

codecov-commenter commented May 6, 2026 •

edited

Loading