Contact
EngineeringNov 16, 2026·38 min read

RAG in Production: Hybrid Search, Reranking and pgvector Filters, Measured

KT
Keplaris TeamNov 16, 2026
On a cream background, a matte black rack of upright frosted glass slabs with three raised slabs marked by green lights, joined by a thin blue line through a black ring to a pedestal holding a ranked stack of three glass tiles

RAG in production can only answer from what retrieval returns, and in our test no single retrieval lever helped on all three public datasets. On pgvector 0.8.7 defaults, a tenant holding 1% of 500,000 vectors got a mean of 0.45 rows when it asked for 10, with no error. Hybrid search beat dense retrieval on SQuAD (mean nDCG@10 +0.090), lost to it on a Natural Questions sample (-0.037, 95% interval -0.056 to -0.019) and made no measurable difference on SciFact. Larger embedding models never scored below smaller ones on point values, but gte-modernbert-base's lead over bge-base-en-v1.5 cleared its interval on SciFact and SQuAD and not on Natural Questions (+0.007, interval -0.007 to +0.021).

We ran the test on three public datasets (SciFact, a 60,000-passage Natural Questions sample and SQuAD v1.1, plus a 500,000-passage Natural Questions sample for pgvector) with open models on an Apple M1 Max, and pgvector 0.8.7 on PostgreSQL 18.7. We measured retrieval only: no LLM generated or scored an answer, and no paid or hosted API was called. The scripts and their unedited output are at the end.

It's for the engineer whose production RAG system works on a demo set and who is deciding what to add before launch. Building retrieval into assistants grounded in a company's own data is part of our automation and AI systems work. The open models here are a test choice so anyone can rerun this without an account or API key, not a claim about what we ship.

RAG in production: the short answer

No retrieval lever helped on all three public datasets we tested. Larger embedding models never scored lower but cleared their intervals on two of three; hybrid search and cross-encoder reranking helped, hurt or did nothing depending on the dataset; chunk size moved results most, with 64-token chunks and 20% overlap putting the answer inside a 1,024-token budget for 0.864 of SQuAD questions against 0.296 for 1,024-token chunks; and on pgvector 0.8.7 defaults a tenant holding 1% of 500,000 vectors got a mean of 0.45 rows of 10, with no error.

Apple M1 Max; parts A, B and C one run each, latency two passes. NQ is the Natural Questions sample. nDCG@10 and recall are means over queries; times are medians. A paired difference is B minus A with a 95% bootstrap interval; "includes 0" means no measurable difference on that set.

LeverWhat we comparedResultVerdict
Hybrid search (RRF)BM25 plus bge-small against bge-small alone, nDCG@10SciFact +0.015, includes 0; NQ -0.037, lower; SQuAD +0.090, higherTest on your data
Cross-encoder rerankingms-marco-MiniLM-L6-v2 cross-encoder on the hybrid top 50SciFact -0.031, lower; NQ +0.061, higher; SQuAD +0.087, higher; 50 candidates took a median of 105.45 to 161.71 ms (latency pass 1)Test on your data
Bigger rerankerbge-reranker-base on the hybrid top 50SciFact includes 0; NQ -0.108, lower; SQuAD +0.034, higherNot safer; test it
Embedding modelgte-modernbert-base against bge-base, nDCG@10SciFact +0.027, higher; NQ +0.007, includes 0; SQuAD +0.051, higherNever lower; test on your data
bge query prefixWith the prefix against without, bge-smallSciFact and SQuAD include 0; NQ +0.039, higherKeep it
Chunk size (SQuAD)Answer inside a 1,024-token budget, bge-small64 tokens with 20% overlap 0.864; paragraphs 0.829; 256 tokens 0.634; 1,024 tokens 0.296Small chunks inside the model's window
HNSW ef_search500,000 vectors, against exact searchDefault 40: index recall 0.9175, nDCG@10 0.6669 against 0.7177 exact; 200: 0.9777 and 0.7054 at 3.95 ms serverRaise it above 40
Tenant filter, defaultsMean rows returned of 10 askedWhale (40%) 9.96, mid (5%) 1.92, small (1%) 0.45, tiny (0.1%) 0.04Don't ship defaults
Iterative scan, relaxed_orderSame filter10.00 rows (tiny: 1 of 500 queries short); small 25.73 ms, tiny 52.45 ms serverTurn it on
B-tree on tenant_idPlanner choosesTiny: exact search, 0.30 ms; small still 0.45 rows with iterative scan offAdd it, with iterative scan
RLS policy instead of WHEREPolicy as app_userDirect current_setting(): same rows and plans; wrapped in a SELECT with the B-tree: whale 138.37 msCheck the plan for your policy form
Partial HNSW indexWhale and mid onlyIndex recall 0.9212 and 0.9526, 0.51 and 0.42 ms serverFine for a few big tenants
Quantizationhalfvec; binary with re-rankhalfvec 0.9179 recall at 557 MB against 975 MB; binary, top 400 re-ranked, 0.9422halfvec yes; binary only with re-rank
Changing modelsEmbed against index build810 s to embed 500,000 passages; 50.0 s to build HNSWBudget for the embed

The rules we'd ship with:

  1. Measure hybrid search and reranking on labelled queries from your own corpus before adding either.
  2. Try a larger embedding model, and keep chunks inside its window.
  3. Keep the query instruction a model's card asks for.
  4. Raise hnsw.ef_search above 40 and check index recall against exact search after every index or setting change.
  5. Don't run a filtered vector query on pgvector defaults: turn on iterative scans, add a B-tree on the filter column, and log how many rows came back.
  6. Read the plan of your RLS policy form with your real indexes before trusting its latency.
  7. Use halfvec indexes; use binary indexes only with a re-rank on full vectors.

How we measured: three public datasets, open models and pgvector 0.8.7 on an Apple M1 Max

Environment. Apple M1 Max, 64 GB, macOS 27.2.1; models ran on the Apple GPU (MPS). Python 3.12.14, torch 2.15.0, sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5, bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector Python client 0.5.1, seed 20261116. Everything was read from local folders with HF_HUB_OFFLINE=1; output headers print model revisions. Postgres ran in Docker 29.7.2 (10 CPUs and 15,690,960,896 bytes of memory in the VM) from pgvector/pgvector:0.8.7-pg18 pinned by digest: PostgreSQL 18.7, pgvector 0.8.7. Non-default server settings: shared_buffers 4GB, maintenance_work_mem 4GB, max_parallel_maintenance_workers 7, max_wal_size 8GB, --shm-size=5g; pgvector settings at defaults. One autocommit connection with prepare_threshold=None, so each query was planned with its real values.

Models. Sizes, windows and licences from each Hugging Face card, read on 2026-11-16; all are ungated.

ModelRoleDimensionsReads at mostQuery prefixSize (card)Licence
all-MiniLM-L6-v2Embedder384256 word piecesNone22.7MApache-2.0
bge-small-en-v1.5Embedder384512 tokensYes33.4MMIT
bge-base-en-v1.5Embedder768512 tokensYes0.1BMIT
gte-modernbert-baseEmbedder7688,192 tokensNone0.1BApache-2.0
ms-marco-MiniLM-L6-v2Cross-encodern/a512 (our setting)n/a22.7MApache-2.0
bge-reranker-baseCross-encodern/a512 (our setting)n/a0.3BMIT

Datasets. We publish aggregate scores only and quote no dataset text. BEIR's README adds: "It remains the user's responsibility to determine whether you as a user have permission to use the dataset under the dataset's license."

DatasetPartSize as runLicence at source (read 2026-11-16)
SciFact (BEIR)A5,183 documents, 300 test queries, 339 labelsClaims CC BY 4.0, abstracts ODC-By 1.0
Natural Questions (BEIR) sampleA60,000 of 2,681,468 passages (all 4,201 passages labelled for any test query, plus seeded random ones), 600 seeded test queriesCC BY-SA 3.0
SQuAD v1.1 paragraphsA20,963 paragraphs from 490 articles, 600 seeded dev questions, one labelled paragraph eachCC BY-SA 4.0
SQuAD v1.1 articlesB490 articles (3,195,442 tokens), 2,000 seeded dev questions, answer spans as labelsCC BY-SA 4.0
Natural Questions (BEIR)C500,000 of 2,681,468 passages (all 4,201 labelled ones kept), 3,452 test queriesCC BY-SA 3.0

Both Natural Questions sets are samples of the full corpus, so their scores are not comparable with published Natural Questions scores. NFCorpus was downloaded and left out, because its site says it "is free to use for academic purposes"; no script reads it. An earlier pass of part A today included FiQA-2018; its site says the data is available only for non-commercial use, so we removed it, left its scores out and reran part A with a Natural Questions sample.

Metrics. nDCG@10 with linear gain, MRR@10, recall@10 and recall@100, each a mean over queries, with a 95% bootstrap interval (10,000 resamples) on nDCG@10. Comparisons are paired: the interval is on the per-query difference. Part C keeps index recall (overlap with exact top 10) separate from nDCG@10 against labels.

Runs. One published run of each part so far, all exit code 0: part C prep (14:11), part C (14:32), part B (14:53), part A run 2 (finished 16:23), then two latency passes on their own (to 16:30 and 16:38). Part A run 2 reused the SciFact and SQuAD embeddings cached by part A run 1 earlier the same day (12:40 to 12:44 and 13:40 to 13:47) and says so in its output, so SciFact and SQuAD embedding throughput was measured by part A run 1, a pass that also included FiQA-2018 and is otherwise retired; embedding speed does not depend on which other datasets ran. Part A also times stages between its heavy passes; those rows stay in its output unedited, but they run beside heavy work, so the latency tables in this post, outside part C, come from the separate pass, rag_bench_latency.py. Earlier, a first start of part A was stopped, the scripts patched and the cache emptied, and part B had a 200-question smoke test; neither is used here. Earlier latency passes that included FiQA-2018 were discarded.

Hybrid search vs vector search: BM25, embeddings and RRF on each dataset

Hybrid search beat dense search on SQuAD, made no measurable difference on SciFact and lost to it on the Natural Questions sample. BM25 here is bm25s (Lucene variant, k1 1.5, b 0.75, stopwords, Snowball stemmer), not Postgres full-text search; we did not test Postgres full-text search (ts_rank) as the keyword side, so these results say nothing about a hybrid built entirely in Postgres. Dense retrieval is exact cosine top 100 in numpy. The hybrid is reciprocal rank fusion (RRF, from Cormack, Clarke and Büttcher, SIGIR 2009): each document scores 1 / (60 + rank) in the BM25 top 100 and in the dense top 100, summed.

Apple M1 Max, MPS, part A run 2 (one published run); mean nDCG@10 with 95% bootstrap interval. SciFact 300 queries, Natural Questions sample 600, SQuAD 600.

RetrieverSciFactNQ sampleSQuAD
BM250.686 [0.642, 0.729]0.722 [0.693, 0.752]0.828 [0.803, 0.852]
Dense all-MiniLM-L6-v20.645 [0.600, 0.690]0.870 [0.851, 0.890]0.713 [0.683, 0.743]
Dense bge-small-en-v1.50.713 [0.670, 0.754]0.892 [0.873, 0.910]0.734 [0.705, 0.763]
Dense bge-base-en-v1.50.740 [0.700, 0.779]0.915 [0.898, 0.930]0.752 [0.724, 0.779]
Dense gte-modernbert-base0.767 [0.729, 0.804]0.921 [0.906, 0.936]0.803 [0.778, 0.829]
Hybrid, BM25 + bge-small0.728 [0.686, 0.768]0.855 [0.833, 0.875]0.823 [0.799, 0.847]
Hybrid, BM25 + bge-base0.738 [0.698, 0.778]0.863 [0.842, 0.884]0.831 [0.808, 0.854]
Hybrid, BM25 + gte-modernbert-base0.770 [0.732, 0.808]0.872 [0.852, 0.892]0.861 [0.839, 0.881]

Paired, hybrid minus dense: with bge-small, SciFact +0.015 [-0.006, +0.035], NQ -0.037 [-0.056, -0.019], SQuAD +0.090 [+0.070, +0.109]; with bge-base, SciFact -0.002 (includes 0), NQ -0.051 (lower), SQuAD +0.079 (higher). On SQuAD the bge-small hybrid did not beat BM25 alone (0.823 against 0.828, not paired).

RRF rewards documents both lists rank well, so a much weaker list drags the fused one toward the middle: on SQuAD the hybrid landed between its inputs, on SciFact, where they were close, above both (point values), and on the Natural Questions sample, where BM25 trailed bge-small by 0.170 (paired), it landed between them, below dense. SQuAD's questions were written by crowdworkers reading the paragraph (Rajpurkar et al., 2016), so they share words with it and favour BM25.

Anthropic reported a 67% relative cut in top-20 retrieval failure (5.7% to 1.9%), with paid models (Anthropic, 2024-09-19); our assistant guide covers the retrieval techniques and Anthropic's numbers. Their metric is 1 minus recall@20 on other corpora, so the sizes aren't comparable with ours.

Embedding models and the query prefix

gte-modernbert-base was higher than bge-base on SciFact (+0.027 [+0.003, +0.051]) and SQuAD (+0.051 [+0.030, +0.073]); on the Natural Questions sample the interval included 0 (+0.007 [-0.007, +0.021]). bge-base over bge-small cleared the interval on SciFact (+0.028) and Natural Questions (+0.023), not on SQuAD. Leaving out the bge query instruction made no measurable difference on SciFact or SQuAD for either bge model, and cost bge-small 0.039 [+0.028, +0.051] and bge-base 0.014 [+0.005, +0.023] on Natural Questions. The bge cards say v1.5 works without it with a slight drop.

Two cautions. Benchmark data sits in training mixes: the all-MiniLM-L6-v2 card lists Natural Questions and SQuAD2.0 among its training pairs, and the bge and gte cards don't name their datasets. As a check, the gte-modernbert-base card gives a BEIR nDCG@10 (times 100) of 77.4 on SciFact against our 0.767; the bge cards give no per-dataset scores to check.

Verdict: test hybrid search per corpus, not by default: it helped on SQuAD, where BM25 was strong, did nothing measurable on SciFact, and hurt on Natural Questions, where BM25 was weak. Larger embedding models never scored lower, but no model change cleared its interval on all three sets.

Is a cross-encoder reranker worth the latency?

A cross-encoder reranker was not worth adding by default in our test: reranking the top 50 of a BM25 plus bge-small-en-v1.5 hybrid with ms-marco-MiniLM-L6-v2 raised mean nDCG@10 on SQuAD (+0.087) and the Natural Questions sample (+0.061), lowered it on SciFact (-0.031), and took a median of 105.45 to 161.71 ms per query on an Apple M1 Max GPU (latency pass 1); bge-reranker-base lowered it on Natural Questions (-0.108). Each cross-encoder rescored the head of a first-stage list and left the rest of the top 100 in order.

Apple M1 Max, MPS, part A run 2 (one published run); mean nDCG@10. Same queries as above.

First stage, then rerankerSciFactNQ sampleSQuAD
Hybrid, BM25 + bge-small, no rerank0.7280.8550.823
Hybrid, then MiniLM cross-encoder on top 100.7120.9050.888
Hybrid, then MiniLM cross-encoder on top 200.7010.9130.904
Hybrid, then MiniLM cross-encoder on top 500.6960.9150.910
Hybrid, then bge-reranker-base on top 100.7300.8750.861
Hybrid, then bge-reranker-base on top 200.7220.8480.866
Hybrid, then bge-reranker-base on top 500.7190.7470.858
BM25, no rerank0.6860.7220.828
BM25, then MiniLM cross-encoder on top 500.6850.8720.904
BM25, then bge-reranker-base on top 500.7090.8340.867
Dense bge-small, no rerank0.7130.8920.734
Dense bge-small, then MiniLM cross-encoder on top 500.6960.9160.883
Dense bge-small, then bge-reranker-base on top 500.7150.6850.819

Paired, over the hybrid top 50: MiniLM -0.031 [-0.060, -0.003] on SciFact, +0.061 [+0.044, +0.079] on Natural Questions, +0.087 [+0.068, +0.106] on SQuAD; bge-reranker-base included 0 on SciFact, was higher on SQuAD (+0.034) and lower on Natural Questions (-0.108 [-0.135, -0.081]). Over BM25 alone, bge-reranker-base was higher on all three (SciFact +0.023 [+0.000, +0.045], lower bound rounding to zero; Natural Questions +0.111; SQuAD +0.040). With bge-reranker-base, depth 50 against depth 10 included 0 on SciFact and SQuAD and was far lower on Natural Questions (-0.128 [-0.148, -0.108]). On that set it also took the dense bge-small top 50 from 0.892 to 0.685 (not paired): this reranker reordered good dense results badly on this set at depth 50.

Median ms to rerank one query, Apple M1 Max GPU, two latency passes, 100 queries after 5 warm-ups: pass 1 median (pass 1 p95); pass 2 median.

Reranker and candidatesSciFactNQ sampleSQuAD
MiniLM cross-encoder, 1035.91 (38.34); 34.6719.67 (28.56); 20.1822.35 (30.77); 22.14
MiniLM cross-encoder, 2070.27 (79.57); 64.0336.97 (58.18); 37.6141.23 (57.59); 40.88
MiniLM cross-encoder, 50161.71 (171.85); 158.32105.45 (157.67); 111.02110.16 (146.20); 117.29
bge-reranker-base, 10173.28 (204.26); 172.9088.34 (142.83); 91.52105.06 (155.26); 113.76
bge-reranker-base, 20346.23 (353.75); 346.13215.10 (328.06); 227.02227.79 (344.69); 240.02
bge-reranker-base, 50845.07 (858.69); 866.77656.14 (822.55); 680.53712.07 (849.63); 755.83

The 22.7M-parameter MiniLM cross-encoder (trained on MS MARCO passage ranking, per its card) scored above the 0.3B-parameter bge-reranker-base on SQuAD and Natural Questions at every depth (point values, not paired) at 15% to 22% of its median latency. It hurt on SciFact, whose queries are scientific claims, not search questions; we didn't test why. The nearest public number, clouatre-labs' rag-reranking-benchmarks, reports a mean overhead of 31.3 ms for FlashRank's ms-marco-MiniLM-L-12-v2 on a MacBook Pro CPU, over 480 timed and 32 scored queries: a different model, setup and device.

Verdict: a reranker isn't a default. Where it helped, the small MiniLM cross-encoder scored at least as well as the larger one at a fraction of the time. A bigger reranker was not safer, and more candidates were not better: with bge-reranker-base, depth 10 did as well as 50 on SciFact and SQuAD and far better on Natural Questions.

RAG chunk size at a fixed context-token budget

At a fixed 1,024-token context budget on SQuAD v1.1, smaller chunks put the answer in front of the model more often with bge-small-en-v1.5: 64-token chunks with 20% overlap reached 0.864 of 2,000 questions, the articles' own paragraphs 0.829, and 1,024-token chunks 0.296, the worst. Part B re-chunks the 490 SQuAD articles by bge-small tokens (an LLM's tokenizer counts differently), then adds ranked chunks until the next would overflow a 512, 1,024 or 2,048-token budget. A fixed budget, not a fixed number of chunks, stops bigger chunks winning just by carrying more text; overlap's duplicated text counts against the budget.

Apple M1 Max, MPS, part B, one run, SQuAD v1.1, 2,000 questions, dense bge-small; share of questions with a chunk containing the answer inside the budget.

ChunkingChunksHit@5512-token budget1,024 tokens [95% CI]2,048 tokens
64 tokens, no overlap50,1660.7020.7480.805 [0.788, 0.823]0.844
64 tokens, 20% overlap62,7600.7770.8190.864 [0.849, 0.878]0.903
128 tokens, no overlap25,2100.6980.6660.754 [0.735, 0.772]0.825
128 tokens, 20% overlap31,4550.7380.7130.790 [0.773, 0.808]0.853
256 tokens, no overlap12,7270.6740.5160.634 [0.613, 0.655]0.754
256 tokens, 20% overlap15,7190.6930.5400.656 [0.635, 0.677]0.760
512 tokens, no overlap6,4870.6620.3640.505 [0.482, 0.527]0.627
512 tokens, 20% overlap7,9140.6800.3880.526 [0.504, 0.548]0.648
1,024 tokens, no overlap3,3670.576does not fit0.296 [0.276, 0.316]0.419
1,024 tokens, 20% overlap4,0250.599does not fit0.321 [0.300, 0.341]0.449
Paragraphs as written20,9630.8180.7500.829 [0.811, 0.845]0.888

Paired against 256 tokens with no overlap at the 1,024 budget, every smaller size and the paragraphs were higher, 512 and 1,024-token chunks lower, and 256 with overlap higher by +0.021 [+0.001, +0.042]; overlap scored higher at every size on point values. BM25 was less sensitive (0.684 to 0.892 across configs) and the hybrid ranged from 0.459 to 0.915 (full tables in the output).

The SQuAD caveat applies with force: each question was written by a person looking at one paragraph, so the paragraph is the unit the labels favour. Chroma's chunking evaluation (2024-07-03) used LLM-generated queries and five retrieved chunks, scored by token overlap; NVIDIA's chunking test (2025-06-18) fixed top-k at 10, scored answers with LLM judges and found page-level chunks most consistent.

Titles and paragraph boundaries

Prepending the article title made no measurable difference: for 256-token chunks with the title inside the window, the paired difference against plain 256-token chunks included 0 for all three retrievers. For paragraphs, with and without the title scored 0.823 and 0.829 with bge-small and 0.892 and 0.884 with BM25 (not paired). The articles' own paragraphs beat every fixed size except 64 tokens with overlap, on point values.

Chunks longer than the embedding model reads

A chunk longer than the model's window is embedded from its opening only. bge-small reads 512 tokens, 510 of them text; gte-modernbert-base reads 8,192. Apple M1 Max, part B, one run, SQuAD v1.1, 1,024-token chunks; hit@5 by where the answer sits in its chunk:

RetrieverAnswer in the first 510 tokens (1,058 questions)Answer after token 510 (939 questions)
Dense bge-small (512-token window)0.7380.395
BM25 (reads the whole chunk)0.8920.902
Hybrid, BM25 + bge-small0.8710.651
Dense gte-modernbert-base (8,192-token window)0.8120.740

BM25 is the control: it reads every token, and it found both halves alike. Nothing warns you when this happens; the embedding call succeeds.

Verdict: chunk to fit the embedder's window with room to spare, try 64 to 128 tokens with 20% overlap or your documents' own paragraphs, and measure at the context budget you will actually send.

pgvector HNSW vs exact search: what ef_search costs in recall and nDCG@10

On pgvector 0.8.7 with 500,000 bge-small vectors, the HNSW index at the default ef_search of 40 lost quality you can see on labels: index recall 0.9175 and nDCG@10 0.6669, against 0.7177 for exact search, at a median of 0.69 ms server time. Part C loads 500,000 bge-small vectors of Natural Questions passages, builds an HNSW index with pgvector defaults (m 16, ef_construction 64) and runs every test query.

Apple M1 Max, Docker 29.7.2, PostgreSQL 18.7, pgvector 0.8.7, part C, one run. Quality over 3,452 queries (means); timing over 500 queries after 50 warm-ups (medians). Exact nDCG@10 on labels: 0.7177, recall@10 0.8780.

SettingIndex recall@10 against exactnDCG@10 on labelsClient ms median (p95)Server ms median
Exact scan, index disabled (50 queries)1.0000n/a86.53 (94.94)88.10
ef_search 100.78080.58642.78 (6.71)0.26
ef_search 200.86360.63591.26 (2.19)0.33
ef_search 40, default0.91750.66691.96 (3.27)0.69
ef_search 1000.95990.69463.68 (5.39)1.89
ef_search 2000.97770.70545.75 (7.62)3.95
ef_search 4000.98760.710910.02 (13.90)7.78

Read the server column for comparisons: client time includes Docker's port mapping on macOS, and ef_search 10 was the first index block after the build, which may explain its client median above 20's. The sample keeps every labelled passage among 500,000, so its nDCG@10 is higher than a full-corpus run would be and is not comparable with published Natural Questions scores; it compares index settings with each other. Loading took 5.4 s by binary COPY; the 782 MB table got a 975 MB index in 50.0 s.

Verdict: don't leave ef_search at 40 for RAG; on this data, 200 recovered about three quarters of the nDCG@10 gap to exact search for about 3 ms more server time.

Filtered vector search: tenant filters, iterative scans and RLS in pgvector

In pgvector 0.8.7, a tenant filter on an HNSW index returned fewer rows than asked, with no error, and the smaller the tenant the fewer: a mean of 1.92 rows of 10 for a tenant holding 5% of 500,000 vectors, 0.45 for 1% and 0.04 for 0.1%, until we turned on iterative scans or gave the planner a B-tree. Tenants are seeded synthetic labels, independent of content: whale 199,922 rows, mid 24,965, small 5,012, tiny 471, and 996 others holding 269,630. Filtered rows report index recall against exact top 10 within the tenant, not nDCG@10, because synthetic tenants make the relevance labels meaningless inside one tenant.

Why a filtered HNSW search can return fewer than 10 rows

The pgvector README for v0.8.7 says it directly: "With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average."

Apple M1 Max, PostgreSQL 18.7, pgvector 0.8.7, part C, one run, block A: one HNSW index, no index on tenant_id, 500 queries per tenant. Recall and rows are means, times are medians.

SettingTenantIndex recall within tenantRows of 10Queries shortServer ms
Iterative scan off, ef_search 40Whale (40%)0.87429.96100.76
Iterative scan off, ef_search 40Mid (5%)0.19181.925000.73
Iterative scan off, ef_search 40Small (1%)0.04520.455000.87
Iterative scan off, ef_search 40Tiny (0.1%)0.00440.045000.80
Iterative scan off, ef_search 200Mid (5%)0.82068.692335.01
Iterative scan off, ef_search 200Small (1%)0.20082.015003.63
Iterative scan off, ef_search 200Tiny (0.1%)0.02000.205004.10

Every row used passages_hnsw with the tenant as a Filter. Raising ef_search is not a fix: at 200 the 5% tenant was still short on 233 of 500 queries.

Iterative scans: strict_order vs relaxed_order

Iterative scans, added in pgvector 0.8.0, keep scanning the index until enough rows pass the filter or the scan reaches hnsw.max_scan_tuples (20,000 by default). The README: "Strict ensures results are in the exact order by distance," while "Relaxed allows results to be slightly out of order by distance, but provides better recall."

Same machine and run, block A, ef_search 40:

SettingTenantIndex recall within tenantRows of 10Queries shortServer ms
relaxed_orderWhale (40%)0.878210.0001.14
relaxed_orderMid (5%)0.909810.0006.80
relaxed_orderSmall (1%)0.936010.00025.73
relaxed_orderTiny (0.1%)0.697410.00152.45
strict_orderMid (5%)0.819010.0007.18
strict_orderSmall (1%)0.841410.00027.20
strict_orderTiny (0.1%)0.685810.00152.50
relaxed_order, max_scan_tuples 200,000Small (1%)0.936010.00021.17
relaxed_order, max_scan_tuples 200,000Tiny (0.1%)0.703210.00137.27

strict_order had lower index recall than relaxed_order for the mid and small tenants, as the README's wording suggests. Prior tests used other setups: Percona's six-engine benchmark (Evgeniy Patlan, 2026-10-29, updated 2026-11-01) measured recall@10 of 0.1034 at 1% selectivity on pgvector 0.8.6 defaults with 990,000 vectors of 1,536 dimensions, and says iterative scan "was not measured here"; AWS's Aurora post (Shayon Sanyal, 2025-05-28) tested pgvector 0.8.0 on 10 million 384-dimension vectors with a 2% filter.

A B-tree on tenant_id or a partial index per tenant

A B-tree on tenant_id fixed only the smallest tenant (0.1%, 471 rows): in block C the planner searched it exactly through passages_tenant in 0.30 ms and stayed on HNSW for larger tenants. So the 1% tenant still got 0.45 rows with iterative scan off, and 10.00 rows at 16.40 ms with relaxed_order. A 2026 paper by Amanbayev et al. (arXiv 2602.11443) reports that "pgvector's cost-based query optimizer frequently selects suboptimal execution plans" for filtered search.

Partial HNSW indexes (block E, whale and mid only) returned 10.00 rows on every query, with the highest index recall of any approximate row at ef_search 40 for those tenants (0.9212 and 0.9526, at 0.51 and 0.42 ms server); they took 17.9 s (390 MB) and 2.5 s (49 MB) to build. The cost is operational: an index per tenant to create, maintain and match by writing the tenant id into the SQL. The README suggests partial indexes "if filtering by only a few distinct values" and partitioning for many.

The same search under a row-level security policy

A row-level security policy in place of an explicit WHERE tenant_id returned the same rows and plans in pgvector 0.8.7 when the policy called current_setting() directly; wrapping it in a SELECT with a B-tree present sent every tenant to the B-tree, costing the 40% tenant a median of 138.37 ms server time. With a policy and no WHERE (blocks B and D, as app_user with NOBYPASSRLS, ENABLE and FORCE ROW LEVEL SECURITY), rows, index recall and plans matched the explicit WHERE rows when the policy called current_setting() directly, and in either form before the B-tree existed. With the policy wrapped in a SELECT and the B-tree present, the planner chose the B-tree for every tenant: index recall 0.9998 to 1.0000, but a median of 138.37 ms server time for the whale (mid 27.19, small 4.71). The script prints node types, not row estimates, so we claim no cause; our RLS benchmark found the wrapped form's row estimate far off for a large tenant, and how a policy is applied before your own filter is covered there.

These blocks set the tenant with set_config('app.tenant_id', ..., false) per session; a pooled app should set it per transaction, as in how to carry tenant context to the database. A metadata filter is not tenant isolation; the ways RLS leaks, tested, are a separate post. Retrieved documents are also input to the model, so treat them as untrusted, as in designing around the lethal trifecta.

Traced: the tiny tenant through every block

For the 0.1% tenant (471 of 500,000 rows), pgvector 0.8.7 defaults returned 0.04 rows of 10, iterative scans filled the result at about 52 ms server time, and a B-tree on tenant_id gave exact results in 0.30 to 0.36 ms. In detail: on defaults, Index Scan passages_hnsw +Filter returned 0.04 rows of 10 on all 500 queries. Iterative scans filled the result (one query still short) at about 52 ms server, with index recall near 0.70 in either order, and the same 0.6974 under the direct RLS policy. Once the B-tree existed, the plan became Bitmap Heap Scan on passages_tenant in blocks C and D, with or without iterative scan: index recall 1.0000 at 0.30 to 0.36 ms.

Verdict by tenant size: the whale (40%) lost little on HNSW (9.96 rows on defaults); mid to small tenants (5% to 1%) need relaxed_order, and mid can get a partial index; the tiny tenant (0.1%) is best served exactly through a B-tree. Check rows returned, not just latency.

halfvec and binary quantization on a 384-dimension model

On pgvector 0.8.7 with 384-dimension vectors, a halfvec index matched the full-precision index (index recall 0.9179 against 0.9175) at 557 MB against 975 MB, 57% of its size; a binary index needed a re-rank on full vectors to be usable (0.4745 recall without, 0.9422 with the top 400 re-ranked).

Apple M1 Max, PostgreSQL 18.7, pgvector 0.8.7, part C, one run, unfiltered, 500,000 vectors, quality over 3,452 queries (means), timing over 500 (medians).

IndexSizeBuildIndex recall@10 against exactnDCG@10 on labelsServer ms
Full vectors, ef_search 40975 MB50.0 s0.91750.66690.69
halfvec, ef_search 40557 MB32.5 s0.91790.66900.45
Binary, no re-rank166 MB21.8 s0.47450.54110.54
Binary, top 40 re-ranked on full vectors166 MB0.71770.63080.57
Binary, top 100 re-ranked166 MB0.84190.67541.43
Binary, top 200 re-ranked166 MB0.90110.69112.66
Binary, top 400 re-ranked166 MB0.94220.70354.76

halfvec against full precision has no interval; we read the gap as no difference. The re-rank reads the full vectors, so they stay in the 782 MB table. Katz's quantization post (2024-04-09, a pre-release of pgvector 0.7.0, results at 128 to 1,536 dimensions) found the same halfvec result and that binary quantization "works better for datasets that produce larger vectors"; at 384 dimensions, 400 candidates were needed to pass full-precision recall at ef_search 40. The pgvector changelog lists a fix in 0.8.3 for a "performance regression with Hamming distance and Jaccard distance with Postgres 18", so 0.8.7 includes it.

Verdict: use halfvec for the index; use binary only with a re-rank deep enough to check on your data.

Changing the embedding model: re-embedding time and index builds

Re-embedding cost far more time than re-indexing: 500,000 Natural Questions passages took 810 s to embed with bge-small (618 passages/s, about 66,953 tokens/s), and the HNSW build took 50.0 s. Scaling the 810 s linearly gives about 1,620 s (27 minutes) for 1,000,000 chunks and 4.5 hours for 10,000,000 on this laptop GPU; that's arithmetic, not a measurement.

Documents per second, measured once per model, Apple M1 Max GPU (MPS); the SciFact and SQuAD columns were measured by part A run 1, whose caches part A run 2 reused:

ModelSciFactNQ sampleSQuAD paragraphs
all-MiniLM-L6-v25441,023879
bge-small-en-v1.5175533395
bge-base-en-v1.574195145
gte-modernbert-base4311195

gte-modernbert-base embedded at roughly a quarter of bge-small's rate on SciFact and SQuAD and about a fifth on Natural Questions. In part B, bge-small's rate in tokens per second rose to 142,666 for 1,024-token chunks, most likely because it read only the first 512 of each, the truncation above. For the dollar side, see what the model bill and re-embedding cost.

Re-indexing without downtime

Store a model version beside each vector, embed into a new column or table, build the new index beside the old one, then switch reads and drop the old index. In production, build it with CREATE INDEX CONCURRENTLY; our builds were plain CREATE INDEX and weren't timed concurrently. The failure mode to plan for is in building the new index without blocking writes.

Where the milliseconds go in a production RAG query

Without an LLM, a cross-encoder reranker was the slowest stage by far: 36.97 to 70.27 ms median for 20 candidates with ms-marco-MiniLM-L6-v2 (latency pass 1), against about 9 ms to embed the query with bge-small-en-v1.5, while BM25, exact search and RRF fusion each took under 3 ms.

Apple M1 Max, MPS, two latency passes, 100 queries after 5 warm-ups, one query at a time: pass 1 median (pass 1 p95); pass 2 median. Search here is exact, in numpy.

StageSciFactNQ sampleSQuAD
BM25, top 1000.12 (0.18); 0.120.42 (0.63); 0.420.20 (0.30); 0.20
Embed query, bge-small9.52 (13.58); 8.668.69 (10.31); 8.598.70 (9.18); 8.70
Embed query, gte-modernbert-base14.26 (17.79); 13.5212.48 (13.76); 12.4212.95 (14.35); 12.76
Exact search, bge-small vectors0.11 (0.12); 0.112.54 (2.72); 2.570.69 (0.77); 0.71
RRF fusion of two top-100 lists0.05 (0.05); 0.050.05 (0.05); 0.050.05 (0.05); 0.06
MiniLM cross-encoder, 20 candidates70.27 (79.57); 64.0336.97 (58.18); 37.6141.23 (57.59); 40.88

For pgvector, part C (one run) on 500,000 vectors: HNSW at ef_search 40 had a median of 1.96 ms client and 0.69 ms server unfiltered; a 1% tenant with relaxed_order took 28.76 ms client and 25.73 ms server. On the Natural Questions sample, query embedding, BM25, exact search, fusion and a 20-candidate rerank add up to 48.67 ms in pass 1 and 49.24 ms in pass 2: sums of medians, not a measured end-to-end median.

How to evaluate retrieval in production

Evaluate retrieval on its own, with labels, before judging answers. Build a labelled set from real user queries with the passages that answer them, and score nDCG@10 and recall@k with paired intervals when comparing setups. After every index or setting change, check index recall against exact search on a sample. Log retrieved ids, scores, rows returned and index version per request, so a short result shows up as data. Generation evaluation is out of scope here.

Numbers you'll see quoted, and what they measured

NumberWhat the source measured
Anthropic's 67%A relative cut in top-20 retrieval failure, 5.7% to 1.9%, for contextual embeddings plus contextual BM25 plus reranking, with paid embedding models (Anthropic names Gemini and Voyage as most effective) and the Cohere reranker (Anthropic, 2024-09-19)
"can improve recall accuracy by 1% to 9% compared to vector search alone"Redis's post on scaling RAG (2026-01-21, updated 2026-11-13) cites arXiv 2410.20381 for it; that paper's 1% to 9% is the accuracy gain from a distribution alignment method inside a graph index for dense-sparse vectors, not hybrid against vector search
"+5% to +24%"Weaviate's Search Mode benchmark (2025-09-23): its Search Mode against Weaviate's own hybrid search, not hybrid against vector search

Fine-tuning vs RAG: what each one fixes

Fine-tuning changes how a model behaves and formats; retrieval supplies facts, keeps them current and can filter them by user. Ovadia et al. (arXiv 2312.05934, 2023) compared the two for knowledge injection and wrote that "unsupervised fine-tuning offers some improvement, RAG consistently outperforms it". RAFT (Zhang et al., arXiv 2403.10131, 2024) combines them, training a model to "ignore those documents that don't help in answering the question" and to cite the right passage. We measured neither. For what fine-tuning costs and when it's worth it, see our AI development cost guide.

Long context vs RAG

Anthropic's guidance is that a knowledge base under about 200,000 tokens can go straight into the prompt, and our assistant guide explains whether you need retrieval at all. Above that size, or when content changes daily, or when each user may see only some documents, retrieval still does a job a long prompt can't: it picks what this user may see from a corpus that doesn't fit. Long inputs have their own limit; Liu et al. ("Lost in the Middle", arXiv 2307.03172, TACL) found performance "significantly degrades when models must access relevant information in the middle of long contexts." We didn't measure long-context answering.

Agentic RAG and GraphRAG

We measured neither. Agentic RAG lets a model decide when and what to retrieve, which turns retrieved documents into input to a system that can act, so design it around the lethal trifecta. GraphRAG (Edge et al., 2024) uses an LLM to build an entity graph and community summaries for questions about a whole corpus, which plain passage retrieval handles poorly.

Retrieval failures that raise no error

  • A filtered HNSW query returns short. On pgvector 0.8.7 defaults, a tenant holding 1% of 500,000 vectors got a mean of 0.45 rows when it asked for 10, and the query succeeded.
  • Text past the embedder's window is ignored. With 1,024-token chunks, bge-small-en-v1.5 (512-token window) put the right chunk in its top 5 for 0.738 of SQuAD questions whose answer sat in the first 510 tokens and 0.395 of those whose answer sat later, and the embedding call succeeded.

Log rows returned per query and check chunk lengths against the model's window; neither failure shows up any other way.

What this benchmark doesn't prove

  • Retrieval only. No LLM generated or scored an answer, so nothing here measures answer accuracy, hallucination or faithfulness. Good retrieval is necessary for good answers, not sufficient.
  • One laptop. Apple M1 Max, models on its GPU (MPS), Postgres in Docker Desktop's VM with other idle development containers running alongside during part C. Not a server or a CUDA GPU; one connection, no concurrency, no load.
  • One run each of parts A, B and C, so no run-to-run ranges for quality or pgvector timings; two latency passes; embedding throughput was measured once.
  • Open models only, not presented as what Keplaris ships; hosted embedding and reranking APIs weren't measured.
  • Public datasets with their own biases. SciFact abstracts and Natural Questions passages are short; SQuAD questions were written by people reading the paragraph, which favours word overlap; SQuAD labels one paragraph per question and Natural Questions few (4,201 labels for 3,452 test queries), so other correct passages count as misses. Model training mixes may include these sets.
  • The Natural Questions samples keep every labelled passage (in 60,000 and in 500,000 of 2,681,468), so their nDCG@10 is not comparable with published scores.
  • Synthetic tenants, independent of content, with one skew pattern; real tenant data clusters by topic, which changes how a filtered HNSW scan behaves. Tenant context was set per session, not per transaction through a pooler.
  • Not tested: HNSW build settings other than defaults, IVFFlat, pgvectorscale, other vector databases, Postgres full-text search, learned sparse or late-interaction retrievers, query rewriting, HyDE and contextual chunk headers (they need an LLM).
  • No client, production or Keplaris product data. We run no RAG product of our own.

The full benchmark scripts and output

Put the seven files in one folder with a Python 3.12 virtual environment in .venv (pip install torch==2.15.0 sentence-transformers==6.3.0 transformers==5.23.0 numpy==2.5.5 bm25s==0.3.16 PyStemmer==3.1.0 psycopg==3.3.9 pgvector==0.5.1), download the models into models/ and the datasets into data/ (folder names are in rag_common.py and rag_bench_c_prep.py), then run:

.venv/bin/python -I rag_bench_a.py > run-a-2.txt 2> run-a-2.err
.venv/bin/python -I rag_bench_c_prep.py > run-c-prep.txt 2> run-c-prep.err
./run-c.sh 1
.venv/bin/python -I rag_bench_b.py > run-b-1.txt 2> run-b-1.err
.venv/bin/python -I rag_bench_latency.py > run-latency-1.txt 2> run-latency-1.err
.venv/bin/python -I rag_bench_latency.py > run-latency-2.txt 2> run-latency-2.err

run-c.sh starts pgvector/pgvector:0.8.7-pg18@sha256:38681dc6eaf55308e2a4c6de392d131652dcfac70e1263e35cd9b5ea3fe39782, runs part C, appends the exit code to runs.log and removes the container, so it needs Docker and nothing else on port 54334. The scripts carry smoke-test switches (RAG_MAXQ, RAG_B_SMOKE, RAG_C_KEY); the published runs used none, as the query counts in each output show. On your own corpus, replace the loaders with your documents, queries and labels; the metrics and timing code stay the same.

If you want this run on your own documents and queries, our automation and AI systems team builds and measures retrieval like this.

rag_common.py

Shared helpers: data loading, metrics, models, BM25 and RRF.

# rag_common.py: shared helpers for rag-bench (data loading, metrics, model loading).
# No network calls: models and datasets are read from local folders.
import json, math, os, platform, random, subprocess, time, glob, hashlib
import numpy as np

SEED = 20261116
HERE = os.path.dirname(os.path.abspath(__file__))
DATA = os.path.join(HERE, "data")
CACHE = os.path.join(DATA, "cache")
os.makedirs(CACHE, exist_ok=True)
os.environ.setdefault("HF_HOME", os.path.join(HERE, "models"))
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TOKENIZERS_PARALLELISM"] = "false"

BGE_PREFIX = "Represent this sentence for searching relevant passages: "
EMBEDDERS = {
    # name: (hub id, query prefix or None, encode batch size)
    "minilm": ("sentence-transformers/all-MiniLM-L6-v2", None, 128),
    "bge-small": ("BAAI/bge-small-en-v1.5", BGE_PREFIX, 128),
    "bge-base": ("BAAI/bge-base-en-v1.5", BGE_PREFIX, 64),
    "gte-mbert": ("Alibaba-NLP/gte-modernbert-base", None, 32),
}
RERANKERS = {
    "ce-minilm": ("cross-encoder/ms-marco-MiniLM-L6-v2", 128),
    "bge-rr-base": ("BAAI/bge-reranker-base", 64),
}


def log(*a):
    print(*a, flush=True)


def sh(cmd):
    try:
        return subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout.strip()
    except Exception:
        return "?"


def model_revision(hub_id):
    d = os.path.join(os.environ["HF_HOME"], "hub", "models--" + hub_id.replace("/", "--"), "snapshots")
    revs = sorted(os.listdir(d)) if os.path.isdir(d) else []
    return revs[0][:12] if revs else "?"


def sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for b in iter(lambda: f.read(1 << 20), b""):
            h.update(b)
    return h.hexdigest()[:16]


def print_env(title):
    import torch, sentence_transformers, transformers
    log(f"# {title}")
    log(f"date (UTC): {time.strftime('%Y-%m-%d', time.gmtime())}")
    log(f"machine: {sh('sysctl -n machdep.cpu.brand_string')}, {int(sh('sysctl -n hw.memsize')) // 2**30} GB, macOS {sh('sw_vers -productVersion')}")
    log(f"python {platform.python_version()}, torch {torch.__version__} (mps available: {torch.backends.mps.is_available()}), "
        f"sentence-transformers {sentence_transformers.__version__}, transformers {transformers.__version__}, numpy {np.__version__}")
    from importlib.metadata import version
    log(f"bm25s {version('bm25s')}, PyStemmer {version('PyStemmer')}, psycopg {version('psycopg')}, pgvector (Python client) {version('pgvector')}")
    log(f"seed: {SEED}")


def device():
    import torch
    return "mps" if torch.backends.mps.is_available() else "cpu"


# ---------- datasets ----------
def load_beir(name, folder=None):
    """Returns doc_ids, doc_texts, query_ids, query_texts, qrels {qid: {docid: rel}} for the test split."""
    root = os.path.join(DATA, f"dl-{folder or name}", name)
    doc_ids, doc_texts = [], []
    with open(os.path.join(root, "corpus.jsonl")) as f:
        for line in f:
            d = json.loads(line)
            doc_ids.append(d["_id"])
            doc_texts.append((d.get("title", "") + " " + d["text"]).strip())
    qtext = {}
    with open(os.path.join(root, "queries.jsonl")) as f:
        for line in f:
            d = json.loads(line)
            qtext[d["_id"]] = d["text"]
    qrels = {}
    with open(os.path.join(root, "qrels", "test.tsv")) as f:
        next(f)
        for line in f:
            q, d, s = line.rstrip("\n").split("\t")
            if int(s) > 0:
                qrels.setdefault(q, {})[d] = int(s)
    qids = sorted(qrels)
    return doc_ids, doc_texts, qids, [qtext[q] for q in qids], qrels


def load_squad():
    """SQuAD v1.1 train+dev articles as the corpus; dev questions as queries.
    Returns articles [{title, paras}], questions [{id, q, art, para, answers[(start, text)]}]."""
    arts, questions = [], []
    for split in ("train", "dev"):
        with open(os.path.join(DATA, "dl-squad", f"{split}-v1.1.json")) as f:
            d = json.load(f)
        for a in d["data"]:
            ai = len(arts)
            arts.append({"title": a["title"].replace("_", " "), "paras": [p["context"] for p in a["paragraphs"]], "split": split})
            if split == "dev":
                for pi, p in enumerate(a["paragraphs"]):
                    for qa in p["qas"]:
                        ans = sorted({(x["answer_start"], x["text"]) for x in qa["answers"]})
                        questions.append({"id": qa["id"], "q": qa["question"], "art": ai, "para": pi, "answers": ans})
    return arts, questions


def squad_sample(questions, n):
    rng = random.Random(SEED)
    idx = list(range(len(questions)))
    rng.shuffle(idx)
    return [questions[i] for i in idx[:n]]


def squad_as_retrieval(n_queries=1000):
    """SQuAD paragraphs as a retrieval set: one relevant paragraph per question (the one it was written on)."""
    arts, questions = load_squad()
    doc_ids, doc_texts = [], []
    for ai, a in enumerate(arts):
        for pi, p in enumerate(a["paras"]):
            doc_ids.append(f"{ai}:{pi}")
            doc_texts.append(a["title"] + " " + p)
    qs = squad_sample(questions, n_queries)
    qids = [q["id"] for q in qs]
    qrels = {q["id"]: {f"{q['art']}:{q['para']}": 1} for q in qs}
    return doc_ids, doc_texts, qids, [q["q"] for q in qs], qrels


def nq_as_retrieval(n_docs=60000, n_queries=600):
    """A Natural Questions sample: every passage labelled for any test query, plus seeded random passages up to n_docs;
    n_queries seeded test queries. Smaller than the full 2.68M-passage corpus, so scores are not comparable with published ones."""
    root = os.path.join(DATA, "dl-nq", "nq")
    qrels = {}
    with open(os.path.join(root, "qrels", "test.tsv")) as f:
        next(f)
        for line in f:
            q, d, s = line.rstrip("\n").split("\t")
            if int(s) > 0:
                qrels.setdefault(q, {})[d] = int(s)
    relevant = {d for v in qrels.values() for d in v}
    ids, texts = [], []
    with open(os.path.join(root, "corpus.jsonl")) as f:
        for line in f:
            d = json.loads(line)
            ids.append(d["_id"])
            texts.append((d.get("title", "") + " " + d["text"]).strip())
    rel_idx = [i for i, d in enumerate(ids) if d in relevant]
    rest = [i for i, d in enumerate(ids) if d not in relevant]
    rng = random.Random(SEED)
    keep = sorted(rel_idx + rng.sample(rest, n_docs - len(rel_idx)))
    qtext = {}
    with open(os.path.join(root, "queries.jsonl")) as f:
        for line in f:
            d = json.loads(line)
            qtext[d["_id"]] = d["text"]
    qids = sorted(random.Random(SEED + 1).sample(sorted(qrels), n_queries))
    return [ids[i] for i in keep], [texts[i] for i in keep], qids, [qtext[q] for q in qids], {q: qrels[q] for q in qids}


# ---------- metrics ----------
def per_query_metrics(ranked, rels, ks=(10, 100)):
    """ranked: list of doc ids best first; rels: {docid: gain>0}."""
    out = {}
    gains = [rels.get(d, 0) for d in ranked[:10]]
    dcg = sum(g / math.log2(i + 2) for i, g in enumerate(gains))
    ideal = sorted(rels.values(), reverse=True)[:10]
    idcg = sum(g / math.log2(i + 2) for i, g in enumerate(ideal))
    out["ndcg10"] = dcg / idcg if idcg else 0.0
    rr = 0.0
    for i, d in enumerate(ranked[:10]):
        if d in rels:
            rr = 1.0 / (i + 1)
            break
    out["mrr10"] = rr
    for k in ks:
        out[f"recall{k}"] = len(set(ranked[:k]) & set(rels)) / len(rels)
    return out


def evaluate(run, qids, qrels, ks=(10, 100)):
    """run: {qid: [docids]}. Returns means and the per-query nDCG@10 vector (same order as qids)."""
    rows = [per_query_metrics(run[q], qrels[q], ks) for q in qids]
    means = {k: float(np.mean([r[k] for r in rows])) for k in rows[0]}
    return means, np.array([r["ndcg10"] for r in rows])


def boot_ci(vec, n=10000):
    rng = np.random.default_rng(SEED)
    idx = rng.integers(0, len(vec), size=(n, len(vec)))
    m = vec[idx].mean(axis=1)
    return float(np.percentile(m, 2.5)), float(np.percentile(m, 97.5))


def pct(xs, p):
    """nearest-rank percentile"""
    s = sorted(xs)
    return s[min(len(s) - 1, max(0, math.ceil(p / 100 * len(s)) - 1))]


def med(xs):
    return float(np.median(xs))


# ---------- models ----------
_models = {}


def embedder(name):
    from sentence_transformers import SentenceTransformer
    if name not in _models:
        _models[name] = SentenceTransformer(EMBEDDERS[name][0], device=device())
    return _models[name]


def reranker(name):
    from sentence_transformers import CrossEncoder
    if name not in _models:
        _models[name] = CrossEncoder(RERANKERS[name][0], device=device(), max_length=512)
    return _models[name]


def drop_model(name):
    import torch, gc
    _models.pop(name, None)
    gc.collect()
    if torch.backends.mps.is_available():
        torch.mps.empty_cache()


def embed_docs(name, texts, cache_key=None):
    """Returns (float32 normalized matrix, seconds or None if read from cache)."""
    path = os.path.join(CACHE, f"{cache_key}__{name}.npy") if cache_key else None
    if path and os.path.exists(path):
        log(f"  (reusing {os.path.basename(path)}, written {time.strftime('%Y-%m-%d %H:%M', time.localtime(os.path.getmtime(path)))} local time; the embedding time below is from that pass)")
        return np.load(path), None
    m = embedder(name)
    t = time.perf_counter()
    e = m.encode(texts, batch_size=EMBEDDERS[name][2], normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=False)
    secs = time.perf_counter() - t
    e = e.astype(np.float32)
    if path:
        np.save(path, e)
        with open(path + ".secs", "w") as f:
            f.write(str(secs))
    return e, secs


def embed_queries(name, texts, prefix=True):
    m = embedder(name)
    p = EMBEDDERS[name][1] if prefix else None
    return m.encode(texts, batch_size=64, normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=False, prompt=p).astype(np.float32)


def top_k(D, Q, k=100, block=256):
    """Exact cosine top-k (vectors are normalized). Returns index matrix [nq, k], best first."""
    out = np.empty((Q.shape[0], k), dtype=np.int64)
    for s in range(0, Q.shape[0], block):
        sims = Q[s:s + block] @ D.T
        part = np.argpartition(-sims, k - 1, axis=1)[:, :k]
        ps = np.take_along_axis(sims, part, axis=1)
        order = np.argsort(-ps, axis=1, kind="stable")
        out[s:s + block] = np.take_along_axis(part, order, axis=1)
    return out


def rrf(lists, k=60, depth=100):
    """Reciprocal rank fusion of ranked index lists."""
    score = {}
    for lst in lists:
        for r, d in enumerate(lst[:depth]):
            score[d] = score.get(d, 0.0) + 1.0 / (k + r + 1)
    return [d for d, _ in sorted(score.items(), key=lambda x: (-x[1], x[0]))]


class BM25:
    def __init__(self, texts):
        import bm25s, Stemmer
        self.bm25s = bm25s
        self.stemmer = Stemmer.Stemmer("english")
        toks = bm25s.tokenize(texts, stopwords="en", stemmer=self.stemmer, show_progress=False)
        self.r = bm25s.BM25()
        self.r.index(toks, show_progress=False)
        self.n = len(texts)

    def params(self):
        return f"bm25s method={self.r.method} k1={self.r.k1} b={self.r.b}, English stopwords, Snowball English stemmer"

    def search(self, queries, k=100):
        qt = self.bm25s.tokenize(queries, stopwords="en", stemmer=self.stemmer, show_progress=False)
        res, _ = self.r.retrieve(qt, k=min(k, self.n), show_progress=False)
        return np.asarray(res)

rag_bench_a.py

Part A: BM25, dense, hybrid and reranking.

# rag_bench_a.py: retrieval quality and latency of BM25, dense, hybrid (RRF) and cross-encoder reranking
# on three public corpora, all local, no paid API.
#   python -I rag_bench_a.py            (datasets in ./data, models in ./models; see rag_common.py)
import os, sys, time, random
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from rag_common import *

DATASETS = ["scifact", "nq", "squad"]
DENSE = ["minilm", "bge-small", "bge-base", "gte-mbert"]
DEPTHS = [10, 20, 50]
LAT_N = 100  # queries timed per dataset for the latency rows


def load(name):
    if name == "squad":
        return squad_as_retrieval(600)
    if name == "nq":
        return nq_as_retrieval(60000, 600)
    return load_beir(name)


def fmt(m, ci):
    return f"{m['ndcg10']:.3f} [{ci[0]:.3f}, {ci[1]:.3f}] | {m['mrr10']:.3f} | {m['recall10']:.3f} | {m['recall100']:.3f}"


def main():
    print_env("rag-bench part A: BM25, dense, hybrid and reranking")
    for n, (hid, pre, _) in EMBEDDERS.items():
        log(f"embedder {n}: {hid} @ {model_revision(hid)}, query prefix: {'yes' if pre else 'none'}")
    for n, (hid, _) in RERANKERS.items():
        log(f"reranker {n}: {hid} @ {model_revision(hid)}, max_length 512")
    log("metrics: nDCG@10 (linear gain) with a 95% bootstrap interval over queries (10,000 resamples), MRR@10, recall@10, recall@100")
    log("hybrid: reciprocal rank fusion, k=60, over the top 100 of BM25 and the top 100 of the dense model")
    log("rerank: the cross-encoder rescored the top N of the named first stage; rows show N")
    log("datasets: scifact = BEIR SciFact, full corpus and test queries; nq = a Natural Questions (BEIR) sample of 60,000 passages (every passage labelled for any test query, "
        "plus seeded random passages) with 600 seeded test queries; squad = SQuAD v1.1 paragraphs from all 490 articles with 600 seeded dev questions")
    log("latency: one query at a time on this machine, median and p95 (nearest rank) over 100 seeded queries after 5 warm-ups")

    for ds in DATASETS:
        doc_ids, docs, qids, queries, qrels = load(ds)
        if os.environ.get("RAG_MAXQ"):  # smoke test only
            qids, queries = qids[:int(os.environ["RAG_MAXQ"])], queries[:int(os.environ["RAG_MAXQ"])]
        ids = np.array(doc_ids)
        log(f"\n## {ds}: {len(docs):,} documents, {len(queries):,} test queries, {sum(len(v) for v in qrels.values()):,} relevance labels")
        runs, lat = {}, {}

        t = time.perf_counter()
        bm = BM25(docs)
        log(f"bm25 index build: {time.perf_counter() - t:.1f} s ({bm.params()})")
        bm_idx = bm.search(queries, 100)
        runs["bm25"] = bm_idx

        dense_idx = {}
        for name in DENSE:
            D, secs = embed_docs(name, docs, cache_key=ds)
            if secs is None:
                secs = float(open(os.path.join(CACHE, f"{ds}__{name}.npy.secs")).read())
            log(f"embed {name}: {len(docs):,} docs in {secs:.1f} s = {len(docs) / secs:,.0f} docs/s, dim {D.shape[1]}, {D.nbytes / 2**20:.0f} MiB as float32")
            Q = embed_queries(name, queries, prefix=True)
            dense_idx[name] = top_k(D, Q, 100)
            runs[f"dense {name}"] = dense_idx[name]
            if EMBEDDERS[name][1]:
                runs[f"dense {name}, query prefix left out"] = top_k(D, embed_queries(name, queries, prefix=False), 100)
            runs[f"hybrid bm25 + {name}"] = np.array([rrf([list(bm_idx[i]), list(dense_idx[name][i])])[:100] for i in range(len(queries))], dtype=object)
            # latency: query embedding and exact search
            rng = random.Random(SEED)
            sample = rng.sample(range(len(queries)), min(LAT_N, len(queries)))
            m = embedder(name)
            te, ts = [], []
            for j, qi in enumerate(sample[:5] + sample):
                t0 = time.perf_counter()
                qv = m.encode([queries[qi]], normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=False, prompt=EMBEDDERS[name][1]).astype(np.float32)
                t1 = time.perf_counter()
                sims = (qv @ D.T)[0]
                top = np.argpartition(-sims, 99)[:100]
                top = top[np.argsort(-sims[top])]
                t2 = time.perf_counter()
                if j >= 5:
                    te.append((t1 - t0) * 1000)
                    ts.append((t2 - t1) * 1000)
            lat[f"embed query, {name}"] = te
            lat[f"exact vector search in numpy, {name} ({len(docs):,} x {D.shape[1]})"] = ts
            if name != "bge-small":
                del D
            else:
                D_small = D
            drop_model(name)

        # BM25 latency
        rng = random.Random(SEED)
        sample = rng.sample(range(len(queries)), min(LAT_N, len(queries)))
        tb = []
        for j, qi in enumerate(sample[:5] + sample):
            t0 = time.perf_counter()
            bm.search([queries[qi]], 100)
            if j >= 5:
                tb.append((time.perf_counter() - t0) * 1000)
        lat["bm25 search"] = tb

        # reranking: score the union of each first stage's top 50 once per reranker
        stages = {"bm25": runs["bm25"], "dense bge-small": runs["dense bge-small"], "hybrid bm25 + bge-small": runs["hybrid bm25 + bge-small"]}
        cand = [sorted({int(d) for s in stages.values() for d in list(s[i])[:50]}) for i in range(len(queries))]
        n_pairs = sum(len(c) for c in cand)
        for rr in RERANKERS:
            ce = reranker(rr)
            pairs = [(queries[i], docs[d]) for i in range(len(queries)) for d in cand[i]]
            t = time.perf_counter()
            scores = ce.predict(pairs, batch_size=RERANKERS[rr][1], show_progress_bar=False)
            secs = time.perf_counter() - t
            log(f"rerank {rr}: {n_pairs:,} pairs in {secs:.1f} s = {n_pairs / secs:,.0f} pairs/s (batched)")
            sc, p = [], 0
            for i in range(len(queries)):
                sc.append({d: float(scores[p + j]) for j, d in enumerate(cand[i])})
                p += len(cand[i])
            for sname, s in stages.items():
                for depth in DEPTHS:
                    if depth != 50 and sname != "hybrid bm25 + bge-small":
                        continue
                    out = []
                    for i in range(len(queries)):
                        head = [int(d) for d in list(s[i])[:depth]]
                        head.sort(key=lambda d: (-sc[i][d], d))
                        out.append(head + [int(d) for d in list(s[i])[depth:100]])
                    runs[f"{sname} -> rerank top {depth}, {rr}"] = np.array(out, dtype=object)
            # latency per depth: one query, N pairs in one batch
            rng = random.Random(SEED)
            sample = rng.sample(range(len(queries)), min(LAT_N, len(queries)))
            hyb = runs["hybrid bm25 + bge-small"]
            for depth in DEPTHS:
                tr = []
                for j, qi in enumerate(sample[:5] + sample):
                    pr = [(queries[qi], docs[int(d)]) for d in list(hyb[qi])[:depth]]
                    t0 = time.perf_counter()
                    ce.predict(pr, batch_size=depth, show_progress_bar=False)
                    if j >= 5:
                        tr.append((time.perf_counter() - t0) * 1000)
                lat[f"rerank {depth} candidates, {rr}"] = tr
            drop_model(rr)

        # RRF latency (pure python)
        tf = []
        for qi in sample:
            t0 = time.perf_counter()
            rrf([list(bm_idx[qi]), list(dense_idx["bge-small"][qi])])
            tf.append((time.perf_counter() - t0) * 1000)
        lat["rrf fusion of two top-100 lists"] = tf

        # ---- tables ----
        log(f"\n### {ds}: quality")
        log("pipeline | nDCG@10 [95% CI] | MRR@10 | recall@10 | recall@100")
        per_q = {}
        for name, r in runs.items():
            run = {q: [ids[int(d)] for d in list(r[i])] for i, q in enumerate(qids)}
            m, v = evaluate(run, qids, qrels)
            per_q[name] = v
            log(f"{name} | {fmt(m, boot_ci(v))}")
        log(f"\n### {ds}: paired differences in nDCG@10 (B minus A), 95% bootstrap interval over the same queries")
        pairs_cmp = [("bm25", "dense bge-small"), ("dense bge-small", "hybrid bm25 + bge-small"), ("dense bge-base", "hybrid bm25 + bge-base"),
                     ("dense minilm", "dense bge-small"), ("dense bge-small", "dense bge-base"), ("dense bge-base", "dense gte-mbert"),
                     ("dense bge-small, query prefix left out", "dense bge-small"), ("dense bge-base, query prefix left out", "dense bge-base"),
                     ("hybrid bm25 + bge-small", "hybrid bm25 + bge-small -> rerank top 50, ce-minilm"),
                     ("hybrid bm25 + bge-small", "hybrid bm25 + bge-small -> rerank top 50, bge-rr-base"),
                     ("bm25", "bm25 -> rerank top 50, bge-rr-base"),
                     ("dense bge-small -> rerank top 50, bge-rr-base", "hybrid bm25 + bge-small -> rerank top 50, bge-rr-base"),
                     ("hybrid bm25 + bge-small -> rerank top 10, bge-rr-base", "hybrid bm25 + bge-small -> rerank top 50, bge-rr-base")]
        for a, b in pairs_cmp:
            d = per_q[b] - per_q[a]
            lo, hi = boot_ci(d)
            verdict = "B higher" if lo > 0 else ("A higher" if hi < 0 else "interval includes 0")
            log(f"A = {a} | B = {b} | {d.mean():+.3f} [{lo:+.3f}, {hi:+.3f}] | {verdict}")
        log(f"\n### {ds}: latency per query in ms (median | p95)")
        for name, xs in lat.items():
            log(f"{name} | {med(xs):.2f} | {pct(xs, 95):.2f}")
        del D_small


if __name__ == "__main__":
    main()

rag_bench_b.py

Part B: chunk size at a fixed context budget.

# rag_bench_b.py: chunk size and overlap at a fixed context budget, on SQuAD v1.1 articles (long documents with labelled answer spans).
#   python -I rag_bench_b.py
import os, sys, time, bisect
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from rag_common import *

N_Q = 2000
BUDGETS = [512, 1024, 2048]
FIXED = [(64, 0), (64, 13), (128, 0), (128, 26), (256, 0), (256, 51), (512, 0), (512, 102), (1024, 0), (1024, 205)]
LONG_CTX = {"256 tokens, no overlap", "512 tokens, no overlap", "1024 tokens, no overlap"}  # configs also embedded with gte-modernbert-base


if os.environ.get("RAG_B_SMOKE"):  # smoke test only
    N_Q, FIXED, LONG_CTX = 200, [(256, 0), (1024, 0)], {"1024 tokens, no overlap"}


def main():
    print_env("rag-bench part B: chunk size at a fixed context budget")
    arts, questions = load_squad()
    qs = squad_sample(questions, N_Q)
    tok = embedder("bge-small").tokenizer
    log(f"corpus: SQuAD v1.1, {len(arts)} Wikipedia articles ({sum(1 for a in arts if a['split'] == 'train')} train + {sum(1 for a in arts if a['split'] == 'dev')} dev), "
        f"{sum(len(a['paras']) for a in arts):,} paragraphs; queries: {N_Q:,} seeded dev questions of {len(questions):,}")
    log("tokens are counted with the bge-small-en-v1.5 WordPiece tokenizer; an LLM's tokenizer will count differently")

    # full article text, paragraph offsets, token offsets
    texts, para_off, tok_start, tok_end = [], [], [], []
    for a in arts:
        offs, pos = [], 0
        for p in a["paras"]:
            offs.append(pos)
            pos += len(p) + 2
        text = "\n\n".join(a["paras"])
        enc = tok(text, add_special_tokens=False, return_offsets_mapping=True, truncation=False, verbose=False)
        texts.append(text)
        para_off.append(offs)
        tok_start.append([o[0] for o in enc["offset_mapping"]])
        tok_end.append([o[1] for o in enc["offset_mapping"]])
    n_tok = sum(len(t) for t in tok_start)
    log(f"corpus size: {n_tok:,} tokens, {sum(len(t) for t in texts):,} characters; article length median {int(np.median([len(t) for t in tok_start])):,} tokens, "
        f"paragraph length median {int(np.median([len(tok(p, add_special_tokens=False, verbose=False)['input_ids']) for a in arts[:60] for p in a['paras']]))} tokens (first 60 articles)")

    gold = []  # per question: (article, [(char start, char end)])
    bad = 0
    for q in qs:
        spans = []
        for st, txt in q["answers"]:
            s = para_off[q["art"]][q["para"]] + st
            if texts[q["art"]][s:s + len(txt)] != txt:
                bad += 1
                continue
            spans.append((s, s + len(txt)))
        gold.append((q["art"], spans))
    log(f"answer spans located in the joined article text: {sum(len(g[1]) for g in gold):,} spans, {bad} mismatches")
    queries = [q["q"] for q in qs]

    title_tok = [len(tok(a["title"] + ". ", add_special_tokens=False, verbose=False)["input_ids"]) for a in arts]

    def chunk_fixed(size, overlap, reserve_title=False):
        """Token windows. With reserve_title the window shrinks by the title's length, so title + text still fits in `size`."""
        out = []
        for ai in range(len(arts)):
            n = len(tok_start[ai])
            sz = size - (title_tok[ai] if reserve_title else 0)
            step = sz - overlap
            s = 0
            while s < n:
                e = min(n, s + sz)
                out.append((ai, tok_start[ai][s], tok_end[ai][e - 1], e - s, s))
                if e == n:
                    break
                s += step
        return out

    def chunk_paragraphs():
        out = []
        for ai, a in enumerate(arts):
            for pi, p in enumerate(a["paras"]):
                cs = para_off[ai][pi]
                ce = cs + len(p)
                ts = bisect.bisect_left(tok_start[ai], cs)
                te = bisect.bisect_right(tok_start[ai], ce - 1)
                out.append((ai, cs, ce, te - ts, ts))
        return out

    configs = [(f"{s} tokens, {'no overlap' if o == 0 else f'{o} overlap (20%)'}", chunk_fixed(s, o), False) for s, o in FIXED]
    configs.append(("paragraphs as written (the article's own boundaries)", chunk_paragraphs(), False))
    configs.append(("paragraphs as written, article title prepended", chunk_paragraphs(), True))
    configs.append(("256 tokens including a prepended article title, no overlap", chunk_fixed(256, 0, reserve_title=True), True))
    if os.environ.get("RAG_B_SMOKE"):
        configs = configs[:2] + configs[-1:]

    Qv = {"bge-small": embed_queries("bge-small", queries, prefix=True)}
    results = {}
    trunc_note = []
    for label, chunks, with_title in configs:
        ctext = [(arts[c[0]]["title"] + ". " if with_title else "") + texts[c[0]][c[1]:c[2]] for c in chunks]
        ntok = np.array([c[3] + (title_tok[c[0]] if with_title else 0) for c in chunks])
        by_art = {}
        for ci, c in enumerate(chunks):
            by_art.setdefault(c[0], []).append(ci)
        gold_chunks = []
        for ai, spans in gold:
            g = {ci for ci in by_art[ai] for (s, e) in spans if chunks[ci][1] <= s and e <= chunks[ci][2]}
            gold_chunks.append(g)
        answerable = np.mean([1.0 if g else 0.0 for g in gold_chunks])

        rankings = {}
        t = time.perf_counter()
        D = embedder("bge-small").encode(ctext, batch_size=128, normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=False).astype(np.float32)
        secs = time.perf_counter() - t
        rankings["dense bge-small"] = top_k(D, Qv["bge-small"], 100)
        bm = BM25(ctext)
        bidx = bm.search(queries, 100)
        rankings["bm25"] = bidx
        rankings["hybrid bm25 + bge-small"] = [rrf([list(bidx[i]), list(rankings["dense bge-small"][i])])[:100] for i in range(len(queries))]
        del D
        if label in LONG_CTX:
            drop_model("bge-small")
            if "gte-mbert" not in Qv:
                Qv["gte-mbert"] = embed_queries("gte-mbert", queries, prefix=True)
            D2 = embedder("gte-mbert").encode(ctext, batch_size=16, normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=False).astype(np.float32)
            rankings["dense gte-modernbert-base (8,192-token window)"] = top_k(D2, Qv["gte-mbert"], 100)
            del D2
            drop_model("gte-mbert")
        log(f"chunked: {label}: {len(chunks):,} chunks, mean {ntok.mean():.0f} tokens, embedded with bge-small in {secs:.1f} s = {len(chunks) / secs:,.0f} chunks/s, {ntok.sum() / secs:,.0f} tokens/s")

        for rname, rk in rankings.items():
            hit1, hit5, hb = [], [], {b: [] for b in BUDGETS}
            for qi in range(len(queries)):
                r = [int(x) for x in rk[qi]]
                g = gold_chunks[qi]
                hit1.append(1.0 if r[0] in g else 0.0)
                hit5.append(1.0 if any(x in g for x in r[:5]) else 0.0)
                for b in BUDGETS:
                    cum, ok = 0, 0.0
                    for x in r:
                        cum += ntok[x]
                        if cum > b:
                            break
                        if x in g:
                            ok = 1.0
                            break
                    hb[b].append(ok)
            results.setdefault(rname, []).append((label, len(chunks), ntok.mean(), answerable, np.mean(hit1), np.mean(hit5), {b: np.array(v) for b, v in hb.items()}))
            # truncation: for chunks longer than the embedder's 512-token window, split questions by where the answer sits
            if label == "1024 tokens, no overlap":
                seen, tail = [], []
                for qi in range(len(queries)):
                    g = gold_chunks[qi]
                    if not g:
                        continue
                    ci = min(g)
                    ai, spans = gold[qi]
                    e_tok = max(bisect.bisect_left(tok_start[ai], e) for (s, e) in spans if chunks[ci][1] <= s and e <= chunks[ci][2])
                    (seen if e_tok - chunks[ci][4] <= 510 else tail).append(1.0 if any(x in g for x in [int(v) for v in rk[qi]][:5]) else 0.0)
                trunc_note.append(f"{rname} | answer in the first 510 tokens of its chunk: {np.mean(seen):.3f} (n={len(seen)}) | answer after token 510: {np.mean(tail):.3f} (n={len(tail)})")

    for rname, rows in results.items():
        log(f"\n## Retriever: {rname}")
        log("chunking | chunks | mean tokens | answer span inside one chunk | hit@1 chunk | hit@5 chunks | answer in a 512-token budget | in 1,024 tokens [95% CI] | in 2,048 tokens")
        for label, n, mt, ans, h1, h5, hb in rows:
            lo, hi = boot_ci(hb[1024])
            size = int(label.split(" ")[0]) if label[0].isdigit() else 0
            b512 = "n/a (a full chunk does not fit)" if size > 512 else f"{hb[512].mean():.3f}"
            b1024 = "n/a (a full chunk does not fit)" if size > 1024 else f"{hb[1024].mean():.3f} [{lo:.3f}, {hi:.3f}]"
            log(f"{label} | {n:,} | {mt:.0f} | {ans:.3f} | {h1:.3f} | {h5:.3f} | {b512} | {b1024} | {hb[2048].mean():.3f}")
        base = next((r for r in rows if r[0] == "256 tokens, no overlap"), None)
        if base is None:
            continue
        log("paired difference in the 1,024-token budget hit rate against 256 tokens, no overlap (95% bootstrap interval):")
        for label, n, mt, ans, h1, h5, hb in rows:
            if label == base[0]:
                continue
            d = hb[1024] - base[6][1024]
            lo, hi = boot_ci(d)
            log(f"  {label} | {d.mean():+.3f} [{lo:+.3f}, {hi:+.3f}] | {'higher' if lo > 0 else ('lower' if hi < 0 else 'interval includes 0')}")
    log("\n## Truncation: 1,024-token chunks, hit@5 by where the answer sits inside its chunk")
    log("bge-small-en-v1.5 reads at most 512 tokens (510 of text); gte-modernbert-base reads 8,192")
    for line in trunc_note:
        log(line)


if __name__ == "__main__":
    main()

rag_bench_c_prep.py

Part C prep: the Natural Questions sample and its embeddings.

# rag_bench_c_prep.py: build the 500,000-passage Natural Questions sample and embed it with bge-small-en-v1.5.
#   python -I rag_bench_c_prep.py
import os, sys, json, random, time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from rag_common import *

N = 500_000


def main():
    print_env("rag-bench part C prep: Natural Questions sample and embeddings")
    root = os.path.join(DATA, "dl-nq", "nq")
    qrels = {}
    with open(os.path.join(root, "qrels", "test.tsv")) as f:
        next(f)
        for line in f:
            q, d, s = line.rstrip("\n").split("\t")
            if int(s) > 0:
                qrels.setdefault(q, {})[d] = int(s)
    relevant = {d for v in qrels.values() for d in v}
    ids, texts = [], []
    with open(os.path.join(root, "corpus.jsonl")) as f:
        for line in f:
            d = json.loads(line)
            ids.append(d["_id"])
            texts.append((d.get("title", "") + " " + d["text"]).strip())
    log(f"nq corpus: {len(ids):,} passages, {len(qrels):,} test queries, {sum(len(v) for v in qrels.values()):,} relevance labels on {len(relevant):,} passages")
    rel_idx = [i for i, d in enumerate(ids) if d in relevant]
    rest = [i for i, d in enumerate(ids) if d not in relevant]
    rng = random.Random(SEED)
    keep = sorted(rel_idx + rng.sample(rest, N - len(rel_idx)))
    ids = [ids[i] for i in keep]
    texts = [texts[i] for i in keep]
    log(f"sample: every labelled passage ({len(rel_idx):,}) plus {N - len(rel_idx):,} seeded random passages = {len(ids):,}")
    tok = embedder("bge-small").tokenizer
    samp = random.Random(SEED).sample(range(len(texts)), 2000)
    lens = [len(tok(texts[i], add_special_tokens=True, truncation=True, max_length=512)["input_ids"]) for i in samp]
    log(f"passage length in bge-small tokens, 2,000 sampled: median {int(np.median(lens))}, mean {np.mean(lens):.0f}, max {max(lens)}")
    E, secs = embed_docs("bge-small", texts, cache_key="nq500k")
    if secs is None:
        secs = float(open(os.path.join(CACHE, "nq500k__bge-small.npy.secs")).read())
    log(f"embed bge-small: {len(texts):,} passages in {secs:.0f} s = {len(texts) / secs:,.0f} passages/s, about {np.mean(lens) * len(texts) / secs:,.0f} tokens/s, dim {E.shape[1]}, {E.nbytes / 2**20:.0f} MiB as float32")
    qtext = {}
    with open(os.path.join(root, "queries.jsonl")) as f:
        for line in f:
            d = json.loads(line)
            qtext[d["_id"]] = d["text"]
    qids = sorted(qrels)
    Q = embed_queries("bge-small", [qtext[q] for q in qids], prefix=True)
    np.save(os.path.join(CACHE, "nq500k__queries__bge-small.npy"), Q)
    with open(os.path.join(CACHE, "nq500k__meta.json"), "w") as f:
        json.dump({"ids": ids, "qids": qids, "qrels": qrels}, f)
    log(f"queries embedded: {Q.shape[0]:,}")


if __name__ == "__main__":
    main()

rag_bench_c.py

Part C: pgvector HNSW, tenant filters, row-level security and quantization.

# rag_bench_c.py: pgvector in a production shape: HNSW vs exact, tenant-filtered search, row-level security, quantization.
# Needs a throwaway Postgres with pgvector (see run-c.sh) and the embeddings from rag_bench_c_prep.py.
#   PGPORT=54334 PGPASSWORD=rag-throwaway python -I rag_bench_c.py
import os, sys, json, time, random
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import psycopg
from pgvector.psycopg import register_vector
from rag_common import *

PORT = int(os.environ.get("PGPORT", "54334"))
PW = os.environ.get("PGPASSWORD", "rag-throwaway")
APP_PW = "rag-bench-app"
LAT_N = 500      # queries timed per row
WARM = 50        # untimed warm-up queries per row
TENANTS = [(1, "whale", 0.40), (2, "mid", 0.05), (3, "small", 0.01), (4, "tiny", 0.001)]
DIM = 384
KEY = os.environ.get("RAG_C_KEY", "nq500k")  # smoke tests point this at a small set


def connect(user="postgres", pw=PW):
    c = psycopg.connect(host="127.0.0.1", port=PORT, user=user, password=pw, dbname="postgres", autocommit=True, prepare_threshold=None)
    if user == "postgres":
        c.execute("CREATE EXTENSION IF NOT EXISTS vector")
    register_vector(c)
    return c


def plan_chain(cur, sql, params):
    cur.execute("EXPLAIN (FORMAT JSON) " + sql, params)
    node = cur.fetchone()[0][0]["Plan"]
    parts = []
    while node:
        s = node["Node Type"]
        if node.get("Index Name"):
            s += " " + node["Index Name"]
        if node.get("Filter"):
            s += " +Filter"
        parts.append(s)
        kids = [k for k in node.get("Plans", []) if k.get("Parent Relationship") != "InitPlan"]
        node = kids[0] if kids else None
    return " > ".join(parts)


def run_queries(cur, sql, make_params, n_quality, timed):
    """Returns (ids per query for the first n_quality queries, client ms list, server ms list) ."""
    for i in range(WARM):
        cur.execute(sql, make_params(i))
        cur.fetchall()
    ids, client, server = [], [], []
    for i in range(n_quality):
        p = make_params(i)
        t0 = time.perf_counter()
        cur.execute(sql, p)
        rows = cur.fetchall()
        t1 = time.perf_counter()
        ids.append([r[0] for r in rows])
        if i < timed:
            client.append((t1 - t0) * 1000)
            cur.execute("EXPLAIN (ANALYZE, TIMING OFF, FORMAT JSON) " + sql, p)
            j = cur.fetchone()[0][0]
            server.append(j["Planning Time"] + j["Execution Time"])
    return ids, client, server


def recall_vs(ids, truth):
    return float(np.mean([len(set(a) & set(t)) / len(t) for a, t in zip(ids, truth)]))


def main():
    print_env("rag-bench part C: pgvector HNSW, tenant filters, row-level security and quantization")
    E = np.load(os.path.join(CACHE, f"{KEY}__bge-small.npy"))
    Q = np.load(os.path.join(CACHE, f"{KEY}__queries__bge-small.npy"))
    meta = json.load(open(os.path.join(CACHE, f"{KEY}__meta.json")))
    N = E.shape[0]
    idx_of = {d: i for i, d in enumerate(meta["ids"])}
    qrels = [{idx_of[d]: g for d, g in meta["qrels"][q].items()} for q in meta["qids"]]
    NQ = Q.shape[0]
    log(f"vectors: {N:,} x {E.shape[1]} (BAAI/bge-small-en-v1.5, normalized, float32); queries: {NQ:,} Natural Questions test queries")

    rng = np.random.default_rng(SEED)
    u = rng.random(N)
    tenant = (5 + (rng.random(N) * 996).astype(np.int32)).astype(np.int32)
    lo = 0.0
    for tid, _, share in TENANTS:
        tenant[(u >= lo) & (u < lo + share)] = tid
        lo += share
    log("synthetic tenant labels (seeded, independent of content): " + ", ".join(
        f"{name} = tenant {tid}: {int((tenant == tid).sum()):,} rows ({(tenant == tid).mean() * 100:.2f}%)" for tid, name, _ in TENANTS)
        + f"; the other {len(np.unique(tenant)) - 4} tenants hold {int((tenant > 4).sum()):,} rows")

    lat_q = random.Random(SEED).sample(range(NQ), LAT_N)   # queries used for timing and filtered tests
    order = lat_q + [i for i in range(NQ) if i not in set(lat_q)]  # timed queries first
    Qo = Q[order]
    qrels_o = [qrels[i] for i in order]

    t = time.perf_counter()
    truth = top_k(E, Qo, 10)
    log(f"exact top 10 in numpy for all {NQ:,} queries: {time.perf_counter() - t:.1f} s (ground truth for index recall)")
    truth_t = {}
    for tid, name, _ in TENANTS:
        rows = np.where(tenant == tid)[0]
        tk = top_k(E[rows], Qo[:LAT_N], 10)
        truth_t[tid] = rows[tk]

    def quality(ids):
        ms = [per_query_metrics(r, qr, ks=(10,)) for r, qr in zip(ids, qrels_o)]
        return float(np.mean([m["ndcg10"] for m in ms])), float(np.mean([m["recall10"] for m in ms]))

    nd, rc = quality([list(map(int, r)) for r in truth])
    log(f"exact search quality on the labels: nDCG@10 {nd:.4f}, recall@10 {rc:.4f}")

    # ---- load ----
    c = connect()
    cur = c.cursor()
    cur.execute("SELECT version()")
    log("server: " + cur.fetchone()[0])
    log("docker: " + sh("docker info --format '{{.ServerVersion}}, {{.NCPU}} CPUs, {{.MemTotal}} bytes of memory in the VM'"))
    log("image: " + sh("docker inspect --format '{{.Config.Image}}' ragbench"))
    cur.execute("SELECT extversion FROM pg_extension WHERE extname = 'vector'")
    log("pgvector: " + cur.fetchone()[0])
    for s in ["shared_buffers", "maintenance_work_mem", "max_parallel_maintenance_workers", "max_parallel_workers_per_gather", "work_mem", "jit", "hnsw.ef_search", "hnsw.iterative_scan", "hnsw.max_scan_tuples", "hnsw.scan_mem_multiplier"]:
        cur.execute(f"SHOW {s}")
        log(f"  {s} = {cur.fetchone()[0]}")
    cur.execute("DROP TABLE IF EXISTS passages")
    cur.execute(f"CREATE TABLE passages (id int PRIMARY KEY, tenant_id int NOT NULL, embedding vector({DIM}) NOT NULL)")
    t = time.perf_counter()
    with cur.copy("COPY passages (id, tenant_id, embedding) FROM STDIN WITH (FORMAT BINARY)") as cp:
        cp.set_types(["int4", "int4", "vector"])
        for i in range(N):
            cp.write_row((i, int(tenant[i]), E[i]))
    log(f"load: {N:,} rows by COPY in {time.perf_counter() - t:.1f} s")
    cur.execute("VACUUM (ANALYZE) passages")

    def size(rel):
        cur.execute("SELECT pg_size_pretty(pg_relation_size(%s::regclass)), pg_relation_size(%s::regclass)", (rel, rel))
        return cur.fetchone()

    def build(name, ddl):
        t = time.perf_counter()
        cur.execute(ddl)
        secs = time.perf_counter() - t
        log(f"build {name}: {secs:.1f} s, {size(name)[0]}")

    log(f"table passages: {size('passages')[0]} (heap only)")
    build("passages_hnsw", "CREATE INDEX passages_hnsw ON passages USING hnsw (embedding vector_cosine_ops)")
    log("  (pgvector defaults: m = 16, ef_construction = 64)")

    SQL = "SELECT id FROM passages ORDER BY embedding <=> %s LIMIT 10"
    SQLF = "SELECT id FROM passages WHERE tenant_id = %s ORDER BY embedding <=> %s LIMIT 10"

    def row(label, ids, client, server, truth_ids, plan, extra=""):
        got = [len(r) for r in ids[:LAT_N]] if extra == "f" else None
        base = f"{label} | {recall_vs(ids, truth_ids):.4f}"
        if extra == "f":
            base += f" | {np.mean(got):.2f} | {sum(1 for g in got if g < 10)}"
        else:
            nd, rc = quality(ids)
            base += f" | {nd:.4f} | {rc:.4f}"
        log(f"{base} | {med(client):.2f} | {pct(client, 95):.2f} | {med(server):.2f} | {plan}")

    # ---- unfiltered ----
    log(f"\n## Unfiltered search, {N:,} vectors: HNSW against exact")
    log(f"quality columns use all {NQ:,} queries; timing columns use {LAT_N} seeded queries after {WARM} warm-ups, one connection, no prepared statements")
    log("client ms = round trip seen by the Python client through Docker's port mapping; server ms = planning + execution time from EXPLAIN ANALYZE")
    log("setting | index recall@10 vs exact | nDCG@10 on labels | recall@10 on labels | client ms median | client ms p95 | server ms median | plan")
    cur.execute("SET enable_indexscan = off")
    ids, cl, sv = run_queries(cur, SQL, lambda i: (Qo[i],), 50, 50)
    log(f"exact scan in Postgres (index disabled), 50 queries | {recall_vs(ids, truth[:50]):.4f} | n/a | n/a | {med(cl):.2f} | {pct(cl, 95):.2f} | {med(sv):.2f} | {plan_chain(cur, SQL, (Qo[0],))}")
    cur.execute("RESET enable_indexscan")
    for ef in [10, 20, 40, 100, 200, 400]:
        cur.execute(f"SET hnsw.ef_search = {ef}")
        ids, cl, sv = run_queries(cur, SQL, lambda i: (Qo[i],), NQ, LAT_N)
        row(f"hnsw ef_search = {ef}{' (default)' if ef == 40 else ''}", ids, cl, sv, truth, plan_chain(cur, SQL, (Qo[0],)))
    cur.execute("RESET hnsw.ef_search")

    # ---- filtered ----
    def filtered_block(cur, title, sql_for, params_for, settings, pre=None):
        log(f"\n### {title}")
        log("setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan")
        for label, sets in settings:
            for s in sets:
                cur.execute(s)
            for tid, name, share in TENANTS:
                if pre:
                    pre(cur, tid)
                ids, cl, sv = run_queries(cur, sql_for(tid), lambda i, tid=tid: params_for(tid, i), LAT_N, LAT_N)
                row(f"{label} | {name} ({share * 100:g}%)", ids, cl, sv, truth_t[tid], plan_chain(cur, sql_for(tid), params_for(tid, 0)), extra="f")
            for s in ["RESET hnsw.ef_search", "RESET hnsw.iterative_scan", "RESET hnsw.max_scan_tuples"]:
                cur.execute(s)

    log(f"\n## Tenant-filtered search: WHERE tenant_id = $1 ORDER BY embedding <=> $2 LIMIT 10 ({LAT_N} queries per tenant)")
    base_settings = [
        ("iterative scan off, ef_search 40 (defaults)", []),
        ("iterative scan off, ef_search 200", ["SET hnsw.ef_search = 200"]),
        ("iterative scan relaxed_order, ef_search 40", ["SET hnsw.iterative_scan = relaxed_order"]),
        ("iterative scan strict_order, ef_search 40", ["SET hnsw.iterative_scan = strict_order"]),
        ("iterative scan relaxed_order, max_scan_tuples 200000", ["SET hnsw.iterative_scan = relaxed_order", "SET hnsw.max_scan_tuples = 200000"]),
    ]
    filtered_block(cur, "A. One HNSW index over all tenants, no index on tenant_id", lambda tid: SQLF, lambda tid, i: (tid, Qo[i]), base_settings)

    # ---- row-level security ----
    cur.execute("DROP ROLE IF EXISTS app_user")
    cur.execute(f"CREATE ROLE app_user LOGIN PASSWORD '{APP_PW}' NOBYPASSRLS")
    cur.execute("GRANT SELECT ON passages TO app_user")
    cur.execute("ALTER TABLE passages ENABLE ROW LEVEL SECURITY")
    cur.execute("ALTER TABLE passages FORCE ROW LEVEL SECURITY")
    rls_settings = [("iterative scan off, ef_search 40 (defaults)", []), ("iterative scan relaxed_order, ef_search 40", ["SET hnsw.iterative_scan = relaxed_order"])]
    set_tenant = lambda cu, tid: cu.execute("SELECT set_config('app.tenant_id', %s, false)", (str(tid),))

    def rls_block(letter, note):
        app = connect("app_user", APP_PW)
        acur = app.cursor()
        acur.execute("SELECT current_user, current_setting('row_security')")
        log(f"\nrow-level security block {letter} runs as: {acur.fetchone()}")
        for pname, using in [("direct current_setting()", "tenant_id = nullif(current_setting('app.tenant_id', true), '')::int"),
                             ("current_setting() wrapped in a SELECT", "tenant_id = (SELECT nullif(current_setting('app.tenant_id', true), '')::int)")]:
            cur.execute("DROP POLICY IF EXISTS tenant_isolation ON passages")
            cur.execute(f"CREATE POLICY tenant_isolation ON passages USING ({using})")
            filtered_block(acur, f"{letter}. {note}: no WHERE in the query, the tenant filter comes from an RLS policy, {pname}", lambda tid: SQL, lambda tid, i: (Qo[i],), rls_settings, pre=set_tenant)
        app.close()

    rls_block("B", "Same HNSW index, no index on tenant_id")

    # ---- btree on tenant_id ----
    build("passages_tenant", "CREATE INDEX passages_tenant ON passages (tenant_id)")
    cur.execute("ANALYZE passages")
    filtered_block(cur, "C. HNSW index plus a B-tree on tenant_id (the planner chooses)", lambda tid: SQLF, lambda tid, i: (tid, Qo[i]),
                   [("iterative scan off, ef_search 40 (defaults)", []), ("iterative scan relaxed_order, ef_search 40", ["SET hnsw.iterative_scan = relaxed_order"])])

    rls_block("D", "HNSW index plus a B-tree on tenant_id")

    # ---- partial indexes ----
    for tid, name, _ in TENANTS[:2]:
        build(f"passages_hnsw_t{tid}", f"CREATE INDEX passages_hnsw_t{tid} ON passages USING hnsw (embedding vector_cosine_ops) WHERE tenant_id = {tid}")
    cur.execute("ANALYZE passages")
    log("\n### E. Partial HNSW index per tenant (whale and mid only), other indexes still present")
    log("setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan")
    for tid, name, share in TENANTS[:2]:
        sql = f"SELECT id FROM passages WHERE tenant_id = {tid} ORDER BY embedding <=> %s LIMIT 10"
        ids, cl, sv = run_queries(cur, sql, lambda i: (Qo[i],), LAT_N, LAT_N)
        row(f"partial index, ef_search 40, tenant id written into the SQL | {name} ({share * 100:g}%)", ids, cl, sv, truth_t[tid], plan_chain(cur, sql, (Qo[0],)), extra="f")
    for tid, name, _ in TENANTS[:2]:
        cur.execute(f"DROP INDEX passages_hnsw_t{tid}")

    # ---- quantization ----
    log(f"\n## Quantized indexes, unfiltered, {N:,} vectors")
    build("passages_hnsw_half", f"CREATE INDEX passages_hnsw_half ON passages USING hnsw ((embedding::halfvec({DIM})) halfvec_cosine_ops)")
    build("passages_hnsw_bit", f"CREATE INDEX passages_hnsw_bit ON passages USING hnsw ((binary_quantize(embedding)::bit({DIM})) bit_hamming_ops)")
    log("setting | index recall@10 vs exact | nDCG@10 on labels | recall@10 on labels | client ms median | client ms p95 | server ms median | plan")
    half = f"SELECT id FROM passages ORDER BY embedding::halfvec({DIM}) <=> %s::halfvec({DIM}) LIMIT 10"
    ids, cl, sv = run_queries(cur, half, lambda i: (Qo[i],), NQ, LAT_N)
    row("halfvec index (16-bit floats), ef_search 40", ids, cl, sv, truth, plan_chain(cur, half, (Qo[0],)))
    bit = f"SELECT id FROM passages ORDER BY binary_quantize(embedding)::bit({DIM}) <~> binary_quantize(%s::vector) LIMIT 10"
    ids, cl, sv = run_queries(cur, bit, lambda i: (Qo[i],), NQ, LAT_N)
    row("binary index (1 bit per dimension), no re-rank, ef_search 40", ids, cl, sv, truth, plan_chain(cur, bit, (Qo[0],)))
    for cand in [40, 100, 200, 400]:
        cur.execute(f"SET hnsw.ef_search = {cand}")
        rer = (f"SELECT id FROM (SELECT id, embedding FROM passages ORDER BY binary_quantize(embedding)::bit({DIM}) <~> binary_quantize(%s::vector) LIMIT {cand}) c "
               f"ORDER BY embedding <=> %s LIMIT 10")
        ids, cl, sv = run_queries(cur, rer, lambda i: (Qo[i], Qo[i]), NQ, LAT_N)
        row(f"binary index, top {cand} re-ranked on the full vectors, ef_search {cand}", ids, cl, sv, truth, plan_chain(cur, rer, (Qo[0], Qo[0])))
    cur.execute("RESET hnsw.ef_search")

    log("\n## Sizes")
    for rel in ["passages", "passages_pkey", "passages_hnsw", "passages_tenant", "passages_hnsw_half", "passages_hnsw_bit"]:
        s = size(rel)
        log(f"{rel} | {s[0]} | {s[1]:,} bytes")
    c.close()


if __name__ == "__main__":
    main()

run-c.sh

One run of part C on a fresh container pinned by digest.

#!/bin/sh
# run-c.sh <n>: one run of rag_bench_c.py on a fresh pgvector container (image pinned by digest).
cd "$(dirname "$0")"
IMAGE=pgvector/pgvector:0.8.7-pg18@sha256:38681dc6eaf55308e2a4c6de392d131652dcfac70e1263e35cd9b5ea3fe39782
docker rm -f ragbench >/dev/null 2>&1
docker run --rm -d --name ragbench -e POSTGRES_PASSWORD=rag-throwaway -p 54334:5432 --shm-size=5g "$IMAGE" \
  -c shared_buffers=4GB -c maintenance_work_mem=4GB -c max_parallel_maintenance_workers=7 -c max_wal_size=8GB >/dev/null
until docker exec ragbench pg_isready -U postgres >/dev/null 2>&1; do sleep 1; done
sleep 2
PGPORT=54334 PGPASSWORD=rag-throwaway .venv/bin/python -I rag_bench_c.py > "run-c-$1.txt" 2> "run-c-$1.err"
echo "C run $1 exit $?" >> runs.log
docker rm -f ragbench >/dev/null 2>&1

rag_bench_latency.py

The separate per-stage latency pass.

# rag_bench_latency.py: per-stage latency of one query, measured on its own with nothing else running.
# Reuses the document embeddings cached by rag_bench_a.py. Run it several times; each pass prints one table per dataset.
#   python -I rag_bench_latency.py
import os, sys, time, random
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from rag_common import *
from rag_bench_a import load, DATASETS, DENSE, DEPTHS, LAT_N

WARM = 5


def timed(fn, sample):
    out = []
    for j, qi in enumerate(sample[:WARM] + sample):
        t0 = time.perf_counter()
        fn(qi)
        if j >= WARM:
            out.append((time.perf_counter() - t0) * 1000)
    return out


def main():
    print_env("rag-bench latency pass: one query at a time")
    log(f"{LAT_N} seeded queries per dataset after {WARM} warm-ups; median and p95 (nearest rank) in ms; batch size 1 for query embedding, one batch of N pairs for reranking")
    log(f"started {time.strftime('%H:%M:%S')} local time")
    for ds in DATASETS:
        doc_ids, docs, qids, queries, qrels = load(ds)
        sample = random.Random(SEED).sample(range(len(queries)), min(LAT_N, len(queries)))
        log(f"\n## {ds}: {len(docs):,} documents")
        log("stage | median ms | p95 ms")
        bm = BM25(docs)
        xs = timed(lambda qi: bm.search([queries[qi]], 100), sample)
        log(f"bm25 search, top 100 | {med(xs):.2f} | {pct(xs, 95):.2f}")
        hyb = {}
        for name in DENSE:
            D = np.load(os.path.join(CACHE, f"{ds}__{name}.npy"))
            m = embedder(name)
            qv = {}

            def emb(qi):
                qv[qi] = m.encode([queries[qi]], normalize_embeddings=True, convert_to_numpy=True, show_progress_bar=False, prompt=EMBEDDERS[name][1]).astype(np.float32)

            xs = timed(emb, sample)
            log(f"embed query, {name} | {med(xs):.2f} | {pct(xs, 95):.2f}")

            def search(qi):
                sims = (qv[qi] @ D.T)[0]
                top = np.argpartition(-sims, 99)[:100]
                return top[np.argsort(-sims[top])]

            xs = timed(search, sample)
            log(f"exact vector search in numpy, {name} ({len(docs):,} x {D.shape[1]}) | {med(xs):.2f} | {pct(xs, 95):.2f}")
            if name == "bge-small":
                bidx = bm.search([queries[qi] for qi in sample], 100)
                for j, qi in enumerate(sample):
                    hyb[qi] = rrf([list(bidx[j]), list(search(qi))])[:100]
                tops = {qi: list(search(qi)) for qi in sample}
                pos = {qi: j for j, qi in enumerate(sample)}
                xs = timed(lambda qi: rrf([list(bidx[pos[qi]]), tops[qi]]), sample)
                log(f"rrf fusion of two top-100 lists | {med(xs):.2f} | {pct(xs, 95):.2f}")
            drop_model(name)
            del D
        for rr in RERANKERS:
            ce = reranker(rr)
            for depth in DEPTHS:
                xs = timed(lambda qi: ce.predict([(queries[qi], docs[int(d)]) for d in hyb[qi][:depth]], batch_size=depth, show_progress_bar=False), sample)
                log(f"rerank {depth} candidates, {rr} | {med(xs):.2f} | {pct(xs, 95):.2f}")
            drop_model(rr)
    log(f"\nfinished {time.strftime('%H:%M:%S')} local time")


if __name__ == "__main__":
    main()

run-a-2.txt

Unedited output of part A run 2, the published part A run. The arrows in row labels are the script's own.

# rag-bench part A: BM25, dense, hybrid and reranking
date (UTC): 2026-11-16
machine: Apple M1 Max, 64 GB, macOS 27.2.1
python 3.12.14, torch 2.15.0 (mps available: True), sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5
bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector (Python client) 0.5.1
seed: 20261116
embedder minilm: sentence-transformers/all-MiniLM-L6-v2 @ 1110a243fdf4, query prefix: none
embedder bge-small: BAAI/bge-small-en-v1.5 @ 5c38ec7c405e, query prefix: yes
embedder bge-base: BAAI/bge-base-en-v1.5 @ a5beb1e3e68b, query prefix: yes
embedder gte-mbert: Alibaba-NLP/gte-modernbert-base @ e7f32e3c00f9, query prefix: none
reranker ce-minilm: cross-encoder/ms-marco-MiniLM-L6-v2 @ 233902d25c44, max_length 512
reranker bge-rr-base: BAAI/bge-reranker-base @ 2cfc18c9415c, max_length 512
metrics: nDCG@10 (linear gain) with a 95% bootstrap interval over queries (10,000 resamples), MRR@10, recall@10, recall@100
hybrid: reciprocal rank fusion, k=60, over the top 100 of BM25 and the top 100 of the dense model
rerank: the cross-encoder rescored the top N of the named first stage; rows show N
datasets: scifact = BEIR SciFact, full corpus and test queries; nq = a Natural Questions (BEIR) sample of 60,000 passages (every passage labelled for any test query, plus seeded random passages) with 600 seeded test queries; squad = SQuAD v1.1 paragraphs from all 490 articles with 600 seeded dev questions
latency: one query at a time on this machine, median and p95 (nearest rank) over 100 seeded queries after 5 warm-ups

## scifact: 5,183 documents, 300 test queries, 339 relevance labels
bm25 index build: 0.5 s (bm25s method=lucene k1=1.5 b=0.75, English stopwords, Snowball English stemmer)
  (reusing scifact__minilm.npy, written 2026-11-16 12:40 local time; the embedding time below is from that pass)
embed minilm: 5,183 docs in 9.5 s = 544 docs/s, dim 384, 8 MiB as float32
  (reusing scifact__bge-small.npy, written 2026-11-16 12:41 local time; the embedding time below is from that pass)
embed bge-small: 5,183 docs in 29.6 s = 175 docs/s, dim 384, 8 MiB as float32
  (reusing scifact__bge-base.npy, written 2026-11-16 12:42 local time; the embedding time below is from that pass)
embed bge-base: 5,183 docs in 70.2 s = 74 docs/s, dim 768, 15 MiB as float32
  (reusing scifact__gte-mbert.npy, written 2026-11-16 12:44 local time; the embedding time below is from that pass)
embed gte-mbert: 5,183 docs in 120.9 s = 43 docs/s, dim 768, 15 MiB as float32
rerank ce-minilm: 26,489 pairs in 83.5 s = 317 pairs/s (batched)
rerank bge-rr-base: 26,489 pairs in 466.3 s = 57 pairs/s (batched)

### scifact: quality
pipeline | nDCG@10 [95% CI] | MRR@10 | recall@10 | recall@100
bm25 | 0.686 [0.642, 0.729] | 0.649 | 0.819 | 0.913
dense minilm | 0.645 [0.600, 0.690] | 0.605 | 0.783 | 0.925
hybrid bm25 + minilm | 0.716 [0.674, 0.757] | 0.681 | 0.848 | 0.948
dense bge-small | 0.713 [0.670, 0.754] | 0.682 | 0.836 | 0.942
dense bge-small, query prefix left out | 0.720 [0.678, 0.762] | 0.684 | 0.845 | 0.953
hybrid bm25 + bge-small | 0.728 [0.686, 0.768] | 0.693 | 0.855 | 0.965
dense bge-base | 0.740 [0.700, 0.779] | 0.703 | 0.874 | 0.967
dense bge-base, query prefix left out | 0.738 [0.697, 0.777] | 0.700 | 0.866 | 0.970
hybrid bm25 + bge-base | 0.738 [0.698, 0.778] | 0.705 | 0.864 | 0.973
dense gte-mbert | 0.767 [0.729, 0.804] | 0.733 | 0.896 | 0.973
hybrid bm25 + gte-mbert | 0.770 [0.732, 0.808] | 0.742 | 0.886 | 0.980
bm25 -> rerank top 50, ce-minilm | 0.685 [0.641, 0.729] | 0.657 | 0.804 | 0.913
dense bge-small -> rerank top 50, ce-minilm | 0.696 [0.652, 0.739] | 0.664 | 0.823 | 0.942
hybrid bm25 + bge-small -> rerank top 10, ce-minilm | 0.712 [0.670, 0.753] | 0.673 | 0.855 | 0.965
hybrid bm25 + bge-small -> rerank top 20, ce-minilm | 0.701 [0.658, 0.743] | 0.667 | 0.838 | 0.965
hybrid bm25 + bge-small -> rerank top 50, ce-minilm | 0.696 [0.654, 0.740] | 0.664 | 0.829 | 0.965
bm25 -> rerank top 50, bge-rr-base | 0.709 [0.666, 0.751] | 0.679 | 0.827 | 0.913
dense bge-small -> rerank top 50, bge-rr-base | 0.715 [0.673, 0.757] | 0.682 | 0.842 | 0.942
hybrid bm25 + bge-small -> rerank top 10, bge-rr-base | 0.730 [0.689, 0.771] | 0.696 | 0.855 | 0.965
hybrid bm25 + bge-small -> rerank top 20, bge-rr-base | 0.722 [0.681, 0.763] | 0.690 | 0.844 | 0.965
hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | 0.719 [0.677, 0.760] | 0.686 | 0.844 | 0.965

### scifact: paired differences in nDCG@10 (B minus A), 95% bootstrap interval over the same queries
A = bm25 | B = dense bge-small | +0.026 [-0.004, +0.057] | interval includes 0
A = dense bge-small | B = hybrid bm25 + bge-small | +0.015 [-0.006, +0.035] | interval includes 0
A = dense bge-base | B = hybrid bm25 + bge-base | -0.002 [-0.024, +0.018] | interval includes 0
A = dense minilm | B = dense bge-small | +0.068 [+0.042, +0.095] | B higher
A = dense bge-small | B = dense bge-base | +0.028 [+0.007, +0.048] | B higher
A = dense bge-base | B = dense gte-mbert | +0.027 [+0.003, +0.051] | B higher
A = dense bge-small, query prefix left out | B = dense bge-small | -0.007 [-0.020, +0.006] | interval includes 0
A = dense bge-base, query prefix left out | B = dense bge-base | +0.003 [-0.015, +0.020] | interval includes 0
A = hybrid bm25 + bge-small | B = hybrid bm25 + bge-small -> rerank top 50, ce-minilm | -0.031 [-0.060, -0.003] | A higher
A = hybrid bm25 + bge-small | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | -0.009 [-0.034, +0.017] | interval includes 0
A = bm25 | B = bm25 -> rerank top 50, bge-rr-base | +0.023 [+0.000, +0.045] | B higher
A = dense bge-small -> rerank top 50, bge-rr-base | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | +0.004 [-0.011, +0.019] | interval includes 0
A = hybrid bm25 + bge-small -> rerank top 10, bge-rr-base | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | -0.011 [-0.029, +0.007] | interval includes 0

### scifact: latency per query in ms (median | p95)
embed query, minilm | 5.51 | 7.40
exact vector search in numpy, minilm (5,183 x 384) | 0.23 | 0.29
embed query, bge-small | 10.11 | 12.53
exact vector search in numpy, bge-small (5,183 x 384) | 0.27 | 0.36
embed query, bge-base | 12.17 | 13.41
exact vector search in numpy, bge-base (5,183 x 768) | 0.56 | 0.64
embed query, gte-mbert | 14.85 | 16.37
exact vector search in numpy, gte-mbert (5,183 x 768) | 0.54 | 0.63
bm25 search | 0.12 | 0.16
rerank 10 candidates, ce-minilm | 35.33 | 42.10
rerank 20 candidates, ce-minilm | 63.33 | 68.17
rerank 50 candidates, ce-minilm | 160.53 | 176.78
rerank 10 candidates, bge-rr-base | 199.36 | 233.23
rerank 20 candidates, bge-rr-base | 387.21 | 432.45
rerank 50 candidates, bge-rr-base | 990.13 | 1171.76
rrf fusion of two top-100 lists | 0.05 | 0.05

## nq: 60,000 documents, 600 test queries, 744 relevance labels
bm25 index build: 2.4 s (bm25s method=lucene k1=1.5 b=0.75, English stopwords, Snowball English stemmer)
embed minilm: 60,000 docs in 58.7 s = 1,023 docs/s, dim 384, 88 MiB as float32
embed bge-small: 60,000 docs in 112.5 s = 533 docs/s, dim 384, 88 MiB as float32
embed bge-base: 60,000 docs in 307.1 s = 195 docs/s, dim 768, 176 MiB as float32
embed gte-mbert: 60,000 docs in 538.6 s = 111 docs/s, dim 768, 176 MiB as float32
rerank ce-minilm: 55,172 pairs in 59.3 s = 930 pairs/s (batched)
rerank bge-rr-base: 55,172 pairs in 314.3 s = 176 pairs/s (batched)

### nq: quality
pipeline | nDCG@10 [95% CI] | MRR@10 | recall@10 | recall@100
bm25 | 0.722 [0.693, 0.752] | 0.700 | 0.841 | 0.939
dense minilm | 0.870 [0.851, 0.890] | 0.851 | 0.962 | 0.994
hybrid bm25 + minilm | 0.841 [0.818, 0.864] | 0.825 | 0.939 | 0.994
dense bge-small | 0.892 [0.873, 0.910] | 0.879 | 0.966 | 0.995
dense bge-small, query prefix left out | 0.852 [0.830, 0.874] | 0.838 | 0.940 | 0.983
hybrid bm25 + bge-small | 0.855 [0.833, 0.875] | 0.837 | 0.954 | 0.994
dense bge-base | 0.915 [0.898, 0.930] | 0.906 | 0.971 | 0.996
dense bge-base, query prefix left out | 0.901 [0.883, 0.918] | 0.890 | 0.961 | 0.994
hybrid bm25 + bge-base | 0.863 [0.842, 0.884] | 0.849 | 0.951 | 0.996
dense gte-mbert | 0.921 [0.906, 0.936] | 0.906 | 0.984 | 0.999
hybrid bm25 + gte-mbert | 0.872 [0.852, 0.892] | 0.859 | 0.961 | 0.998
bm25 -> rerank top 50, ce-minilm | 0.872 [0.849, 0.894] | 0.871 | 0.914 | 0.939
dense bge-small -> rerank top 50, ce-minilm | 0.916 [0.899, 0.932] | 0.906 | 0.973 | 0.995
hybrid bm25 + bge-small -> rerank top 10, ce-minilm | 0.905 [0.886, 0.923] | 0.900 | 0.954 | 0.994
hybrid bm25 + bge-small -> rerank top 20, ce-minilm | 0.913 [0.897, 0.930] | 0.904 | 0.971 | 0.994
hybrid bm25 + bge-small -> rerank top 50, ce-minilm | 0.915 [0.899, 0.931] | 0.906 | 0.974 | 0.994
bm25 -> rerank top 50, bge-rr-base | 0.834 [0.809, 0.858] | 0.820 | 0.912 | 0.939
dense bge-small -> rerank top 50, bge-rr-base | 0.685 [0.657, 0.712] | 0.633 | 0.877 | 0.995
hybrid bm25 + bge-small -> rerank top 10, bge-rr-base | 0.875 [0.856, 0.894] | 0.859 | 0.954 | 0.994
hybrid bm25 + bge-small -> rerank top 20, bge-rr-base | 0.848 [0.828, 0.867] | 0.814 | 0.973 | 0.994
hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | 0.747 [0.722, 0.771] | 0.694 | 0.933 | 0.994

### nq: paired differences in nDCG@10 (B minus A), 95% bootstrap interval over the same queries
A = bm25 | B = dense bge-small | +0.170 [+0.141, +0.198] | B higher
A = dense bge-small | B = hybrid bm25 + bge-small | -0.037 [-0.056, -0.019] | A higher
A = dense bge-base | B = hybrid bm25 + bge-base | -0.051 [-0.069, -0.034] | A higher
A = dense minilm | B = dense bge-small | +0.022 [+0.005, +0.037] | B higher
A = dense bge-small | B = dense bge-base | +0.023 [+0.010, +0.036] | B higher
A = dense bge-base | B = dense gte-mbert | +0.007 [-0.007, +0.021] | interval includes 0
A = dense bge-small, query prefix left out | B = dense bge-small | +0.039 [+0.028, +0.051] | B higher
A = dense bge-base, query prefix left out | B = dense bge-base | +0.014 [+0.005, +0.023] | B higher
A = hybrid bm25 + bge-small | B = hybrid bm25 + bge-small -> rerank top 50, ce-minilm | +0.061 [+0.044, +0.079] | B higher
A = hybrid bm25 + bge-small | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | -0.108 [-0.135, -0.081] | A higher
A = bm25 | B = bm25 -> rerank top 50, bge-rr-base | +0.111 [+0.087, +0.137] | B higher
A = dense bge-small -> rerank top 50, bge-rr-base | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | +0.062 [+0.052, +0.071] | B higher
A = hybrid bm25 + bge-small -> rerank top 10, bge-rr-base | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | -0.128 [-0.148, -0.108] | A higher

### nq: latency per query in ms (median | p95)
embed query, minilm | 6.49 | 8.53
exact vector search in numpy, minilm (60,000 x 384) | 2.76 | 3.13
embed query, bge-small | 10.31 | 11.95
exact vector search in numpy, bge-small (60,000 x 384) | 2.79 | 3.11
embed query, bge-base | 11.95 | 13.30
exact vector search in numpy, bge-base (60,000 x 768) | 6.09 | 6.49
embed query, gte-mbert | 14.15 | 15.67
exact vector search in numpy, gte-mbert (60,000 x 768) | 5.97 | 6.26
bm25 search | 0.48 | 0.72
rerank 10 candidates, ce-minilm | 21.74 | 32.29
rerank 20 candidates, ce-minilm | 40.02 | 63.86
rerank 50 candidates, ce-minilm | 112.32 | 159.68
rerank 10 candidates, bge-rr-base | 103.56 | 165.40
rerank 20 candidates, bge-rr-base | 241.74 | 361.01
rerank 50 candidates, bge-rr-base | 768.22 | 967.30
rrf fusion of two top-100 lists | 0.05 | 0.05

## squad: 20,963 documents, 600 test queries, 600 relevance labels
bm25 index build: 1.2 s (bm25s method=lucene k1=1.5 b=0.75, English stopwords, Snowball English stemmer)
  (reusing squad__minilm.npy, written 2026-11-16 13:40 local time; the embedding time below is from that pass)
embed minilm: 20,963 docs in 23.8 s = 879 docs/s, dim 384, 31 MiB as float32
  (reusing squad__bge-small.npy, written 2026-11-16 13:41 local time; the embedding time below is from that pass)
embed bge-small: 20,963 docs in 53.0 s = 395 docs/s, dim 384, 31 MiB as float32
  (reusing squad__bge-base.npy, written 2026-11-16 13:43 local time; the embedding time below is from that pass)
embed bge-base: 20,963 docs in 144.6 s = 145 docs/s, dim 768, 61 MiB as float32
  (reusing squad__gte-mbert.npy, written 2026-11-16 13:47 local time; the embedding time below is from that pass)
embed gte-mbert: 20,963 docs in 219.7 s = 95 docs/s, dim 768, 61 MiB as float32
rerank ce-minilm: 52,297 pairs in 84.3 s = 620 pairs/s (batched)
rerank bge-rr-base: 52,297 pairs in 467.2 s = 112 pairs/s (batched)

### squad: quality
pipeline | nDCG@10 [95% CI] | MRR@10 | recall@10 | recall@100
bm25 | 0.828 [0.803, 0.852] | 0.795 | 0.928 | 0.982
dense minilm | 0.713 [0.683, 0.743] | 0.668 | 0.853 | 0.953
hybrid bm25 + minilm | 0.814 [0.790, 0.839] | 0.777 | 0.930 | 0.988
dense bge-small | 0.734 [0.705, 0.763] | 0.690 | 0.870 | 0.963
dense bge-small, query prefix left out | 0.733 [0.704, 0.761] | 0.689 | 0.873 | 0.965
hybrid bm25 + bge-small | 0.823 [0.799, 0.847] | 0.787 | 0.937 | 0.990
dense bge-base | 0.752 [0.724, 0.779] | 0.707 | 0.892 | 0.965
dense bge-base, query prefix left out | 0.750 [0.722, 0.777] | 0.705 | 0.892 | 0.965
hybrid bm25 + bge-base | 0.831 [0.808, 0.854] | 0.794 | 0.947 | 0.990
dense gte-mbert | 0.803 [0.778, 0.829] | 0.766 | 0.918 | 0.977
hybrid bm25 + gte-mbert | 0.861 [0.839, 0.881] | 0.831 | 0.952 | 0.990
bm25 -> rerank top 50, ce-minilm | 0.904 [0.885, 0.923] | 0.885 | 0.960 | 0.982
dense bge-small -> rerank top 50, ce-minilm | 0.883 [0.861, 0.904] | 0.865 | 0.937 | 0.963
hybrid bm25 + bge-small -> rerank top 10, ce-minilm | 0.888 [0.867, 0.909] | 0.872 | 0.937 | 0.990
hybrid bm25 + bge-small -> rerank top 20, ce-minilm | 0.904 [0.885, 0.922] | 0.885 | 0.962 | 0.990
hybrid bm25 + bge-small -> rerank top 50, ce-minilm | 0.910 [0.893, 0.928] | 0.889 | 0.973 | 0.990
bm25 -> rerank top 50, bge-rr-base | 0.867 [0.845, 0.889] | 0.840 | 0.950 | 0.982
dense bge-small -> rerank top 50, bge-rr-base | 0.819 [0.794, 0.843] | 0.788 | 0.915 | 0.963
hybrid bm25 + bge-small -> rerank top 10, bge-rr-base | 0.861 [0.838, 0.883] | 0.836 | 0.937 | 0.990
hybrid bm25 + bge-small -> rerank top 20, bge-rr-base | 0.866 [0.844, 0.886] | 0.836 | 0.957 | 0.990
hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | 0.858 [0.836, 0.879] | 0.825 | 0.958 | 0.990

### squad: paired differences in nDCG@10 (B minus A), 95% bootstrap interval over the same queries
A = bm25 | B = dense bge-small | -0.094 [-0.124, -0.063] | A higher
A = dense bge-small | B = hybrid bm25 + bge-small | +0.090 [+0.070, +0.109] | B higher
A = dense bge-base | B = hybrid bm25 + bge-base | +0.079 [+0.060, +0.098] | B higher
A = dense minilm | B = dense bge-small | +0.021 [-0.004, +0.045] | interval includes 0
A = dense bge-small | B = dense bge-base | +0.018 [-0.002, +0.039] | interval includes 0
A = dense bge-base | B = dense gte-mbert | +0.051 [+0.030, +0.073] | B higher
A = dense bge-small, query prefix left out | B = dense bge-small | +0.001 [-0.010, +0.011] | interval includes 0
A = dense bge-base, query prefix left out | B = dense bge-base | +0.002 [-0.009, +0.013] | interval includes 0
A = hybrid bm25 + bge-small | B = hybrid bm25 + bge-small -> rerank top 50, ce-minilm | +0.087 [+0.068, +0.106] | B higher
A = hybrid bm25 + bge-small | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | +0.034 [+0.009, +0.061] | B higher
A = bm25 | B = bm25 -> rerank top 50, bge-rr-base | +0.040 [+0.017, +0.062] | B higher
A = dense bge-small -> rerank top 50, bge-rr-base | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | +0.038 [+0.023, +0.054] | B higher
A = hybrid bm25 + bge-small -> rerank top 10, bge-rr-base | B = hybrid bm25 + bge-small -> rerank top 50, bge-rr-base | -0.004 [-0.021, +0.014] | interval includes 0

### squad: latency per query in ms (median | p95)
embed query, minilm | 6.00 | 7.75
exact vector search in numpy, minilm (20,963 x 384) | 0.94 | 1.07
embed query, bge-small | 10.04 | 11.47
exact vector search in numpy, bge-small (20,963 x 384) | 0.94 | 1.04
embed query, bge-base | 11.72 | 12.70
exact vector search in numpy, bge-base (20,963 x 768) | 2.05 | 2.21
embed query, gte-mbert | 14.63 | 34.17
exact vector search in numpy, gte-mbert (20,963 x 768) | 2.09 | 4.14
bm25 search | 0.22 | 0.31
rerank 10 candidates, ce-minilm | 23.59 | 32.81
rerank 20 candidates, ce-minilm | 42.99 | 61.54
rerank 50 candidates, ce-minilm | 131.81 | 166.96
rerank 10 candidates, bge-rr-base | 125.33 | 191.68
rerank 20 candidates, bge-rr-base | 264.20 | 384.45
rerank 50 candidates, bge-rr-base | 758.30 | 907.88
rrf fusion of two top-100 lists | 0.05 | 0.20

run-c-prep.txt

Unedited output of the part C prep run.

# rag-bench part C prep: Natural Questions sample and embeddings
date (UTC): 2026-11-16
machine: Apple M1 Max, 64 GB, macOS 27.2.1
python 3.12.14, torch 2.15.0 (mps available: True), sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5
bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector (Python client) 0.5.1
seed: 20261116
nq corpus: 2,681,468 passages, 3,452 test queries, 4,201 relevance labels on 4,201 passages
sample: every labelled passage (4,201) plus 495,799 seeded random passages = 500,000
passage length in bge-small tokens, 2,000 sampled: median 94, mean 108, max 512
embed bge-small: 500,000 passages in 810 s = 618 passages/s, about 66,953 tokens/s, dim 384, 732 MiB as float32
queries embedded: 3,452

run-c-1.txt

Unedited output of part C run 1.

# rag-bench part C: pgvector HNSW, tenant filters, row-level security and quantization
date (UTC): 2026-11-16
machine: Apple M1 Max, 64 GB, macOS 27.2.1
python 3.12.14, torch 2.15.0 (mps available: True), sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5
bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector (Python client) 0.5.1
seed: 20261116
vectors: 500,000 x 384 (BAAI/bge-small-en-v1.5, normalized, float32); queries: 3,452 Natural Questions test queries
synthetic tenant labels (seeded, independent of content): whale = tenant 1: 199,922 rows (39.98%), mid = tenant 2: 24,965 rows (4.99%), small = tenant 3: 5,012 rows (1.00%), tiny = tenant 4: 471 rows (0.09%); the other 996 tenants hold 269,630 rows
exact top 10 in numpy for all 3,452 queries: 12.2 s (ground truth for index recall)
exact search quality on the labels: nDCG@10 0.7177, recall@10 0.8780
server: PostgreSQL 18.7 (Debian 18.7-1.pgdg12+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
docker: 29.7.2, 10 CPUs, 15690960896 bytes of memory in the VM
image: pgvector/pgvector:0.8.7-pg18@sha256:38681dc6eaf55308e2a4c6de392d131652dcfac70e1263e35cd9b5ea3fe39782
pgvector: 0.8.7
  shared_buffers = 4GB
  maintenance_work_mem = 4GB
  max_parallel_maintenance_workers = 7
  max_parallel_workers_per_gather = 2
  work_mem = 4MB
  jit = on
  hnsw.ef_search = 40
  hnsw.iterative_scan = off
  hnsw.max_scan_tuples = 20000
  hnsw.scan_mem_multiplier = 1
load: 500,000 rows by COPY in 5.4 s
table passages: 782 MB (heap only)
build passages_hnsw: 50.0 s, 975 MB
  (pgvector defaults: m = 16, ef_construction = 64)

## Unfiltered search, 500,000 vectors: HNSW against exact
quality columns use all 3,452 queries; timing columns use 500 seeded queries after 50 warm-ups, one connection, no prepared statements
client ms = round trip seen by the Python client through Docker's port mapping; server ms = planning + execution time from EXPLAIN ANALYZE
setting | index recall@10 vs exact | nDCG@10 on labels | recall@10 on labels | client ms median | client ms p95 | server ms median | plan
exact scan in Postgres (index disabled), 50 queries | 1.0000 | n/a | n/a | 86.53 | 94.94 | 88.10 | Limit > Gather Merge > Sort > Seq Scan
hnsw ef_search = 10 | 0.7808 | 0.5864 | 0.7199 | 2.78 | 6.71 | 0.26 | Limit > Index Scan passages_hnsw
hnsw ef_search = 20 | 0.8636 | 0.6359 | 0.7810 | 1.26 | 2.19 | 0.33 | Limit > Index Scan passages_hnsw
hnsw ef_search = 40 (default) | 0.9175 | 0.6669 | 0.8191 | 1.96 | 3.27 | 0.69 | Limit > Index Scan passages_hnsw
hnsw ef_search = 100 | 0.9599 | 0.6946 | 0.8512 | 3.68 | 5.39 | 1.89 | Limit > Index Scan passages_hnsw
hnsw ef_search = 200 | 0.9777 | 0.7054 | 0.8627 | 5.75 | 7.62 | 3.95 | Limit > Index Scan passages_hnsw
hnsw ef_search = 400 | 0.9876 | 0.7109 | 0.8687 | 10.02 | 13.90 | 7.78 | Limit > Index Scan passages_hnsw

## Tenant-filtered search: WHERE tenant_id = $1 ORDER BY embedding <=> $2 LIMIT 10 (500 queries per tenant)

### A. One HNSW index over all tenants, no index on tenant_id
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
iterative scan off, ef_search 40 (defaults) | whale (40%) | 0.8742 | 9.96 | 10 | 1.78 | 2.46 | 0.76 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | mid (5%) | 0.1918 | 1.92 | 500 | 1.72 | 2.39 | 0.73 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | small (1%) | 0.0452 | 0.45 | 500 | 1.83 | 2.50 | 0.87 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | tiny (0.1%) | 0.0044 | 0.04 | 500 | 1.78 | 2.51 | 0.80 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 200 | whale (40%) | 0.9690 | 10.00 | 0 | 5.18 | 6.85 | 3.31 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 200 | mid (5%) | 0.8206 | 8.69 | 233 | 6.88 | 11.60 | 5.01 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 200 | small (1%) | 0.2008 | 2.01 | 500 | 5.72 | 7.85 | 3.63 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 200 | tiny (0.1%) | 0.0200 | 0.20 | 500 | 6.24 | 8.72 | 4.10 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | whale (40%) | 0.8782 | 10.00 | 0 | 2.29 | 3.71 | 1.14 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | mid (5%) | 0.9098 | 10.00 | 0 | 9.41 | 16.10 | 6.80 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | small (1%) | 0.9360 | 10.00 | 0 | 28.76 | 49.54 | 25.73 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | tiny (0.1%) | 0.6974 | 10.00 | 1 | 54.94 | 77.64 | 52.45 | Limit > Index Scan passages_hnsw +Filter
iterative scan strict_order, ef_search 40 | whale (40%) | 0.8752 | 10.00 | 0 | 4.20 | 10.74 | 1.68 | Limit > Index Scan passages_hnsw +Filter
iterative scan strict_order, ef_search 40 | mid (5%) | 0.8190 | 10.00 | 0 | 9.42 | 18.93 | 7.18 | Limit > Index Scan passages_hnsw +Filter
iterative scan strict_order, ef_search 40 | small (1%) | 0.8414 | 10.00 | 0 | 29.63 | 45.67 | 27.20 | Limit > Index Scan passages_hnsw +Filter
iterative scan strict_order, ef_search 40 | tiny (0.1%) | 0.6858 | 10.00 | 1 | 55.74 | 81.33 | 52.50 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, max_scan_tuples 200000 | whale (40%) | 0.8782 | 10.00 | 0 | 3.31 | 7.75 | 1.20 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, max_scan_tuples 200000 | mid (5%) | 0.9098 | 10.00 | 0 | 10.78 | 22.40 | 7.32 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, max_scan_tuples 200000 | small (1%) | 0.9360 | 10.00 | 0 | 23.45 | 41.41 | 21.17 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, max_scan_tuples 200000 | tiny (0.1%) | 0.7032 | 10.00 | 1 | 38.98 | 46.11 | 37.27 | Limit > Index Scan passages_hnsw +Filter

row-level security block B runs as: ('app_user', 'on')

### B. Same HNSW index, no index on tenant_id: no WHERE in the query, the tenant filter comes from an RLS policy, direct current_setting()
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
iterative scan off, ef_search 40 (defaults) | whale (40%) | 0.8742 | 9.96 | 10 | 1.75 | 2.39 | 0.55 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | mid (5%) | 0.1918 | 1.92 | 500 | 1.74 | 2.52 | 0.59 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | small (1%) | 0.0452 | 0.45 | 500 | 1.69 | 2.39 | 0.54 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | tiny (0.1%) | 0.0044 | 0.04 | 500 | 1.72 | 2.66 | 0.55 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | whale (40%) | 0.8782 | 10.00 | 0 | 1.73 | 2.73 | 0.56 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | mid (5%) | 0.9098 | 10.00 | 0 | 5.85 | 9.40 | 3.56 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | small (1%) | 0.9360 | 10.00 | 0 | 20.15 | 29.72 | 17.16 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | tiny (0.1%) | 0.6974 | 10.00 | 1 | 41.42 | 55.36 | 39.74 | Limit > Index Scan passages_hnsw +Filter

### B. Same HNSW index, no index on tenant_id: no WHERE in the query, the tenant filter comes from an RLS policy, current_setting() wrapped in a SELECT
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
iterative scan off, ef_search 40 (defaults) | whale (40%) | 0.8742 | 9.96 | 10 | 2.39 | 8.88 | 1.18 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | mid (5%) | 0.1918 | 1.92 | 500 | 1.64 | 2.36 | 0.55 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | small (1%) | 0.0452 | 0.45 | 500 | 1.68 | 2.38 | 0.54 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | tiny (0.1%) | 0.0044 | 0.04 | 500 | 1.67 | 2.37 | 0.54 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | whale (40%) | 0.8782 | 10.00 | 0 | 1.68 | 2.50 | 0.55 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | mid (5%) | 0.9098 | 10.00 | 0 | 5.92 | 9.88 | 3.71 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | small (1%) | 0.9360 | 10.00 | 0 | 19.57 | 29.33 | 16.59 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | tiny (0.1%) | 0.6974 | 10.00 | 1 | 40.64 | 49.88 | 38.67 | Limit > Index Scan passages_hnsw +Filter
build passages_tenant: 0.1 s, 3520 kB

### C. HNSW index plus a B-tree on tenant_id (the planner chooses)
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
iterative scan off, ef_search 40 (defaults) | whale (40%) | 0.8742 | 9.96 | 10 | 1.64 | 2.29 | 0.53 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | mid (5%) | 0.1918 | 1.92 | 500 | 1.69 | 2.51 | 0.54 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | small (1%) | 0.0452 | 0.45 | 500 | 1.70 | 2.45 | 0.54 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | tiny (0.1%) | 1.0000 | 10.00 | 0 | 0.56 | 0.64 | 0.30 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan relaxed_order, ef_search 40 | whale (40%) | 0.8782 | 10.00 | 0 | 1.69 | 2.57 | 0.56 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | mid (5%) | 0.9098 | 10.00 | 0 | 5.49 | 8.45 | 3.27 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | small (1%) | 0.9360 | 10.00 | 0 | 19.35 | 28.44 | 16.40 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | tiny (0.1%) | 1.0000 | 10.00 | 0 | 0.57 | 0.76 | 0.30 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant

row-level security block D runs as: ('app_user', 'on')

### D. HNSW index plus a B-tree on tenant_id: no WHERE in the query, the tenant filter comes from an RLS policy, direct current_setting()
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
iterative scan off, ef_search 40 (defaults) | whale (40%) | 0.8742 | 9.96 | 10 | 1.72 | 2.62 | 0.55 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | mid (5%) | 0.1918 | 1.92 | 500 | 1.71 | 2.42 | 0.54 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | small (1%) | 0.0452 | 0.45 | 500 | 1.68 | 2.53 | 0.57 | Limit > Index Scan passages_hnsw +Filter
iterative scan off, ef_search 40 (defaults) | tiny (0.1%) | 1.0000 | 10.00 | 0 | 0.57 | 0.79 | 0.30 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan relaxed_order, ef_search 40 | whale (40%) | 0.8782 | 10.00 | 0 | 1.79 | 2.70 | 0.66 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | mid (5%) | 0.9098 | 10.00 | 0 | 5.84 | 9.66 | 3.63 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | small (1%) | 0.9360 | 10.00 | 0 | 19.73 | 29.52 | 17.04 | Limit > Index Scan passages_hnsw +Filter
iterative scan relaxed_order, ef_search 40 | tiny (0.1%) | 1.0000 | 10.00 | 0 | 0.56 | 0.72 | 0.30 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant

### D. HNSW index plus a B-tree on tenant_id: no WHERE in the query, the tenant filter comes from an RLS policy, current_setting() wrapped in a SELECT
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
iterative scan off, ef_search 40 (defaults) | whale (40%) | 0.9998 | 10.00 | 0 | 133.86 | 142.01 | 138.37 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan off, ef_search 40 (defaults) | mid (5%) | 1.0000 | 10.00 | 0 | 27.42 | 30.39 | 27.19 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan off, ef_search 40 (defaults) | small (1%) | 1.0000 | 10.00 | 0 | 5.22 | 8.08 | 4.71 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan off, ef_search 40 (defaults) | tiny (0.1%) | 1.0000 | 10.00 | 0 | 0.59 | 0.96 | 0.30 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan relaxed_order, ef_search 40 | whale (40%) | 0.9998 | 10.00 | 0 | 135.08 | 152.08 | 139.38 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan relaxed_order, ef_search 40 | mid (5%) | 1.0000 | 10.00 | 0 | 27.48 | 30.17 | 27.34 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan relaxed_order, ef_search 40 | small (1%) | 1.0000 | 10.00 | 0 | 5.06 | 6.15 | 4.58 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
iterative scan relaxed_order, ef_search 40 | tiny (0.1%) | 1.0000 | 10.00 | 0 | 0.68 | 1.48 | 0.36 | Limit > Sort > Bitmap Heap Scan > Bitmap Index Scan passages_tenant
build passages_hnsw_t1: 17.9 s, 390 MB
build passages_hnsw_t2: 2.5 s, 49 MB

### E. Partial HNSW index per tenant (whale and mid only), other indexes still present
setting | tenant | index recall@10 vs exact within the tenant | rows returned (mean of 10 asked) | queries with fewer than 10 rows | client ms median | client ms p95 | server ms median | plan
partial index, ef_search 40, tenant id written into the SQL | whale (40%) | 0.9212 | 10.00 | 0 | 1.47 | 1.86 | 0.51 | Limit > Index Scan passages_hnsw_t1
partial index, ef_search 40, tenant id written into the SQL | mid (5%) | 0.9526 | 10.00 | 0 | 1.04 | 1.55 | 0.42 | Limit > Index Scan passages_hnsw_t2

## Quantized indexes, unfiltered, 500,000 vectors
build passages_hnsw_half: 32.5 s, 557 MB
build passages_hnsw_bit: 21.8 s, 166 MB
setting | index recall@10 vs exact | nDCG@10 on labels | recall@10 on labels | client ms median | client ms p95 | server ms median | plan
halfvec index (16-bit floats), ef_search 40 | 0.9179 | 0.6690 | 0.8204 | 1.46 | 2.06 | 0.45 | Limit > Index Scan passages_hnsw_half
binary index (1 bit per dimension), no re-rank, ef_search 40 | 0.4745 | 0.5411 | 0.6939 | 1.45 | 1.99 | 0.54 | Limit > Index Scan passages_hnsw_bit
binary index, top 40 re-ranked on the full vectors, ef_search 40 | 0.7177 | 0.6308 | 0.7644 | 1.54 | 2.24 | 0.57 | Limit > Sort > Subquery Scan > Limit > Index Scan passages_hnsw_bit
binary index, top 100 re-ranked on the full vectors, ef_search 100 | 0.8419 | 0.6754 | 0.8213 | 2.80 | 3.66 | 1.43 | Limit > Sort > Subquery Scan > Limit > Index Scan passages_hnsw_bit
binary index, top 200 re-ranked on the full vectors, ef_search 200 | 0.9011 | 0.6911 | 0.8433 | 4.49 | 6.26 | 2.66 | Limit > Sort > Subquery Scan > Limit > Index Scan passages_hnsw_bit
binary index, top 400 re-ranked on the full vectors, ef_search 400 | 0.9422 | 0.7035 | 0.8592 | 7.17 | 9.92 | 4.76 | Limit > Sort > Subquery Scan > Limit > Index Scan passages_hnsw_bit

## Sizes
passages | 782 MB | 819,462,144 bytes
passages_pkey | 11 MB | 11,255,808 bytes
passages_hnsw | 975 MB | 1,022,337,024 bytes
passages_tenant | 3520 kB | 3,604,480 bytes
passages_hnsw_half | 557 MB | 584,196,096 bytes
passages_hnsw_bit | 166 MB | 173,981,696 bytes

run-b-1.txt

Unedited output of part B run 1.

# rag-bench part B: chunk size at a fixed context budget
date (UTC): 2026-11-16
machine: Apple M1 Max, 64 GB, macOS 27.2.1
python 3.12.14, torch 2.15.0 (mps available: True), sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5
bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector (Python client) 0.5.1
seed: 20261116
corpus: SQuAD v1.1, 490 Wikipedia articles (442 train + 48 dev), 20,963 paragraphs; queries: 2,000 seeded dev questions of 10,570
tokens are counted with the bge-small-en-v1.5 WordPiece tokenizer; an LLM's tokenizer will count differently
corpus size: 3,195,442 tokens, 15,544,792 characters; article length median 5,641 tokens, paragraph length median 116 tokens (first 60 articles)
answer spans located in the joined article text: 3,478 spans, 0 mismatches
chunked: 64 tokens, no overlap: 50,166 chunks, mean 64 tokens, embedded with bge-small in 33.5 s = 1,499 chunks/s, 95,506 tokens/s
chunked: 64 tokens, 13 overlap (20%): 62,760 chunks, mean 64 tokens, embedded with bge-small in 42.0 s = 1,493 chunks/s, 95,257 tokens/s
chunked: 128 tokens, no overlap: 25,210 chunks, mean 127 tokens, embedded with bge-small in 31.8 s = 793 chunks/s, 100,472 tokens/s
chunked: 128 tokens, 26 overlap (20%): 31,455 chunks, mean 127 tokens, embedded with bge-small in 39.0 s = 807 chunks/s, 102,655 tokens/s
chunked: 256 tokens, no overlap: 12,727 chunks, mean 251 tokens, embedded with bge-small in 39.1 s = 326 chunks/s, 81,828 tokens/s
chunked: 256 tokens, 51 overlap (20%): 15,719 chunks, mean 253 tokens, embedded with bge-small in 51.4 s = 306 chunks/s, 77,342 tokens/s
chunked: 512 tokens, no overlap: 6,487 chunks, mean 493 tokens, embedded with bge-small in 42.1 s = 154 chunks/s, 75,925 tokens/s
chunked: 512 tokens, 102 overlap (20%): 7,914 chunks, mean 499 tokens, embedded with bge-small in 54.3 s = 146 chunks/s, 72,733 tokens/s
chunked: 1024 tokens, no overlap: 3,367 chunks, mean 949 tokens, embedded with bge-small in 22.4 s = 150 chunks/s, 142,666 tokens/s
chunked: 1024 tokens, 205 overlap (20%): 4,025 chunks, mean 974 tokens, embedded with bge-small in 27.9 s = 144 chunks/s, 140,676 tokens/s
chunked: paragraphs as written (the article's own boundaries): 20,963 chunks, mean 152 tokens, embedded with bge-small in 52.0 s = 403 chunks/s, 61,455 tokens/s
chunked: paragraphs as written, article title prepended: 20,963 chunks, mean 156 tokens, embedded with bge-small in 52.0 s = 403 chunks/s, 62,914 tokens/s
chunked: 256 tokens including a prepended article title, no overlap: 12,906 chunks, mean 251 tokens, embedded with bge-small in 42.4 s = 304 chunks/s, 76,361 tokens/s

## Retriever: dense bge-small
chunking | chunks | mean tokens | answer span inside one chunk | hit@1 chunk | hit@5 chunks | answer in a 512-token budget | in 1,024 tokens [95% CI] | in 2,048 tokens
64 tokens, no overlap | 50,166 | 64 | 0.967 | 0.490 | 0.702 | 0.748 | 0.805 [0.788, 0.823] | 0.844
64 tokens, 13 overlap (20%) | 62,760 | 64 | 0.999 | 0.553 | 0.777 | 0.819 | 0.864 [0.849, 0.878] | 0.903
128 tokens, no overlap | 25,210 | 127 | 0.985 | 0.420 | 0.698 | 0.666 | 0.754 [0.735, 0.772] | 0.825
128 tokens, 26 overlap (20%) | 31,455 | 127 | 1.000 | 0.472 | 0.738 | 0.713 | 0.790 [0.773, 0.808] | 0.853
256 tokens, no overlap | 12,727 | 251 | 0.993 | 0.370 | 0.674 | 0.516 | 0.634 [0.613, 0.655] | 0.754
256 tokens, 51 overlap (20%) | 15,719 | 253 | 1.000 | 0.409 | 0.693 | 0.540 | 0.656 [0.635, 0.677] | 0.760
512 tokens, no overlap | 6,487 | 493 | 0.997 | 0.364 | 0.662 | 0.364 | 0.505 [0.482, 0.527] | 0.627
512 tokens, 102 overlap (20%) | 7,914 | 499 | 1.000 | 0.387 | 0.680 | 0.388 | 0.526 [0.504, 0.548] | 0.648
1024 tokens, no overlap | 3,367 | 949 | 0.999 | 0.296 | 0.576 | n/a (a full chunk does not fit) | 0.296 [0.276, 0.316] | 0.419
1024 tokens, 205 overlap (20%) | 4,025 | 974 | 1.000 | 0.321 | 0.599 | n/a (a full chunk does not fit) | 0.321 [0.300, 0.341] | 0.449
paragraphs as written (the article's own boundaries) | 20,963 | 152 | 1.000 | 0.618 | 0.818 | 0.750 | 0.829 [0.811, 0.845] | 0.888
paragraphs as written, article title prepended | 20,963 | 156 | 1.000 | 0.607 | 0.813 | 0.744 | 0.823 [0.806, 0.840] | 0.879
256 tokens including a prepended article title, no overlap | 12,906 | 251 | 0.995 | 0.380 | 0.668 | 0.510 | 0.633 [0.613, 0.655] | 0.745
paired difference in the 1,024-token budget hit rate against 256 tokens, no overlap (95% bootstrap interval):
  64 tokens, no overlap | +0.171 [+0.147, +0.193] | higher
  64 tokens, 13 overlap (20%) | +0.229 [+0.207, +0.251] | higher
  128 tokens, no overlap | +0.119 [+0.100, +0.139] | higher
  128 tokens, 26 overlap (20%) | +0.156 [+0.136, +0.177] | higher
  256 tokens, 51 overlap (20%) | +0.021 [+0.001, +0.042] | higher
  512 tokens, no overlap | -0.130 [-0.150, -0.108] | lower
  512 tokens, 102 overlap (20%) | -0.108 [-0.131, -0.086] | lower
  1024 tokens, no overlap | -0.339 [-0.362, -0.315] | lower
  1024 tokens, 205 overlap (20%) | -0.314 [-0.339, -0.289] | lower
  paragraphs as written (the article's own boundaries) | +0.194 [+0.173, +0.215] | higher
  paragraphs as written, article title prepended | +0.189 [+0.168, +0.209] | higher
  256 tokens including a prepended article title, no overlap | -0.001 [-0.021, +0.019] | interval includes 0

## Retriever: bm25
chunking | chunks | mean tokens | answer span inside one chunk | hit@1 chunk | hit@5 chunks | answer in a 512-token budget | in 1,024 tokens [95% CI] | in 2,048 tokens
64 tokens, no overlap | 50,166 | 64 | 0.967 | 0.517 | 0.710 | 0.747 | 0.801 [0.783, 0.818] | 0.834
64 tokens, 13 overlap (20%) | 62,760 | 64 | 0.999 | 0.558 | 0.775 | 0.817 | 0.866 [0.851, 0.880] | 0.900
128 tokens, no overlap | 25,210 | 127 | 0.985 | 0.597 | 0.794 | 0.778 | 0.833 [0.816, 0.849] | 0.877
128 tokens, 26 overlap (20%) | 31,455 | 127 | 1.000 | 0.636 | 0.847 | 0.827 | 0.883 [0.869, 0.897] | 0.917
256 tokens, no overlap | 12,727 | 251 | 0.993 | 0.634 | 0.853 | 0.753 | 0.831 [0.814, 0.848] | 0.885
256 tokens, 51 overlap (20%) | 15,719 | 253 | 1.000 | 0.677 | 0.870 | 0.781 | 0.848 [0.833, 0.864] | 0.908
512 tokens, no overlap | 6,487 | 493 | 0.997 | 0.666 | 0.879 | 0.666 | 0.787 [0.769, 0.805] | 0.864
512 tokens, 102 overlap (20%) | 7,914 | 499 | 1.000 | 0.684 | 0.888 | 0.684 | 0.800 [0.782, 0.817] | 0.871
1024 tokens, no overlap | 3,367 | 949 | 0.999 | 0.684 | 0.895 | n/a (a full chunk does not fit) | 0.684 [0.664, 0.705] | 0.811
1024 tokens, 205 overlap (20%) | 4,025 | 974 | 1.000 | 0.685 | 0.896 | n/a (a full chunk does not fit) | 0.685 [0.665, 0.706] | 0.802
paragraphs as written (the article's own boundaries) | 20,963 | 152 | 1.000 | 0.699 | 0.872 | 0.807 | 0.884 [0.870, 0.898] | 0.921
paragraphs as written, article title prepended | 20,963 | 156 | 1.000 | 0.705 | 0.882 | 0.812 | 0.892 [0.878, 0.905] | 0.929
256 tokens including a prepended article title, no overlap | 12,906 | 251 | 0.995 | 0.656 | 0.858 | 0.770 | 0.837 [0.820, 0.852] | 0.893
paired difference in the 1,024-token budget hit rate against 256 tokens, no overlap (95% bootstrap interval):
  64 tokens, no overlap | -0.030 [-0.049, -0.011] | lower
  64 tokens, 13 overlap (20%) | +0.035 [+0.018, +0.052] | higher
  128 tokens, no overlap | +0.002 [-0.013, +0.015] | interval includes 0
  128 tokens, 26 overlap (20%) | +0.052 [+0.038, +0.067] | higher
  256 tokens, 51 overlap (20%) | +0.017 [+0.004, +0.030] | higher
  512 tokens, no overlap | -0.043 [-0.059, -0.029] | lower
  512 tokens, 102 overlap (20%) | -0.031 [-0.047, -0.015] | lower
  1024 tokens, no overlap | -0.147 [-0.168, -0.128] | lower
  1024 tokens, 205 overlap (20%) | -0.145 [-0.166, -0.125] | lower
  paragraphs as written (the article's own boundaries) | +0.053 [+0.039, +0.067] | higher
  paragraphs as written, article title prepended | +0.061 [+0.049, +0.075] | higher
  256 tokens including a prepended article title, no overlap | +0.005 [-0.008, +0.019] | interval includes 0

## Retriever: hybrid bm25 + bge-small
chunking | chunks | mean tokens | answer span inside one chunk | hit@1 chunk | hit@5 chunks | answer in a 512-token budget | in 1,024 tokens [95% CI] | in 2,048 tokens
64 tokens, no overlap | 50,166 | 64 | 0.967 | 0.537 | 0.753 | 0.800 | 0.847 [0.831, 0.862] | 0.883
64 tokens, 13 overlap (20%) | 62,760 | 64 | 0.999 | 0.597 | 0.831 | 0.871 | 0.915 [0.903, 0.927] | 0.948
128 tokens, no overlap | 25,210 | 127 | 0.985 | 0.584 | 0.821 | 0.804 | 0.855 [0.839, 0.870] | 0.902
128 tokens, 26 overlap (20%) | 31,455 | 127 | 1.000 | 0.628 | 0.863 | 0.837 | 0.897 [0.883, 0.910] | 0.936
256 tokens, no overlap | 12,727 | 251 | 0.993 | 0.572 | 0.835 | 0.701 | 0.807 [0.790, 0.825] | 0.873
256 tokens, 51 overlap (20%) | 15,719 | 253 | 1.000 | 0.595 | 0.846 | 0.726 | 0.827 [0.810, 0.844] | 0.893
512 tokens, no overlap | 6,487 | 493 | 0.997 | 0.537 | 0.827 | 0.537 | 0.691 [0.670, 0.711] | 0.796
512 tokens, 102 overlap (20%) | 7,914 | 499 | 1.000 | 0.584 | 0.834 | 0.584 | 0.716 [0.696, 0.736] | 0.806
1024 tokens, no overlap | 3,367 | 949 | 0.999 | 0.459 | 0.766 | n/a (a full chunk does not fit) | 0.459 [0.437, 0.480] | 0.616
1024 tokens, 205 overlap (20%) | 4,025 | 974 | 1.000 | 0.500 | 0.785 | n/a (a full chunk does not fit) | 0.500 [0.478, 0.522] | 0.630
paragraphs as written (the article's own boundaries) | 20,963 | 152 | 1.000 | 0.698 | 0.891 | 0.832 | 0.899 [0.886, 0.912] | 0.939
paragraphs as written, article title prepended | 20,963 | 156 | 1.000 | 0.707 | 0.889 | 0.831 | 0.899 [0.886, 0.912] | 0.938
256 tokens including a prepended article title, no overlap | 12,906 | 251 | 0.995 | 0.579 | 0.835 | 0.702 | 0.810 [0.793, 0.827] | 0.877
paired difference in the 1,024-token budget hit rate against 256 tokens, no overlap (95% bootstrap interval):
  64 tokens, no overlap | +0.039 [+0.020, +0.058] | higher
  64 tokens, 13 overlap (20%) | +0.108 [+0.090, +0.126] | higher
  128 tokens, no overlap | +0.048 [+0.032, +0.063] | higher
  128 tokens, 26 overlap (20%) | +0.089 [+0.073, +0.105] | higher
  256 tokens, 51 overlap (20%) | +0.019 [+0.004, +0.035] | higher
  512 tokens, no overlap | -0.117 [-0.135, -0.098] | lower
  512 tokens, 102 overlap (20%) | -0.091 [-0.111, -0.073] | lower
  1024 tokens, no overlap | -0.349 [-0.372, -0.326] | lower
  1024 tokens, 205 overlap (20%) | -0.307 [-0.330, -0.283] | lower
  paragraphs as written (the article's own boundaries) | +0.092 [+0.076, +0.108] | higher
  paragraphs as written, article title prepended | +0.091 [+0.076, +0.107] | higher
  256 tokens including a prepended article title, no overlap | +0.003 [-0.013, +0.018] | interval includes 0

## Retriever: dense gte-modernbert-base (8,192-token window)
chunking | chunks | mean tokens | answer span inside one chunk | hit@1 chunk | hit@5 chunks | answer in a 512-token budget | in 1,024 tokens [95% CI] | in 2,048 tokens
256 tokens, no overlap | 12,727 | 251 | 0.993 | 0.517 | 0.784 | 0.654 | 0.754 [0.735, 0.772] | 0.837
512 tokens, no overlap | 6,487 | 493 | 0.997 | 0.494 | 0.781 | 0.494 | 0.644 [0.623, 0.664] | 0.757
1024 tokens, no overlap | 3,367 | 949 | 0.999 | 0.462 | 0.777 | n/a (a full chunk does not fit) | 0.462 [0.441, 0.484] | 0.611
paired difference in the 1,024-token budget hit rate against 256 tokens, no overlap (95% bootstrap interval):
  512 tokens, no overlap | -0.110 [-0.128, -0.092] | lower
  1024 tokens, no overlap | -0.292 [-0.315, -0.270] | lower

## Truncation: 1,024-token chunks, hit@5 by where the answer sits inside its chunk
bge-small-en-v1.5 reads at most 512 tokens (510 of text); gte-modernbert-base reads 8,192
dense bge-small | answer in the first 510 tokens of its chunk: 0.738 (n=1058) | answer after token 510: 0.395 (n=939)
bm25 | answer in the first 510 tokens of its chunk: 0.892 (n=1058) | answer after token 510: 0.902 (n=939)
hybrid bm25 + bge-small | answer in the first 510 tokens of its chunk: 0.871 (n=1058) | answer after token 510: 0.651 (n=939)
dense gte-modernbert-base (8,192-token window) | answer in the first 510 tokens of its chunk: 0.812 (n=1058) | answer after token 510: 0.740 (n=939)

run-latency-1.txt

Unedited output of latency pass 1.

# rag-bench latency pass: one query at a time
date (UTC): 2026-11-16
machine: Apple M1 Max, 64 GB, macOS 27.2.1
python 3.12.14, torch 2.15.0 (mps available: True), sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5
bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector (Python client) 0.5.1
seed: 20261116
100 seeded queries per dataset after 5 warm-ups; median and p95 (nearest rank) in ms; batch size 1 for query embedding, one batch of N pairs for reranking
started 16:23:07 local time

## scifact: 5,183 documents
stage | median ms | p95 ms
bm25 search, top 100 | 0.12 | 0.18
embed query, minilm | 5.31 | 6.70
exact vector search in numpy, minilm (5,183 x 384) | 0.10 | 0.12
embed query, bge-small | 9.52 | 13.58
exact vector search in numpy, bge-small (5,183 x 384) | 0.11 | 0.12
rrf fusion of two top-100 lists | 0.05 | 0.05
embed query, bge-base | 11.05 | 13.31
exact vector search in numpy, bge-base (5,183 x 768) | 0.31 | 0.33
embed query, gte-mbert | 14.26 | 17.79
exact vector search in numpy, gte-mbert (5,183 x 768) | 0.31 | 0.35
rerank 10 candidates, ce-minilm | 35.91 | 38.34
rerank 20 candidates, ce-minilm | 70.27 | 79.57
rerank 50 candidates, ce-minilm | 161.71 | 171.85
rerank 10 candidates, bge-rr-base | 173.28 | 204.26
rerank 20 candidates, bge-rr-base | 346.23 | 353.75
rerank 50 candidates, bge-rr-base | 845.07 | 858.69

## nq: 60,000 documents
stage | median ms | p95 ms
bm25 search, top 100 | 0.42 | 0.63
embed query, minilm | 4.50 | 5.31
exact vector search in numpy, minilm (60,000 x 384) | 2.58 | 2.72
embed query, bge-small | 8.69 | 10.31
exact vector search in numpy, bge-small (60,000 x 384) | 2.54 | 2.72
rrf fusion of two top-100 lists | 0.05 | 0.05
embed query, bge-base | 10.03 | 11.42
exact vector search in numpy, bge-base (60,000 x 768) | 5.60 | 5.79
embed query, gte-mbert | 12.48 | 13.76
exact vector search in numpy, gte-mbert (60,000 x 768) | 5.49 | 7.63
rerank 10 candidates, ce-minilm | 19.67 | 28.56
rerank 20 candidates, ce-minilm | 36.97 | 58.18
rerank 50 candidates, ce-minilm | 105.45 | 157.67
rerank 10 candidates, bge-rr-base | 88.34 | 142.83
rerank 20 candidates, bge-rr-base | 215.10 | 328.06
rerank 50 candidates, bge-rr-base | 656.14 | 822.55

## squad: 20,963 documents
stage | median ms | p95 ms
bm25 search, top 100 | 0.20 | 0.30
embed query, minilm | 4.86 | 6.31
exact vector search in numpy, minilm (20,963 x 384) | 0.71 | 0.78
embed query, bge-small | 8.70 | 9.18
exact vector search in numpy, bge-small (20,963 x 384) | 0.69 | 0.77
rrf fusion of two top-100 lists | 0.05 | 0.05
embed query, bge-base | 10.18 | 11.15
exact vector search in numpy, bge-base (20,963 x 768) | 1.79 | 1.89
embed query, gte-mbert | 12.95 | 14.35
exact vector search in numpy, gte-mbert (20,963 x 768) | 1.82 | 1.93
rerank 10 candidates, ce-minilm | 22.35 | 30.77
rerank 20 candidates, ce-minilm | 41.23 | 57.59
rerank 50 candidates, ce-minilm | 110.16 | 146.20
rerank 10 candidates, bge-rr-base | 105.06 | 155.26
rerank 20 candidates, bge-rr-base | 227.79 | 344.69
rerank 50 candidates, bge-rr-base | 712.07 | 849.63

finished 16:30:43 local time

run-latency-2.txt

Unedited output of latency pass 2.

# rag-bench latency pass: one query at a time
date (UTC): 2026-11-16
machine: Apple M1 Max, 64 GB, macOS 27.2.1
python 3.12.14, torch 2.15.0 (mps available: True), sentence-transformers 6.3.0, transformers 5.23.0, numpy 2.5.5
bm25s 0.3.16, PyStemmer 3.1.0, psycopg 3.3.9, pgvector (Python client) 0.5.1
seed: 20261116
100 seeded queries per dataset after 5 warm-ups; median and p95 (nearest rank) in ms; batch size 1 for query embedding, one batch of N pairs for reranking
started 16:30:47 local time

## scifact: 5,183 documents
stage | median ms | p95 ms
bm25 search, top 100 | 0.12 | 0.16
embed query, minilm | 5.15 | 6.33
exact vector search in numpy, minilm (5,183 x 384) | 0.11 | 0.13
embed query, bge-small | 8.66 | 9.51
exact vector search in numpy, bge-small (5,183 x 384) | 0.11 | 0.13
rrf fusion of two top-100 lists | 0.05 | 0.05
embed query, bge-base | 10.41 | 11.65
exact vector search in numpy, bge-base (5,183 x 768) | 0.31 | 0.35
embed query, gte-mbert | 13.52 | 17.98
exact vector search in numpy, gte-mbert (5,183 x 768) | 0.31 | 0.33
rerank 10 candidates, ce-minilm | 34.67 | 36.62
rerank 20 candidates, ce-minilm | 64.03 | 70.65
rerank 50 candidates, ce-minilm | 158.32 | 165.17
rerank 10 candidates, bge-rr-base | 172.90 | 200.39
rerank 20 candidates, bge-rr-base | 346.13 | 353.50
rerank 50 candidates, bge-rr-base | 866.77 | 883.57

## nq: 60,000 documents
stage | median ms | p95 ms
bm25 search, top 100 | 0.42 | 0.63
embed query, minilm | 4.51 | 5.39
exact vector search in numpy, minilm (60,000 x 384) | 2.58 | 2.76
embed query, bge-small | 8.59 | 9.29
exact vector search in numpy, bge-small (60,000 x 384) | 2.57 | 2.76
rrf fusion of two top-100 lists | 0.05 | 0.05
embed query, bge-base | 9.96 | 11.13
exact vector search in numpy, bge-base (60,000 x 768) | 5.59 | 5.79
embed query, gte-mbert | 12.42 | 13.74
exact vector search in numpy, gte-mbert (60,000 x 768) | 5.47 | 5.76
rerank 10 candidates, ce-minilm | 20.18 | 28.24
rerank 20 candidates, ce-minilm | 37.61 | 58.64
rerank 50 candidates, ce-minilm | 111.02 | 160.73
rerank 10 candidates, bge-rr-base | 91.52 | 162.09
rerank 20 candidates, bge-rr-base | 227.02 | 343.14
rerank 50 candidates, bge-rr-base | 680.53 | 854.20

## squad: 20,963 documents
stage | median ms | p95 ms
bm25 search, top 100 | 0.20 | 0.29
embed query, minilm | 4.68 | 5.84
exact vector search in numpy, minilm (20,963 x 384) | 0.71 | 0.78
embed query, bge-small | 8.70 | 9.55
exact vector search in numpy, bge-small (20,963 x 384) | 0.71 | 0.78
rrf fusion of two top-100 lists | 0.06 | 0.06
embed query, bge-base | 10.09 | 10.87
exact vector search in numpy, bge-base (20,963 x 768) | 1.80 | 1.89
embed query, gte-mbert | 12.76 | 14.15
exact vector search in numpy, gte-mbert (20,963 x 768) | 1.83 | 1.91
rerank 10 candidates, ce-minilm | 22.14 | 31.44
rerank 20 candidates, ce-minilm | 40.88 | 57.08
rerank 50 candidates, ce-minilm | 117.29 | 147.87
rerank 10 candidates, bge-rr-base | 113.76 | 172.26
rerank 20 candidates, bge-rr-base | 240.02 | 352.60
rerank 50 candidates, bge-rr-base | 755.83 | 915.42

finished 16:38:35 local time

Frequently asked questions

Is hybrid search better than vector search for RAG?

It depended on the dataset in our test. Reciprocal rank fusion of BM25 and bge-small-en-v1.5, against bge-small alone, had a mean nDCG@10 0.090 higher on SQuAD (95% interval +0.070 to +0.109), 0.037 lower on a Natural Questions sample (interval -0.056 to -0.019) and no measurable difference on SciFact (+0.015, interval -0.006 to +0.035). On SQuAD the hybrid (0.823) did not beat BM25 alone (0.828); that pair had no paired comparison. Measure it on labelled queries from your own corpus.

Is a cross-encoder reranker worth the added latency?

Only on some data in our test. Reranking the top 50 of a BM25 plus bge-small hybrid with the ms-marco-MiniLM-L6-v2 cross-encoder (22.7M parameters) had a mean nDCG@10 0.087 higher on SQuAD, 0.061 higher on a Natural Questions sample and 0.031 lower on SciFact. Scoring 50 candidates for one query took a median of 105.45 to 161.71 ms depending on the dataset on an Apple M1 Max GPU, and 656.14 to 845.07 ms with bge-reranker-base (latency pass 1). The bigger reranker was not safer: on the Natural Questions sample, bge-reranker-base on the hybrid top 50 scored 0.108 lower than no rerank, and 0.128 lower than reranking only the top 10.

What chunk size should I use for RAG?

Small chunks inside the embedding model's window did best in our SQuAD test at a fixed 1,024-token context budget. With bge-small-en-v1.5, the answer was in the budget for 0.864 of 2,000 questions with 64-token chunks and 20% overlap, 0.829 with the articles' own paragraphs, 0.634 with 256-token chunks and 0.296 with 1,024-token chunks. With 1,024-token chunks, bge-small (which reads 512 tokens) put the right chunk in its top 5 for 0.738 of questions whose answer sat in the first 510 tokens and 0.395 of those whose answer sat later. SQuAD questions were written by people reading one paragraph, which favours paragraph-sized units, so check on your own documents.

Why does pgvector return fewer rows than my LIMIT when I filter by tenant or metadata?

Because with an approximate index the filter runs after the index scan: the pgvector README says that if a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average. On pgvector 0.8.7 and PostgreSQL 18.7 with 500,000 vectors, asking for 10 rows returned a mean of 1.92 for a tenant holding 5% of them, 0.45 for 1% and 0.04 for 0.1%, with no error. Setting hnsw.iterative_scan to relaxed_order returned 10 rows on every query for the 5% and 1% tenants and on 499 of 500 for the 0.1% tenant, at a median of 25.73 ms server time for the 1% tenant; a B-tree on tenant_id let the planner search the 0.1% tenant exactly in 0.30 ms.

Is pgvector good enough for production RAG, or do I need a dedicated vector database?

We tested only pgvector, so we make no comparison. At 500,000 vectors of 384 dimensions on one laptop, pgvector 0.8.7's HNSW index answered an unfiltered top 10 in a median of 0.69 ms server time at the default ef_search of 40 (index recall 0.9175 against exact search) and 3.95 ms at ef_search 200 (0.9777). Filtered queries needed work: iterative scans, a B-tree on the filter column or partial indexes. The operational argument for Postgres is one database, with transactions and row-level security, beside the rows your vectors describe.

Do I still need RAG with long-context models?

For small, stable knowledge bases, often not: Anthropic's guidance is that a knowledge base under about 200,000 tokens can go straight into the prompt. Retrieval still earns its place when the corpus is larger than that, changes often, or must be filtered by what each user is allowed to see, and Liu et al. found that models use information in the middle of long inputs less well than information at the start or end. We did not measure long-context answering.

What about agentic RAG and GraphRAG?

We measured neither. Agentic RAG lets a model decide when and what to retrieve, which turns retrieved documents into input to a system that can act, so design it around the lethal trifecta described in our post on AI agents in SaaS. GraphRAG (Edge et al., 2024) uses an LLM to build an entity graph and community summaries for questions about a whole corpus, which plain passage retrieval handles poorly.

Next articlePostgres RLS Performance: What Makes a Policy Slow, Measured on PostgreSQL 18.7

Get in touch.

Whether you have questions or just want to explore what's possible, we're here to help.