📌 Highlights
ZeroRAG targets a structural limitation of RAG: documents may remain distributed across organizations, devices, or individuals rather than being copied into a central vector database. Its answer is a local-only semantic routing layer that keeps ownership and access decisions at the peers holding the documents.
The strongest evidence is breadth across workloads and operating conditions. ZeroRAG reaches 0.91 final AOL-IA recall@1 versus 0.65 for Semantica and 0.80 for DHT-LSH with 50 hash tables; on Amazon Electronics it retrieves 94.8–98.9% of available relevant products; and on PubMedQA it reaches 93.0% recall and 77% RAG accuracy versus 97.3% and 79% for centralized HNSW.
The key limitation is the cost and uncertainty of decentralization: ZeroRAG requires 379 ms/query excluding network latency versus 24 ms/query for HNSW, offers best-effort routing without global guarantees, and relies on assumptions about shared embeddings and honest peers.
- Local state only: bounded advertisements, neighbors, and routed-query memory.
- Best-first semantic walks progressively rewire the overlay.
- 99% of final PubMedQA queries find the best supporting document within 13 requests.
- Recall recovers to 95.5% after simultaneous failure of 50% of peers.
- The protocol is not a complete privacy or security solution.
🎯 Introduction
RAG improves factual reliability by retrieving external documents and placing them in an LLM prompt, but conventional systems assume that documents can be collected, indexed, and served from a centrally controlled vector store. The paper argues that this assumption is unsuitable when data is constrained by privacy, copyright, regulation, ownership, or operational boundaries.
ZeroRAG addresses two related problems: finding semantically relevant peers without broadcasting or a global index, and keeping peer neighborhoods useful as documents, queries, and membership change. The paper frames the design as analogous to incremental routing protocols: local observations update connections, and useful paths emerge from traffic rather than from network-wide coordination.
The paper's objective is specifically the retriever component of RAG. It does not attempt to replace local vector databases or define the downstream document-transfer and attribution layer; instead, it augments local retrieval with remote peer results and referrals.
🔬 Methodology
Each peer maintains local documents D, a FIFO neighbor list N with capacity δn, advertisements A received from neighbors, and a FIFO routed-query memory M with capacity δm. Advertisements summarize document embeddings that peers choose to expose. When a query arrives, the receiving peer compares the query embedding with both advertised document embeddings and cached query embeddings, allowing referrals to reflect observed query traffic as well as local inventory.
The querying peer maintains a frontier of candidate peers. It selects the highest-scoring candidate, sends a query containing the embedding and requested result count, merges the returned document embeddings, and adds scored referrals to the frontier. After the search, it connects to the peer associated with the best result and removes the oldest neighbor if the bounded neighborhood is full. The protocol supports asynchronous or dropped messages through timeouts and removes state associated with departed peers.
The paper gives communication and state bounds for fixed-size embeddings. It also notes that sequential search is used in the experiments to avoid variance in the reported metrics, while parallel fanout is a possible latency optimization whose effects are evaluated separately.
Persistent peer state
Meaning: This tuple defines the local state used for routing: documents, outgoing neighbors, neighbor advertisements, and routed-query memory.
Query similarity used by caching
Meaning: Semantically similar queries can cause a receiving peer to refer a later querier toward a peer associated with an earlier query.
Per-query communication bound
Meaning: This bounds the payload carried by a search under contact budget C, result limit k, embedding dimension d, and referral limit r.
Persistent routing-state bound
Meaning: This bounds routing state using neighbor capacity, advertisement capacity, query-memory capacity, and embedding dimension.
Retrieving top-k documents for query q of peer u
Steps: ['Embed the query and initialize the frontier, contacted-peer set, result set, and best referral scores.', 'Perform a best-first walk for at most C contacts, merging results and adding referrals after each response.', 'Rewire one neighbor toward the peer returning the strongest result and evict the oldest neighbor if needed.', 'Retrieve the selected documents from their owning peers.']
Why It Matters: This is the main decentralized retrieval and overlay-maintenance procedure.
Query handling at peer p
Steps: ['Search local documents.', 'Score advertisements and cached queries against the incoming embedding.', 'Return local results and the top-r peer referrals.', 'Cache the incoming peer-query pair with bounded FIFO eviction.']
Why It Matters: This procedure turns each peer into both a local retriever and a routing intermediary.
📊 Experiments
The evaluation uses three real-world workloads. AOL-IA contains query-and-click events and is evaluated as single-target retrieval; Amazon Electronics contains category-derived product queries with multiple relevant products; and PubMedQA contains 1,000 biomedical questions, each expanded with five GPT-5-nano rephrasings. PubMedQA is also used for end-to-end RAG with Phi3.5-mini-instruct. The main decentralized baselines are Semantica, which uses a globally maintained semantic trie, and DHT-LSH, which combines locality-sensitive hashing with DHT lookups. Centralized HNSW is used as an upper-bound-like comparison for end-to-end PubMedQA.
Recall is the primary retrieval metric, with higher values better. Request count and message overhead measure routing efficiency, with lower values better. On AOL-IA, ZeroRAG reaches 0.91 final recall@1 compared with 0.65 for Semantica and 0.80 for DHT-LSH with 50 hash tables. On Amazon Electronics, ZeroRAG retrieves 94.8–98.9% of available relevant products, compared with 90.3–91.7% for DHT-LSH with 50 tables. The paper estimates approximately 664 return messages for a 10,000-peer DHT-LSH search versus 50 for ZeroRAG at comparable settings.
For PubMedQA, centralized HNSW reaches 97.3% recall and 79% answer accuracy, while ZeroRAG reaches 93.0% recall and 77% answer accuracy. ZeroRAG's final routing state lets 99% of queries find the best supporting document within 13 requests, but its average query duration is 379 ms excluding network latency versus 24 ms/query for HNSW.
The ablations support the role of routed-query memory more strongly than topology adaptation alone. With no query memory, recall falls to 77.860% and requests rise to 16.796; a cache of 100 reaches 97.612% recall and 5.900 requests, while 1,000 reaches 99.652% recall and 3.448 requests. After a 50% catastrophic peer failure, recall recovers to 95.5% in the final 10% of the run, although post-failure requests rise to 15.1 versus 8.9 before failure.
tab:pubmedqa-query-memory-limit
Caption: The effect of query cache size.
| Max. cache size (δm) | Avg. Recall@1 | Avg. requests required |
|---|---|---|
| 0 | (77.860 ± 0.177)% | 16.796 ± 0.067 |
| 10 | (85.834 ± 0.120)% | 12.757 ± 0.041 |
| 100 | (97.612 ± 0.037)% | 5.900 ± 0.017 |
| 1,000 | (99.652 ± 0.007)% | 3.448 ± 0.006 |
| 10,000 | (99.652 ± 0.007)% | 3.448 ± 0.006 |
Why It Matters: It quantifies the cache-size trade-off.
tab:pubmedqa-user-scalability
Caption: The effect of network size.
| Network size (peers) | Avg. Recall@1 | Avg. requests required |
|---|---|---|
| 1,000 | (99.365 ± 0.272)% | 3.209 ± 0.111 |
| 5,000 | (98.276 ± 0.066)% | 4.330 ± 0.059 |
| 10,000 | (98.378 ± 0.051)% | 4.656 ± 0.015 |
| 50,000 | (99.652 ± 0.007)% | 3.448 ± 0.006 |
| 100,000 | (99.913 ± 0.005)% | 5.724 ± 0.011 |
Why It Matters: It shows high conditional recall across network sizes up to 100,000 peers.
tab:pubmedqa-doc-scalability
Caption: The effect of document count per peer.
| Docs. per peer (t) | Avg. Recall@1 | Avg. requests required |
|---|---|---|
| 10 | (98.665 ± 0.018)% | 5.940 ± 0.023 |
| 20 | (99.292 ± 0.016)% | 5.545 ± 0.017 |
| 30 | (99.667 ± 0.007)% | 5.361 ± 0.012 |
| 40 | (99.836 ± 0.006)% | 5.276 ± 0.009 |
| 50 | (99.919 ± 0.004)% | 5.222 ± 0.007 |
Why It Matters: It measures the effect of document density per peer.
fig:rag
Caption: The standard RAG pipeline: offline indexing (left) encodes documents into a vector database; online retrieval (right) matches a query to relevant chunks before LLM answer generation.
Why It Matters: This establishes the centralized RAG workflow and identifies retrieval as the component replaced by ZeroRAG.
fig:zerorag
Caption: (left) Example of 4 query steps of a document searched by peer $a$ with query $q$. After the search concludes, peer $a$ adds peer $i$, which has the most relevant document to $a$'s query, as neighbor. (right) The dynamics of query caching in .
Why It Matters: It visualizes the best-first walk, adaptive neighbor formation, and query-cache referral mechanism.
fig:aolia
Caption: Average recall@1 for query batches, for and different baseline configurations.
Why It Matters: It provides the main AOL-IA retrieval comparison across ZeroRAG, Semantica, and DHT-LSH configurations.
fig:pubmed_recall_requests
Caption: (a) The end-to-end RAG performance , using and HNSW. (b) the distribution of number of requests before the first relevant document.
Why It Matters: It connects routing quality to end-to-end answer accuracy and shows how request counts fall as the overlay adapts.
fig:churn
Caption: Average recall@1 per processed query batch in with the workload, when there is a catastrophic failure event (dashed red line).
Why It Matters: It shows the recall collapse and recovery pattern after half of the peers leave simultaneously.
fig:ablation-inputs
Caption: Ablation of query memory and adaptive topology on the (a) and (b) workloads.
Why It Matters: It isolates the contribution of routed-query memory and adaptive topology to retrieval performance.
🔮 Conclusion
The main result is that semantic routing with only local state can approach centralized retrieval quality while retaining peer-level document custody. Across the reported workloads, ZeroRAG improves recall over the decentralized baselines, reduces routing messages relative to DHT-LSH, and translates retrieval quality into near-centralized PubMedQA answer accuracy.
The evidence also supports an adaptive-systems claim: routed-query memory and neighbor rewiring allow performance to improve as queries flow through the network, and replacement peers can rebuild useful state after catastrophic failure. These benefits come with higher latency, best-effort guarantees, and dependence on the quality and comparability of exchanged embeddings.
🛠️ Future Research Improvements
The next research step should be evaluation beyond the simulated global connection graph. Real deployments need measurements that include network latency, document transfer, timeouts, peer availability, asynchronous message loss, and the cost of maintaining advertisements as documents change.
Security and privacy deserve first-class treatment. The current system model excludes Sybil, Eclipse, and Byzantine behavior, while the method acknowledges that embeddings and referrals may leak information. Future work should test authenticated advertisements, malicious routing, access-control policies, and defenses against poisoned or stale semantic metadata.
The paper also leaves open whether topology can converge under realistic heterogeneous interests and bounded neighbor capacity. Longer-lived workloads with changing topic distributions, stronger concept drift, and explicit comparisons between sequential and parallel search would clarify where adaptation helps and where it saturates.
🏭 Potential Industry Use Scenarios
A federation of hospitals could keep patient records within institutional boundaries while exposing controlled semantic advertisements and routing questions toward peers likely to hold relevant cases. Such a deployment would require strict identity, purpose, and query-based access control because the paper explicitly treats privacy-preserving retrieval as outside the protocol itself.
Research institutions or corporate divisions could use ZeroRAG to search across proprietary datasets without copying every document into a shared index. The same pattern could apply to distributed technical documentation, regulated archives, or device-local knowledge bases where ownership and operational boundaries prevent central ingestion.
The architecture is most credible for workloads that tolerate progressive, approximate retrieval and can trade extra latency for data locality. Centralized HNSW remains preferable when unified governance, low latency, and global availability are acceptable.
💬 Critical Analysis
ZeroRAG's strongest design choice is aligning routing adaptation with the data-ownership constraint: peers do not need to replicate documents or maintain a globally consistent semantic index. The combination of advertisements and query memory is technically meaningful, and the cache-size ablation shows a large performance change from a bounded local mechanism rather than from a centralized coordinator.
The main empirical claims are substantial but should be interpreted within the evaluation boundary. Recall is conditional on a relevant document being advertised by at least one remote peer in the scalability experiment, and the downstream RAG test evaluates answer-label accuracy rather than citation correctness or hallucination reduction. The simulated document store and HISTORY() shortcut also leave deployment costs unresolved.
The latency gap is the central practical concern: ZeroRAG reaches comparable quality through many peer interactions, but the reported 379 ms/query excludes network latency and exceeds centralized HNSW's 24 ms/query. Whether the improved governance and locality justify that cost depends on workload sensitivity, peer reliability, and the strength of access-control requirements.
Overall, the paper provides convincing evidence for adaptive decentralized routing under its stated assumptions, but production readiness would require real-network experiments, stronger threat modeling, privacy analysis, and validation that the learned topology remains useful under adversarial or rapidly changing semantic demand.