Development of Reflective Retrieval-Augmented Generation System

ReflectiveRAG: Rethinking Adaptivity in Retrieval-Augmented Generation
ReflectiveRAG uses a lightweight language-model controller to judge whether the retrieved evidence is sufficient and to reformulate the query when necessary. A contrastive embedding filter then removes redundant or weakly related passages, improving grounding under heavy retrieval noise with limited extra latency.
Proposed Project: Adaptive Reflective RAG with a Jury of Judges
We already have a working modified RAG system that combines techniques such as hybrid retrieval, reranking, and context compression. These methods improve retrieval accuracy and reduce irrelevant information. However, the current system follows a mostly fixed process and cannot decide for itself when the retrieved evidence is insufficient.
The next stage of the project is to develop a self-improving and reflective RAG system. A lightweight language-model controller will examine the retrieved documents before the final answer is generated. It will decide whether the evidence is relevant, complete, and sufficient. If important information is missing, the controller will rewrite the query and perform another search. A contrastive embedding filter will then remove duplicate, weakly related, or unnecessary passages. This should improve the quality of the evidence without adding too much processing time.
Instead of relying on a single model to evaluate the answer, the proposed system will use a jury of several independent judges. Each judge will assess factors such as factual accuracy, relevance, completeness, faithfulness to the retrieved evidence, and citation support. Their decisions will be combined through majority voting or a weighted agreement score. When the judges disagree strongly or have low confidence, the system can retrieve more evidence, revise the answer, or clearly state that it does not have enough information.
The system will also learn from previous retrieval and evaluation results. It will record which queries, retrieval methods, and evidence-selection strategies produced the strongest answers. Over time, this feedback can help the system choose better retrieval settings, improve query reformulation, and adjust the importance assigned to each judge. This makes the RAG pipeline more adaptive instead of applying the same strategy to every question.
The proposed workflow is:
Question → Hybrid retrieval → Reranking → Evidence compression → Reflection → Query reformulation if needed → Answer generation → Jury evaluation → Accept, revise, or abstain
The main novelty of this project is the combination of three components: reflective retrieval, adaptive improvement, and multi-judge evaluation. Reflective retrieval helps the system recognize missing evidence. Adaptive improvement allows it to learn from previous results. The jury reduces the risk of depending on one judge whose decisions may be biased or unreliable.
The proposed system can be evaluated by comparing the following versions:
1. The current RAG system
2. RAG with a single reflective judge
3. Reflective RAG with a jury of judges
4. Reflective RAG with both jury evaluation and adaptive learning
Retrieval performance can be measured using Recall@k, MRR@k, nDCG@k, and Complete@k. Answer quality can be evaluated using accuracy, faithfulness, completeness, hallucination rate, and citation correctness. Latency, computational cost, the number of retrieval rounds, and agreement between judges should also be measured. This comparison will show whether the proposed system improves reliability enough to justify its additional processing cost.