Abstract

During their training, Large Language Models (LLMs) learn their knowledge in the form of parameters. As a result, LLMs can be prone to hallucination (generation of misinformation) and have outdated information. They are also limited in terms of what they can understand and how well they can operate. Retrieval-Augmented Generation (RAG) is a proposed solution that aims to address these limitations. RAG includes an additional text generator that can query an external knowledge base to access and include up-to-date information in the model’s generated response. The paper includes 15 citations and covers the theory behind RAG. The authors have not designed or developed any new system, and they have not achieved any gains in system performance. Instead, they provide the definition and design of RAG and discuss the technologies associated with RAG including dense passage retrieval, retrieval-augmented pre-training, and decoding using a fused generator. The authors address the various methods for retrieval and context integration and assess the potential for RAG in the legal and medical domains, along with various design decisions. Finally, they list open issues and research directions. They conclude that retrieval methods have enabled better system performance in knowledge-based tasks; however, retrieval methods alone are not able to completely eliminate hallucination in system outputs. The identified issues have existing solutions such as multi-hop reasoning, selecting answers based on evidential documents, latency and trade-offs of system coverage and breadth, and adapting to a target domain. Nevertheless, there are still research problems that need to be addressed.

Keywords

How to Cite This Article

Rights