همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

Implementation of a Retrieval-Augmented Generation-Based Question Answering System Using Large Language Models

نویسندگان :

Mohammad Mahdi HajiRamezanAli¹ Mohammad Mehdi Homayounpour²

1- موسسه آموزش عالی شهاب دانش قم 2- موسسه آموزش عالی شهاب دانش قم

کلمات کلیدی :

Retrieval augmented generation،RAG،Chatbot،Large language models،Out of domain detection،Out-of-scope detection

چکیده :

Question-answering (QA) systems, as a major subfield of natural language processing, play a crucial role in the development of search engines, virtual assistants, customer service platforms, and educational tools. This requirement is especially important for Persian, a language that faces limitations in training data and computational resources; In this study, we designed and implemented a hybrid framework based on large language models (LLMs) and retrieval-augmented methods. First, the architecture of the QA system was examined using several LLMs and their performances were compared. Next, the system’s ability to respond multi-step and complex questions was evaluated to assess its effectiveness under more realistic conditions. To increase system reliability, we developed a mechanism for identifying irrelevant (out-of-domain) queries that leverages model-generated data and a lightweight neural network. Experimental results indicate that “Qwen2-7B-Instruct”, when combined with retrieval augmentation and ranker models, achieved the best performance among the models considered. Furthermore, the proposed out-of-domain detection mechanism substantially improved the system’s performance outperforming humans by 9.51% in accuracy metric and prevented generation of irrelevant responses.