Benchmarking retrieval augmented generation LLMs for Arabic noise robustness.

Authors

Saleh Almohaimeed, Abdulrahman Alabduljabbar, Mousa Jari, Mohammed Alkhowaiter, Saad Almohaimeed, Mohamad Mahmoud Al Rahhal

Published in

Frontiers in big data. Volume 9. Pages 1884673. Epub Aug 17, 2026.

Abstract

Hallucination has become a serious concern in large language models (LLMs), as these models can generate useful yet incorrect or misleading information, which has led to growing research interest in retrieval-augmented generation (RAG) as a mitigation approach. RAG provides LLMs with access to external information, such as databases or documents, which can help them to answer users' questions. Currently, several benchmarks released measure RAG performance on various LLMs; however, evaluations of the noise robustness ability in Arabic are absent. In this paper, we systematically investigated the capabilities of state-of-the-art multilingual LLMs with regard to two essential RAG abilities, noise robustness and negative rejection. To accomplish this, we generated an Arabic benchmark consisting of 300 questions along with 6,196 documents. Then, we assessed the performance of six LLMs in relation to the two aforementioned RAG abilities. The results reveal that all six LLMs were negatively affected when the noise ratio in the external documents was increased. Under the highest noise settings at 80%, the best LLM performance was for Claude-4 sonnet, in which their performance decreased by only 4.67 percentage points. Furthermore, when it comes to the negative rejection task, there has been a significant impact on all six models. The best two models, Claude-4 sonnet and Llama-4, scored 90.67% and 85.67%, respectively, while smaller models, like GPT-3.5, scored 69.33%. Furthermore, our manual analysis reveals that many errors made by LLMs are attributed to over-caution behavior. LLMs often decline to respond probably due to training mechanisms designed to reduce hallucinations. Additionally, other errors occur when there is a high lexical similarity between the question and the words of noisy documents, which causes the model to rely on irrelevant content instead of the correct information.

PMID:
42676370
Bibliographic data and abstract were imported from PubMed on 01 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.