The Arabic RAG Leaderboard

Evaluation Status

About Retrieval Evaluation

The retrieval evaluation assesses a model's ability to find and retrieve relevant information from a large corpus of Arabic text. Models are evaluated on:

Web Search Dataset Metrics

• MRR (Mean Reciprocal Rank): Measures the ranking quality by focusing on the position of the first relevant result
• nDCG (Normalized Discounted Cumulative Gain): Evaluates the ranking quality considering all relevant results
• Recall@5: Measures the proportion of relevant documents found in the top 5 results
• Overall Score: Combined score calculated as the average of MRR, nDCG, and Recall@5

Model Requirements

• Must support Arabic text embeddings
• Should handle queries of at least 512 tokens
• Must work with sentence-transformers library

Evaluation Process

1. Models process Arabic web search queries
2. Retrieved documents are evaluated using:
   • MRR for first relevant result positioning
   • nDCG for overall ranking quality
   • Recall@5 for top results accuracy
3. Metrics are averaged to calculate the overall score
4. Models are ranked based on their overall performance

How to Prepare Your Model

• Ensure your model is publicly available on HuggingFace Hub (We don't support private model evaluations yet)
• Model should output fixed-dimension embeddings for text
• Support batch processing for efficient evaluation (this is default if you use sentence-transformers)

Evaluation Status

About Reranking Evaluation

The reranking evaluation assesses a model's ability to improve search quality by reordering initially retrieved results. Models are evaluated across multiple unseen Arabic datasets to ensure robust performance.

Evaluation Metrics

• MRR@10 (Mean Reciprocal Rank at 10): Measures the ranking quality focusing on the first relevant result in top-10
• NDCG@10 (Normalized DCG at 10): Evaluates the ranking quality of all relevant results in top-10
• MAP (Mean Average Precision): Measures the overall precision across all relevant documents

All metrics are averaged across multiple evaluation datasets to provide a comprehensive assessment of model performance.

Model Requirements

• Must accept query-document pairs as input
• Should output relevance scores for reranking (has cross-attention or similar mechanism for query-document matching)
• Support for Arabic text processing

Evaluation Process

1. Models are tested on multiple unseen Arabic datasets
2. For each dataset:
   • Initial candidate documents are provided
   • Model reranks the candidates
   • MRR@10, NDCG@10, and MAP are calculated
3. Final scores are averaged across all datasets
4. Models are ranked based on overall performance

How to Prepare Your Model

• Model should be public on HuggingFace Hub (private models are not supported yet)
• Make sure it works coherently with sentence-transformers library