The Arabic RAG Leaderboard
The only leaderboard you will require for your RAG needs 🏆
For technical details, check our blog post here.
10 | 64.75 | 1,067.91 | 8,192 | 1,024 | 70.35 | 59.15 |
Precision
Evaluation Status
Omartificial-Intelligence-Space/Arabic-MiniLM-L12-v2-all-nli-triplet | apache-2.0 | main | f16 | 305 |
No failed evaluations.
About Retrieval Evaluation
The retrieval evaluation assesses a model's ability to find and retrieve relevant information from a large corpus of Arabic text. Models are evaluated on:
Web Search Dataset Metrics
- MRR (Mean Reciprocal Rank): Measures the ranking quality by focusing on the position of the first relevant result
- nDCG (Normalized Discounted Cumulative Gain): Evaluates the ranking quality considering all relevant results
- Recall@5: Measures the proportion of relevant documents found in the top 5 results
- Overall Score: Combined score calculated as the average of MRR, nDCG, and Recall@5
Model Requirements
- Must support Arabic text embeddings
- Should handle queries of at least 512 tokens
- Must work with
sentence-transformerslibrary
Evaluation Process
- Models process Arabic web search queries
- Retrieved documents are evaluated using:
- MRR for first relevant result positioning
- nDCG for overall ranking quality
- Recall@5 for top results accuracy
- Metrics are averaged to calculate the overall score
- Models are ranked based on their overall performance
How to Prepare Your Model
- Ensure your model is publicly available on HuggingFace Hub (We don't support private model evaluations yet)
- Model should output fixed-dimension embeddings for text
- Support batch processing for efficient evaluation (this is default if you use
sentence-transformers)
10 | 87.44 | 2,165.81 | 8,192 | 1,024 | 81.27 | 87.05 |
Precision
Evaluation Status
Omartificial-Intelligence-Space/Arabic-MiniLM-L12-v2-all-nli-triplet | cc-by-nc-4.0 | main | f16 | 306 |
jinaai/jina-embeddings-v3 | cc-by-nc-4.0 | main | bf16 | 572 |
About Reranking Evaluation
The reranking evaluation assesses a model's ability to improve search quality by reordering initially retrieved results. Models are evaluated across multiple unseen Arabic datasets to ensure robust performance.
Evaluation Metrics
- MRR@10 (Mean Reciprocal Rank at 10): Measures the ranking quality focusing on the first relevant result in top-10
- NDCG@10 (Normalized DCG at 10): Evaluates the ranking quality of all relevant results in top-10
- MAP (Mean Average Precision): Measures the overall precision across all relevant documents
All metrics are averaged across multiple evaluation datasets to provide a comprehensive assessment of model performance.
Model Requirements
- Must accept query-document pairs as input
- Should output relevance scores for reranking (has cross-attention or similar mechanism for query-document matching)
- Support for Arabic text processing
Evaluation Process
- Models are tested on multiple unseen Arabic datasets
- For each dataset:
- Initial candidate documents are provided
- Model reranks the candidates
- MRR@10, NDCG@10, and MAP are calculated
- Final scores are averaged across all datasets
- Models are ranked based on overall performance
How to Prepare Your Model
- Model should be public on HuggingFace Hub (private models are not supported yet)
- Make sure it works coherently with
sentence-transformerslibrary