RAGChecker v0.1.9

Modify the required version of RefChecker.

RAGChecker v0.1.8

What's Changed

• saves final metrics to path by @aghassel in #13

New Contributors

• @aghassel made their first contribution in #13

Full Changelog: v0.1.7...v0.1.8

RAGChecker v0.1.7

Support customized function for LLM invoking.

RAGChecker v0.1.6

Customized function to get response for sagemaker.

RAGChecker v0.1.5

Additional params for Sagemaker

RAGChecker v0.1.4

Add support of AWS Sagemaker to avoid the bugs in litellm.

RAGChecker v0.1.3

Add "**kwargs" support for invoking LLMs.

RAGChecker v0.1.2

What's Changed

• Fix run.sh metrics type typo by @alapha23 in #7
• docs: update CONTRIBUTING.md by @eltociear in #9
• Change the dependency on RefChecker to v0.2.3 to fix joint checking bug for single reference

New Contributors

• @alapha23 made their first contribution in #7
• @eltociear made their first contribution in #9

Full Changelog: v0.1.1...v0.1.2

RAGChecker v0.1.1

New features:

• Add integration with LlamaIndex
• Update dependency of RefChecker to v0.2.2 for joint checking.

RAGChecker v0.1.0

RAGChecker v0.1.0 Release Note

We are excited to announce the initial release of RAGChecker, version 0.1.0. RAGChecker is a comprehensive evaluation framework designed for in-depth analysis and diagnostics of Retrieval-Augmented Generation (RAG) systems.

Key Features

• Fine-grained Evaluation: Utilizes claim-level entailment checking for detailed analysis of RAG system performance.
• Comprehensive Metric Suite: Includes metrics for overall performance, retriever effectiveness, and generator capabilities.
• Flexible Model Integration: Supports various LLMs for claim extraction and checking, including AWS Bedrock models.
• Easy-to-use CLI: Provides a command-line interface for quick evaluation of RAG outputs.
• Python API: Offers a Python API for seamless integration into existing workflows and scripts.

Metrics Included

• Overall: Precision, Recall, F1 Score
• Retriever: Claim Recall, Context Precision
• Generator: Context Utilization, Noise Sensitivity, Hallucination, Self-knowledge, Faithfulness

Getting Started

To start using RAGChecker, install it via pip:

pip install ragchecker
python -m spacy download en_core_web_sm

For detailed usage instructions and examples, please refer to our GitHub repository: https://github.com/amazon-science/RAGChecker

Feedback and Contributions

As an open-source project, we welcome feedback, bug reports, and contributions from the community. Please use the GitHub issues section for reporting bugs or suggesting enhancements.

Thank you for your interest in RAGChecker. We look forward to seeing how it helps improve RAG systems across various applications!