evaluation-framework

The LLM Evaluation Framework

A framework for few-shot evaluation of language models.

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

☁️ 🚀 📊 📈 Evaluating state of the art in AI

This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

Data-Driven Evaluation for LLM-Powered Applications

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection

Code and Resources for the paper, "Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries"

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

Track 33,707+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.

Python SDK for running evaluations on LLM generated responses

MedEvalKit: A Unified Medical Evaluation Framework

The official evaluation suite and dynamic data release for MixEval.

build and benchmark deep research

A New End-to-end Framework for Evaluating Voice Agents

A research library for automating experiments on Deep Graph Networks