Vibeleaderboard

Search

Searching the tools index
Filters

Type

Ready to use
Build with
Connect / Integrate
Extend
Operate
Reference

Topics

More topics

Source

Ranked under Top · All Time

  1. 626

    OpenJudge

    Evaluation framework that scores LLM quality with reward models and grader skills for RLHF and agent alignment.

    Library / Frameworkhttps://github.com/agentscope-ai/openjudgeOpen Source

    agentscope-ai · AI Tools

  2. 303

    OpenAI Evals

    Framework from OpenAI for evaluating LLM systems, with an open registry of benchmarks the community can extend.

    Library / Frameworkhttps://github.com/openai/evalsOpen Source

    openai · Developer Tools

  3. 1726

    UltraEval

    Open source framework that evaluates foundation models through modular benchmarks for capability assessment.

    Library / Frameworkhttps://github.com/openbmb/ultraevalOpen Source

    openbmb · AI Tools

  4. 751

    DeepEval

    Open-source LLM evaluation framework whose 40+ research-backed metrics for agents, RAG, and safety run as unit tests.

    Library / Frameworkgithub.com/confident-ai/deepevalOpen Source

    confident-ai · Developer Tools

  5. 654

    LLM Council

    Karpathy experiment where an ensemble of LLMs debates a question and arrives at a synthesized answer.

    https://github.com/karpathy/llm-councilOpen Source

    karpathy · AI Agents

  6. 868

    Ragas

    Evaluation framework for RAG pipelines scoring retrieval and generation on faithfulness and context precision.

    Library / Frameworkgithub.com/explodinggradients/ragasOpen Source

    explodinggradients · Developer Tools

  7. 1172

    Self-Adaptive LLMs

    Framework that lets language models adapt to unseen tasks in real time by composing expert modules on the fly.

    Library / Frameworkhttps://github.com/sakanaai/self-adaptive-llmsOpen Source

    sakanaai · AI Tools

  8. 1571

    Hegelion

    Runs thesis, antithesis, synthesis loops with LLMs to improve answer quality on hard questions.

    https://github.com/hmbown/hegelionOpen Source

    Hmbown · AI Agents

  9. 810

    WhichLLM

    Benchmarks local LLMs on your specific hardware and ranks which models run best on it.

    Utilitygithub.com/andyyyy64/whichllm

    andyyyy64 · Developer Tools

  10. 1094

    AgentCPM

    End-to-end infrastructure for training and evaluating LLM agents, covering the lifecycle from data to eval.

    Library / Frameworkhttps://github.com/openbmb/AgentCPMOpen Source

    openbmb · AI Agents

  11. 1442

    Agent Eval

    Playground for evaluating LLM agent runs by scoring tool calls, traces, and outputs side by side.

    Applicationhttps://github.com/vercel-labs/agent-evalOpen Source

    vercel-labs · AI Agents

  12. 2050

    LLM Wiki

    Desktop app that turns your documents into a self-updating, interlinked wiki using an LLM.

    github.com/nashsu/llm_wiki

    Productivity

  13. 510

    Purple Llama

    Meta's trust-and-safety toolkit for LLM security, spanning code scanning, jailbreak benchmarks, and I/O classifiers.

    https://github.com/meta-llama/purplellamaOpen Source

    meta-llama · Cybersecurity

  14. 1821

    Autodialectics

    Agentic harness that keeps research LLM runs on task using contracts, evidence, verification, and anti-slop gates.

    Runtimehttps://github.com/hmbown/autodialecticsOpen Source

    Hmbown · AI Agents

  15. 917

    Harvey LAB (Legal Agent Benchmark)

    Open-source benchmark of 1,671 legal tasks with an execution harness for running and scoring LLM agents.

    Evaluation / Datasetgithub.com/harveyai/harvey-labsOpen Source

    harveyai · AI Agents

  16. 2810

    TrackLLM

    Continuously monitors LLM API endpoints for undisclosed output changes over time, using logprobs.

    www.trackllm.net

    Developer Tools

  17. 1120

    llm-bridge

    Universal adapter that translates LLM input formats between providers, with built-in observability and error handling.

    Library / Frameworkhttps://github.com/supermemoryai/llm-bridgeOpen Source

    supermemoryai · AI Tools

  18. 2240

    Toulmini

    Forces LLMs into structured sequential reasoning based on Toulmin's argumentation model.

    https://github.com/hmbown/toulminiOpen Source

    Hmbown · AI Tools

  19. 316

    Prompt Flow

    Microsoft toolkit for building, testing, and deploying LLM apps with visual orchestration and prompt evaluation.

    Developer Toolchainhttps://github.com/microsoft/promptflowOpen Source

    microsoft · AI Tools

  20. 1378

    Anthropic Evals

    Public suite of reference tasks and frameworks for benchmarking Claude and other language models.

    Evaluation / Datasethttps://github.com/anthropics/evalsOpen Source

    anthropics · AI Tools

  21. 311

    Langfuse

    Open-source LLM engineering platform with tracing, prompt management, evaluations, and cost monitoring for AI apps.

    Applicationgithub.com/langfuse/langfuseFreemium

    langfuse · Developer Tools

  22. 1921

    Midtry

    Reasoning harness that queries several LLM CLIs in parallel, each prompted from a different perspective.

    Utilityhttps://github.com/hmbown/midtryOpen Source

    Hmbown · AI Agents

  23. 1958

    MLX Benchmark

    Scores how well LLMs know Apple's MLX framework across a 441-question coding dataset.

    Evaluation / Datasetgithub.com/goekdeniz-guelmez/mlx-benchmarkOpen Source

    @ActuallyIsaak · Developer Tools

  24. 2677

    VernLLM

    Adds retries, fallback, rate limiting, and caching around LLM API calls without a gateway.

    vernllm.dev

    AI Tools

  25. 1457

    LLM Codes

    Converts developer documentation into clean LLM-ready Markdown, with llms.txt support built in.

    Utilityhttps://github.com/amantus-ai/llm-codesOpen Source

    amantus-ai · Developer Tools

  26. 1052

    ToolBench

    OpenBMB platform for training, serving, and evaluating tool-learning LLMs on a set of over 16K real-world APIs.

    Evaluation / Datasethttps://github.com/openbmb/toolbenchOpen Source

    openbmb · AI Tools

  27. 1333

    LangSmith

    Tracing, evaluation, and monitoring for LLM apps, framework-agnostic and usable without LangChain.

    Infrastructuresmith.langchain.comFreemium

    @LangChain · Developer Tools

  28. 1214

    LLMRec

    Recommendation framework from a WSDM 2024 paper that augments graphs with LLMs and multi-modal side signals.

    https://github.com/hkuds/llmrecOpen Source

    hkuds · AI Tools

  29. 1895

    Prompt Replay

    Regression testing framework for AI prompts that records, replays, and compares LLM responses.

    Library / Frameworkhttps://github.com/yeachan-heo/prompt-replayOpen Source

    Yeachan-Heo · Developer Tools

  30. 660

    Llama Prompt Ops

    Toolkit from Meta for optimizing LLM prompts through evals and structured experiments rather than guesswork.

    Library / Frameworkhttps://github.com/meta-llama/prompt-opsOpen Source

    meta-llama · AI Tools