Vibeleaderboard

Search

Searching the tools index
Filters

Type

Ready to use
Build with
Connect / Integrate
Extend
Operate
Reference

Topics

More topics

Source

Ranked under Top · All Time

  1. 622

    OpenJudge

    Evaluation framework that scores LLM quality with reward models and grader skills for RLHF and agent alignment.

    Library / Frameworkhttps://github.com/agentscope-ai/openjudgeOpen Source

    agentscope-ai · AI Tools

  2. 314

    OpenAI Evals

    Framework from OpenAI for evaluating LLM systems, with an open registry of benchmarks the community can extend.

    Library / Frameworkhttps://github.com/openai/evalsOpen Source

    openai · Developer Tools

  3. 3185

    UltraEval

    Open source framework that evaluates foundation models through modular benchmarks for capability assessment.

    Library / Frameworkhttps://github.com/openbmb/ultraevalOpen Source

    openbmb · AI Tools

  4. 768

    DeepEval

    Open-source LLM evaluation framework whose 40+ research-backed metrics for agents, RAG, and safety run as unit tests.

    Library / Frameworkgithub.com/confident-ai/deepevalOpen Source

    confident-ai · Developer Tools

  5. 654

    LLM Council

    Karpathy experiment where an ensemble of LLMs debates a question and arrives at a synthesized answer.

    https://github.com/karpathy/llm-councilOpen Source

    karpathy · AI Agents

  6. 961

    Ragas

    Evaluation framework for RAG pipelines scoring retrieval and generation on faithfulness and context precision.

    Library / Frameworkgithub.com/explodinggradients/ragasOpen Source

    explodinggradients · Developer Tools

  7. 3103

    Self-Adaptive LLMs

    Framework that lets language models adapt to unseen tasks in real time by composing expert modules on the fly.

    Library / Frameworkhttps://github.com/sakanaai/self-adaptive-llmsOpen Source

    sakanaai · AI Tools

  8. 3018

    Hegelion

    Runs thesis, antithesis, synthesis loops with LLMs to improve answer quality on hard questions.

    https://github.com/hmbown/hegelionOpen Source

    Hmbown · AI Agents

  9. 2798

    WhichLLM

    Benchmarks local LLMs on your specific hardware and ranks which models run best on it.

    Utilitygithub.com/andyyyy64/whichllm

    andyyyy64 · Developer Tools

  10. 1194

    AgentCPM

    End-to-end infrastructure for training and evaluating LLM agents, covering the lifecycle from data to eval.

    Library / Frameworkhttps://github.com/openbmb/AgentCPMOpen Source

    openbmb · AI Agents

  11. 770

    LLM Wiki

    Desktop app that turns your documents into a self-updating, interlinked wiki using an LLM.

    Applicationgithub.com/nashsu/llm_wiki

    Productivity

  12. 501

    Purple Llama

    Meta's trust-and-safety toolkit for LLM security, spanning code scanning, jailbreak benchmarks, and I/O classifiers.

    https://github.com/meta-llama/purplellamaOpen Source

    meta-llama · Cybersecurity

  13. 3371

    Autodialectics

    Agentic harness that keeps research LLM runs on task using contracts, evidence, verification, and anti-slop gates.

    Runtimehttps://github.com/hmbown/autodialecticsOpen Source

    Hmbown · AI Agents

  14. 1000

    Harvey LAB (Legal Agent Benchmark)

    Open-source benchmark of 1,671 legal tasks with an execution harness for running and scoring LLM agents.

    Evaluation / Datasetgithub.com/harveyai/harvey-labsOpen Source

    harveyai · AI Agents

  15. 2652

    TrackLLM

    Continuously monitors LLM API endpoints for undisclosed output changes over time, using logprobs.

    Servicewww.trackllm.net

    Developer Tools

  16. 1601

    llm-bridge

    Universal adapter that translates LLM input formats between providers, with built-in observability and error handling.

    Library / Frameworkhttps://github.com/supermemoryai/llm-bridgeOpen Source

    supermemoryai · AI Tools

  17. 4282

    Toulmini

    Forces LLMs into structured sequential reasoning based on Toulmin's argumentation model.

    https://github.com/hmbown/toulminiOpen Source

    Hmbown · AI Tools

  18. 262

    Prompt Flow

    Microsoft toolkit for building, testing, and deploying LLM apps with visual orchestration and prompt evaluation.

    Developer Toolchainhttps://github.com/microsoft/promptflowOpen Source

    microsoft · AI Tools

  19. 2770

    Anthropic Evals

    Public suite of reference tasks and frameworks for benchmarking Claude and other language models.

    Evaluation / Datasethttps://github.com/anthropics/evalsOpen Source

    anthropics · AI Tools

  20. 308

    Langfuse

    Open-source LLM engineering platform with tracing, prompt management, evaluations, and cost monitoring for AI apps.

    Applicationgithub.com/langfuse/langfuseFreemium

    langfuse · Developer Tools

  21. 3421

    Midtry

    Reasoning harness that queries several LLM CLIs in parallel, each prompted from a different perspective.

    Utilityhttps://github.com/hmbown/midtryOpen Source

    Hmbown · AI Agents

  22. 3494

    LLMxRay

    Local observability interface for Ollama covering chat, streaming, RAG, introspection and metrics

    Applicationgithub.com/lognebudo/llmxrayOpen Source

    Developer Tools

  23. 3575

    MLX Benchmark

    Scores how well LLMs know Apple's MLX framework across a 441-question coding dataset.

    Evaluation / Datasetgithub.com/goekdeniz-guelmez/mlx-benchmarkOpen Source

    @ActuallyIsaak · Developer Tools

  24. 1808

    VernLLM

    Adds retries, fallback, rate limiting, and caching around LLM API calls without a gateway.

    Library / Frameworkvernllm.dev

    AI Tools

  25. 2907

    LLM Codes

    Converts developer documentation into clean LLM-ready Markdown, with llms.txt support built in.

    Utilityhttps://github.com/amantus-ai/llm-codesOpen Source

    amantus-ai · Developer Tools

  26. 1377

    ToolBench

    OpenBMB platform for training, serving, and evaluating tool-learning LLMs on a set of over 16K real-world APIs.

    Evaluation / Datasethttps://github.com/openbmb/toolbenchOpen Source

    openbmb · AI Tools

  27. 2585

    LangSmith

    Tracing, evaluation, and monitoring for LLM apps, framework-agnostic and usable without LangChain.

    Infrastructuresmith.langchain.comFreemium

    @LangChain · Developer Tools

  28. 2170

    LLMRec

    Recommendation framework from a WSDM 2024 paper that augments graphs with LLMs and multi-modal side signals.

    https://github.com/hkuds/llmrecOpen Source

    hkuds · AI Tools

  29. 3383

    Prompt Replay

    Regression testing framework for AI prompts that records, replays, and compares LLM responses.

    Library / Frameworkhttps://github.com/yeachan-heo/prompt-replayOpen Source

    Yeachan-Heo · Developer Tools

  30. 668

    Llama Prompt Ops

    Toolkit from Meta for optimizing LLM prompts through evals and structured experiments rather than guesswork.

    Library / Frameworkhttps://github.com/meta-llama/prompt-opsOpen Source

    meta-llama · AI Tools