Vibeleaderboard

Search

Searching the tools index
Filters

Type

Ready to use
Build with
Connect / Integrate
Extend
Operate
Reference

Topics

More topics

Source

Ranked under Top · All Time

  1. 2751

    telemetry.dev

    OpenTelemetry-native tracing for LLM apps, capturing tokens, cost, latency, and errors per call.

    Servicetelemetry.devFreemium

    @telemetrydev · Developer Tools

  2. 1333

    LangSmith

    Tracing, evaluation, and monitoring for LLM apps, framework-agnostic and usable without LangChain.

    Infrastructuresmith.langchain.comFreemium

    @LangChain · Developer Tools

  3. 799

    OpenLLMetry

    OpenTelemetry extensions that add tracing to LLM apps and agents, exporting to any OTel-compatible backend.

    Library / Frameworkgithub.com/traceloop/openllmetryOpen Source

    traceloop · Developer Tools

  4. 366

    Laminar

    Open-source observability platform for AI agents with run tracing, plain-English failure alerts, and evaluations.

    Infrastructuregithub.com/lmnr-ai/lmnrOpen Source

    lmnr-ai · AI Agents

  5. 2810

    TrackLLM

    Continuously monitors LLM API endpoints for undisclosed output changes over time, using logprobs.

    www.trackllm.net

    Developer Tools

  6. 1011

    AgentScope Studio

    Traces, debugs, and monitors multi-agent LLM systems through an OpenTelemetry-aware visual interface.

    Applicationhttps://github.com/agentscope-ai/agentscope-studioOpen Source

    agentscope-ai · AI Agents

  7. 2482

    AgentTrace

    SDK and dashboard that auto-repairs malformed LLM tool-call arguments before they crash agents.

    github.com/mohitkumar188/agenttrace

    Developer Tools

  8. 311

    Langfuse

    Open-source LLM engineering platform with tracing, prompt management, evaluations, and cost monitoring for AI apps.

    Applicationgithub.com/langfuse/langfuseFreemium

    langfuse · Developer Tools

  9. 1681

    PostHog for OpenClaw

    Plugin that wires PostHog's LLM analytics into OpenClaw.

    Extensionhttps://github.com/posthog/posthog-openclawOpen Source

    PostHog · Developer Tools

  10. 1120

    llm-bridge

    Universal adapter that translates LLM input formats between providers, with built-in observability and error handling.

    Library / Frameworkhttps://github.com/supermemoryai/llm-bridgeOpen Source

    supermemoryai · AI Tools

  11. 1928

    Vercel AI SDK Observability

    VoltAgent integration that traces and inspects LLM calls made through the Vercel AI SDK.

    Library / Frameworkhttps://github.com/VoltAgent/vercel-ai-sdk-observabilityOpen Source

    voltagent · Developer Tools

  12. 1515

    Abacus

    Monitors the token consumption of coding agents, giving teams visibility into their LLM spend.

    Applicationhttps://github.com/getsentry/abacusOpen Source

    getsentry · Developer Tools

  13. 316

    Prompt Flow

    Microsoft toolkit for building, testing, and deploying LLM apps with visual orchestration and prompt evaluation.

    Developer Toolchainhttps://github.com/microsoft/promptflowOpen Source

    microsoft · AI Tools

  14. 1887

    Nominal Dev

    Observability platform built around wide events rather than fragmented logs for insight into application behavior.

    Infrastructureloggingsucks.comFreemium

    Developer Tools

  15. 1821

    Autodialectics

    Agentic harness that keeps research LLM runs on task using contracts, evidence, verification, and anti-slop gates.

    Runtimehttps://github.com/hmbown/autodialecticsOpen Source

    Hmbown · AI Agents

  16. 654

    LLM Council

    Karpathy experiment where an ensemble of LLMs debates a question and arrives at a synthesized answer.

    https://github.com/karpathy/llm-councilOpen Source

    karpathy · AI Agents

  17. 510

    Purple Llama

    Meta's trust-and-safety toolkit for LLM security, spanning code scanning, jailbreak benchmarks, and I/O classifiers.

    https://github.com/meta-llama/purplellamaOpen Source

    meta-llama · Cybersecurity

  18. 1921

    Midtry

    Reasoning harness that queries several LLM CLIs in parallel, each prompted from a different perspective.

    Utilityhttps://github.com/hmbown/midtryOpen Source

    Hmbown · AI Agents

  19. 1442

    Agent Eval

    Playground for evaluating LLM agent runs by scoring tool calls, traces, and outputs side by side.

    Applicationhttps://github.com/vercel-labs/agent-evalOpen Source

    vercel-labs · AI Agents

  20. 626

    OpenJudge

    Evaluation framework that scores LLM quality with reward models and grader skills for RLHF and agent alignment.

    Library / Frameworkhttps://github.com/agentscope-ai/openjudgeOpen Source

    agentscope-ai · AI Tools

  21. 2738

    Owl-24

    Groups production errors via OpenTelemetry and routes them to your coding agent for a fix PR.

    owl24.devFreemium

    Developer Tools

  22. 810

    WhichLLM

    Benchmarks local LLMs on your specific hardware and ranks which models run best on it.

    Utilitygithub.com/andyyyy64/whichllm

    andyyyy64 · Developer Tools

  23. 542

    Honeycomb

    Observability platform built on high-cardinality event data and distributed tracing for debugging microservices.

    Servicehoneycomb.ioPaid

    Developer Tools

  24. 1052

    ToolBench

    OpenBMB platform for training, serving, and evaluating tool-learning LLMs on a set of over 16K real-world APIs.

    Evaluation / Datasethttps://github.com/openbmb/toolbenchOpen Source

    openbmb · AI Tools

  25. 609

    Maple

    OpenTelemetry-based observability platform that analyzes traces, logs, and metrics with AI diagnostics.

    Infrastructuremaple.devOpen Source

    @Makisuo · Developer Tools

  26. 976

    ctrlb-decompose

    A Rust log analysis tool clustering millions of noisy log lines into typed, statistically-annotated patterns.

    github.com/ctrlb-hq/ctrlb-decomposeOpen Source

    Developer Tools

  27. 2552

    OmnisBench

    Benchmarks LLM routing policies against an ideal cost-quality frontier using fresh, re-gradable tasks.

    github.com/fortitude-group/omnisbenchOpen Source

    Developer Tools

  28. 2677

    VernLLM

    Adds retries, fallback, rate limiting, and caching around LLM API calls without a gateway.

    vernllm.dev

    AI Tools

  29. 2642

    conduct-litellm-guard

    Runs every LiteLLM request through a policy check that can block or require approval.

    pypi.org

    Developer Tools

  30. 1895

    Prompt Replay

    Regression testing framework for AI prompts that records, replays, and compares LLM responses.

    Library / Frameworkhttps://github.com/yeachan-heo/prompt-replayOpen Source

    Yeachan-Heo · Developer Tools