Vibeleaderboard

Search

Searching the tools index
Filters

Type

Ready to use
Build with
Connect / Integrate
Extend
Operate
Reference

Topics

More topics

Source

Ranked under Top · All Time

  1. 2586

    LangSmith

    Tracing, evaluation, and monitoring for LLM apps, framework-agnostic and usable without LangChain.

    Infrastructuresmith.langchain.comFreemium

    @LangChain · Developer Tools

  2. 836

    OpenLLMetry

    OpenTelemetry extensions that add tracing to LLM apps and agents, exporting to any OTel-compatible backend.

    Library / Frameworkgithub.com/traceloop/openllmetryOpen Source

    traceloop · Developer Tools

  3. 4055

    telemetry.dev

    Trace AI workflows with a clear account of ingestion costs, capture defaults, redaction and retention.

    Servicetelemetry.devFreemium

    @telemetrydev · Developer Tools

  4. 362

    Laminar

    Open-source observability platform for AI agents with run tracing, plain-English failure alerts, and evaluations.

    Infrastructuregithub.com/lmnr-ai/lmnrOpen Source

    lmnr-ai · AI Agents

  5. 2653

    TrackLLM

    Continuously monitors LLM API endpoints for undisclosed output changes over time, using logprobs.

    Servicewww.trackllm.net

    Developer Tools

  6. 3495

    LLMxRay

    Local observability interface for Ollama covering chat, streaming, RAG, introspection and metrics

    Applicationgithub.com/lognebudo/llmxrayOpen Source

    Developer Tools

  7. 1280

    AgentScope Studio

    Traces, debugs, and monitors multi-agent LLM systems through an OpenTelemetry-aware visual interface.

    Applicationhttps://github.com/agentscope-ai/agentscope-studioOpen Source

    agentscope-ai · AI Agents

  8. 1232

    Traccia

    Python SDK for tracing AI agents with OpenTelemetry and enforcing runtime policies like spend and loop caps.

    github.com/traccia-ai/traccia-pyOpen Source

    Developer Tools

  9. 4369

    AgentTrace

    SDK and dashboard that auto-repairs malformed LLM tool-call arguments before they crash agents.

    Library / Frameworkgithub.com/mohitkumar188/agenttrace

    Developer Tools

  10. 308

    Langfuse

    Open-source LLM engineering platform with tracing, prompt management, evaluations, and cost monitoring for AI apps.

    Applicationgithub.com/langfuse/langfuseFreemium

    langfuse · Developer Tools

  11. 2768

    Tracelane

    Self-hosted Rust LLM gateway with guardrails, observability and a tamper-evident audit ledger of agent actions.

    Infrastructuregithub.com/tracelane/tracelaneOpen Source

    Developer Tools

  12. 3129

    PostHog for OpenClaw

    Plugin that wires PostHog's LLM analytics into OpenClaw.

    Extensionhttps://github.com/posthog/posthog-openclawOpen Source

    PostHog · Developer Tools

  13. 1602

    llm-bridge

    Universal adapter that translates LLM input formats between providers, with built-in observability and error handling.

    Library / Frameworkhttps://github.com/supermemoryai/llm-bridgeOpen Source

    supermemoryai · AI Tools

  14. 3433

    Vercel AI SDK Observability

    VoltAgent integration that traces and inspects LLM calls made through the Vercel AI SDK.

    Library / Frameworkhttps://github.com/VoltAgent/vercel-ai-sdk-observabilityOpen Source

    voltagent · Developer Tools

  15. 2943

    Abacus

    Monitors the token consumption of coding agents, giving teams visibility into their LLM spend.

    Applicationhttps://github.com/getsentry/abacusOpen Source

    getsentry · Developer Tools

  16. 3787

    llm-cost-watch

    MCP server that analyzes logged LLM calls to explain cost spikes, slow calls and cheaper model routing.

    github.com/rohith0707/llm-cost-watchOpen Source

    Developer Tools

  17. 262

    Prompt Flow

    Microsoft toolkit for building, testing, and deploying LLM apps with visual orchestration and prompt evaluation.

    Developer Toolchainhttps://github.com/microsoft/promptflowOpen Source

    microsoft · AI Tools

  18. 3346

    Nominal Dev

    Observability platform built around wide events rather than fragmented logs for insight into application behavior.

    Infrastructureloggingsucks.comFreemium

    Developer Tools

  19. 3372

    Autodialectics

    Agentic harness that keeps research LLM runs on task using contracts, evidence, verification, and anti-slop gates.

    Runtimehttps://github.com/hmbown/autodialecticsOpen Source

    Hmbown · AI Agents

  20. 654

    LLM Council

    Karpathy experiment where an ensemble of LLMs debates a question and arrives at a synthesized answer.

    https://github.com/karpathy/llm-councilOpen Source

    karpathy · AI Agents

  21. 501

    Purple Llama

    Meta's trust-and-safety toolkit for LLM security, spanning code scanning, jailbreak benchmarks, and I/O classifiers.

    https://github.com/meta-llama/purplellamaOpen Source

    meta-llama · Cybersecurity

  22. 3421

    Midtry

    Reasoning harness that queries several LLM CLIs in parallel, each prompted from a different perspective.

    Utilityhttps://github.com/hmbown/midtryOpen Source

    Hmbown · AI Agents

  23. 622

    OpenJudge

    Evaluation framework that scores LLM quality with reward models and grader skills for RLHF and agent alignment.

    Library / Frameworkhttps://github.com/agentscope-ai/openjudgeOpen Source

    agentscope-ai · AI Tools

  24. 4045

    Owl-24

    Groups production errors via OpenTelemetry and routes them to your coding agent for a fix PR.

    Infrastructureowl24.devFreemium

    Developer Tools

  25. 2799

    WhichLLM

    Benchmarks local LLMs on your specific hardware and ranks which models run best on it.

    Utilitygithub.com/andyyyy64/whichllm

    andyyyy64 · Developer Tools

  26. 542

    Honeycomb

    Observability platform built on high-cardinality event data and distributed tracing for debugging microservices.

    Servicehoneycomb.ioPaid

    Developer Tools

  27. 1377

    ToolBench

    OpenBMB platform for training, serving, and evaluating tool-learning LLMs on a set of over 16K real-world APIs.

    Evaluation / Datasethttps://github.com/openbmb/toolbenchOpen Source

    openbmb · AI Tools

  28. 557

    Maple

    OpenTelemetry-based observability platform that analyzes traces, logs, and metrics with AI diagnostics.

    Infrastructuremaple.devOpen Source

    @Makisuo · Developer Tools

  29. 1170

    ctrlb-decompose

    A Rust log analysis tool clustering millions of noisy log lines into typed, statistically-annotated patterns.

    Utilitygithub.com/ctrlb-hq/ctrlb-decomposeOpen Source

    Developer Tools

  30. 3766

    OmnisBench

    Benchmarks LLM routing policies against an ideal cost-quality frontier using fresh, re-gradable tasks.

    Evaluation / Datasetgithub.com/fortitude-group/omnisbenchOpen Source

    Developer Tools