Tools

LangSmith

LangSmith is a platform for LLM observability, tracing, evaluation, and monitoring from the LangChain team — framework-agnostic, works with or without LangChain.

smith.langchain.comFreemium29d ago

OpenLLMetry

OpenLLMetry by Traceloop is a set of open-source OpenTelemetry extensions that add tracing and observability to LLM apps and agents, exporting to any OTel-compatible backend.

github.com/traceloop/openllmetryOpen Source29d ago

Laminar

Laminar is an open-source observability platform built specifically for AI agents. It lets you trace every agent run, set up plain-English alerts for failure conditions, and run evaluations to catch regressions — all in one place. It integrates with Vercel AI SDK, LangChain, OpenAI, Anthropic, and more with just a line of code.

github.com/lmnr-ai/lmnrOpen Source29d ago

AgentScope Studio

Development-oriented visualization toolkit for AgentScope. Trace, debug, and monitor multi-agent LLM systems with an OpenTelemetry-aware UI.

https://github.com/agentscope-ai/agentscope-studioOpen Source2mo ago

Langfuse

Langfuse is an open-source LLM engineering platform that provides observability, prompt management, evaluation, and experimentation tools for AI applications. It helps teams trace every LLM call, monitor cost and latency, run evaluations, and continuously improve their AI products from prototype to production. It integrates with 100+ frameworks and model providers with no vendor lock-in.

github.com/langfuse/langfuseFreemium1mo ago

PostHog for OpenClaw

PostHog LLM analytics plugin for OpenClaw.

https://github.com/posthog/posthog-openclawOpen Source2mo ago

llm-bridge

Universal LLM input-format adapter with built-in observability and error handling — swap models without rewriting prompts.

https://github.com/supermemoryai/llm-bridgeOpen Source2mo ago

Vercel AI SDK Observability

VoltAgent observability integration for the Vercel AI SDK, traces and inspects LLM calls.

https://github.com/VoltAgent/vercel-ai-sdk-observabilityOpen Source2mo ago

Abacus

Tracks token consumption of coding agents so teams can monitor LLM spend.

https://github.com/getsentry/abacusOpen Source2mo ago

Prompt Flow

Microsoft's toolkit for building, testing, and deploying high-quality LLM applications — visual orchestration, tracing, and prompt evaluation.

https://github.com/microsoft/promptflowOpen Source2mo ago

Nominal Dev

A modern observability platform that addresses the limitations of traditional logging systems. It promotes using wide events instead of fragmented logs to provide better insights into application behavior and system performance.

loggingsucks.comFreemium5mo ago

Autodialectics

Agentic harness for keeping research-oriented LLM runs on-task with contracts, evidence, dialectics, verification, and anti-slop gates.

https://github.com/hmbown/autodialecticsOpen Source2mo ago

LLM Council

Andrej Karpathy's experiment: an ensemble of LLMs debating your hardest questions and arriving at a synthesized answer.

https://github.com/karpathy/llm-councilOpen Source2mo ago

Purple Llama

Meta's set of trust-and-safety tools for assessing and improving LLM security — code-scanning, jailbreak benchmarks, and input/output classifiers.

https://github.com/meta-llama/purplellamaOpen Source2mo ago

Midtry

Multi-perspective reasoning harness that queries multiple LLM CLIs in parallel with different perspectives.

https://github.com/hmbown/midtryOpen Source2mo ago

Agent Eval

Playground for evaluating LLM agent runs — score tool calls, traces, and outputs side-by-side.

https://github.com/vercel-labs/agent-evalOpen Source2mo ago

OpenJudge

Unified framework for holistic LLM evaluation and quality rewards, with reward models and grader skills for RLHF and agent alignment.

https://github.com/agentscope-ai/openjudgeOpen Source2mo ago

WhichLLM

Benchmarks and ranks which local LLM runs best on your specific hardware.

github.com/andyyyy64/whichllm1mo ago

ToolBench

OpenBMB's open platform for training, serving, and evaluating LLMs for tool learning — over 16K real-world APIs in the training set.

https://github.com/openbmb/toolbenchOpen Source2mo ago

Maple

An open-source observability platform built on OpenTelemetry that collects, visualizes, and analyzes distributed traces, logs, and metrics from your services. Features AI-powered diagnostics and can query billions of rows in milliseconds to help monitor service health and performance.

maple.devOpen Source5mo ago

Prompt Replay

AI prompt regression testing framework — record, replay, and compare LLM responses.

https://github.com/yeachan-heo/prompt-replayOpen Source2mo ago

frfr

High-confidence document Q&A system for complex docs, designed to reduce LLM hallucinations on dense technical material.

https://github.com/block/frfrOpen Source2mo ago

AgentCPM

End-to-end infrastructure for training and evaluating LLM agents. Tooling for the full agent lifecycle from data to eval.

https://github.com/openbmb/AgentCPMOpen Source2mo ago

omlx

An LLM inference server for Apple Silicon with continuous batching and SSD caching.

github.com/jundot/omlx1mo ago

Helicone

Helicone is an AI Gateway and LLM observability platform that lets developers route, monitor, and debug AI applications across 100+ models with a single API key. It provides real-time cost and latency tracking, prompt management, agent tracing, and one-line integration with OpenAI, Anthropic, and other major providers. It's backed by Y Combinator and used by fast-growing AI companies to build more reliable AI products.

github.com/helicone/heliconeFreemium1mo ago