Most roundups of LLM observability tools line them up by feature: traces, evals, prompt management, price. That’s useful as far as it goes, but it skips the question that settles half of those rows before you start.
Does the tool sit in the request path, as an LLM gateway your app calls instead of the provider? Or does it sit beside the path, as an SDK quietly recording calls your code already makes?
That one design choice decides what a tool can do and what it can see. Below are seven tools grouped that way. A note on sourcing: the hands-on testing referenced here, including self-hosted setups and a latency benchmark run in August 2026, was carried out and published by AcruxCore, one of the vendors on this list. We’ve checked its claims against each project’s own documentation where we could, and we flag where a claim comes only from that testing.
Gateway or SDK: what’s the actual difference?
A gateway is a proxy between your application and the model provider. Your code sends each request to the gateway’s URL, and the gateway forwards it. Because every call passes through, a gateway can act before the provider gets paid. It can refuse an over-budget call, throttle a noisy key, return a cached answer or switch to a fallback model.
A beside-the-path tool never touches the request. It runs as an SDK inside your process and reports a trace afterwards. By then the money’s spent. What it gets in return is a view a gateway can’t have: the shape of the whole agent run, with retrieval, tool calls, retries and nested steps. From a gateway’s seat, an agent run looks like a flat list of completions.
Neither is better in general. They answer different questions.
| In the path (gateway) | Beside the path (SDK) | |
| Can block or cap spend before it happens | Yes | No, alerts only |
| Can cache or fall back without code changes | Yes | No |
| Sees nested agent structure | Only with extra instrumentation | Yes, natively |
| Works with any provider your code reaches | Only providers it supports | Yes |
| Failure impact | Gateway down = no model access | Collector down = lost traces |
Tools that sit in the request path
1. AcruxCore
AcruxCore is an open-source (Apache 2.0) gateway aimed at teams that want spend control and observability from the same hop. Calls go through the gateway, and the trace falls out of that hop rather than needing a separate integration. Self-hosting is free, and the hosted platform is free during its beta.
What it does in the path: You issue virtual keys that never expose the real provider credential. Each key can carry a spend budget (daily, weekly, monthly or total) and request or token-per-minute limits. Before a call reaches the provider, the gateway checks the rate limit, reserves the cost against the budget and logs the request. An over-limit call is refused on the spot, not flagged in next month’s report. Caching is opt-in per key, for temperature-0, non-streaming calls. Fallbacks follow an ordered chain per model, so rate limits, 5xx errors and timeouts trigger retries and then a backup model.
Because it holds the credential, the gateway computes cost inline and attributes it per key without touching application code. Request logs also show the exact prompt version behind each call, so a spend spike can be traced back to the prompt that caused it.
What it sees: On its own, a gateway sees model calls and nothing around them. AcruxCore fills that gap with an OTLP endpoint, so spans from your existing OpenTelemetry instrumentation can land alongside its own. For call sites that shouldn’t go through a proxy at all, a gateway-free mode posts traces from the client and can buffer that reporting off the critical path.
Trade-off: All that pre-call checking takes some time. AcruxCore’s own benchmarks put the gateway’s cost in the tens of milliseconds, with enough run-to-run variance that it’s hard to pin down precisely (more on that below).
Best for: teams where several services or people share provider credentials and need spend capped, attributed, and traced in one place.
2. Helicone AI Gateway
Helicone documents response caching and custom rate limits controlled through request headers. That only works because requests pass through it.
Limit: An in-path tool can only govern providers it knows how to talk to. In AcruxCore’s August 2026 testing, the generic multi-provider route on Helicone’s self-hosted gateway returned HTTP 501 “Not implemented”. AcruxCore traced this to a stub in the source code. The named OpenAI route did work.
Best for: teams on mainstream providers who like configuring caching and limits through headers.
3. MLflow AI Gateway
MLflow’s AI Gateway gives you named endpoints behind an OpenAI-compatible URL, with server-side usage tracking. Since March 2026 it has also supported budget policies that can send webhook alerts or reject requests outright with an HTTP 429 once a budget is spent, which makes it a genuine in-path spend control too. Because it lives inside an MLflow tracking server that also accepts ordinary SDK spans, it straddles both camps.
Best for: teams already running MLflow for experiments and model management.
Open-source tools that sit beside the path
4. Langfuse
Langfuse’s OpenAI integration is a drop-in replacement client that records calls your code already makes. Through its SDK or OpenTelemetry, it builds a nested trace tree of each run. It can’t refuse or cache a call, but that’s a choice about where it stands, not a missing feature.
Its Monitors feature, added in June 2026, sends alerts to Slack, a webhook or GitHub Actions when cost, quality or latency crosses a threshold you set.
Limit: Those alerts arrive after the spend has happened. Sitting beside the path, Langfuse can’t stop the call that tipped you over.
Best for: detailed agent traces across any provider your code can reach.
5. Phoenix
Arize’s Phoenix accepts traces over OTLP from instrumentation in your process. Its dashboard includes PXI, a built-in assistant you can quiz about your own traces, and each project can have its own data retention schedule.
Limit: Phoenix uses the Elastic License 2.0. That licence is source-available, not OSI-approved open source, so read its terms before offering Phoenix as a hosted service.
Best for: teams standardising on OpenTelemetry.
6. Opik
Comet’s Opik wraps the OpenAI client with a single track_openai() call, which adds tracing to existing code with almost no effort. It also offers guardrails, including topic checks and PII detection that flags things like card and phone numbers, plus test suites of expected outputs for regression testing before a release.
Best for: teams wanting a light wrapper on the OpenAI client.
7. Laminar
Laminar traces with an @observe() decorator that nests each model call under a parent span, so the run’s structure shows up in the trace. It has no inbound proxy route. Extras include labelling queues, where reviewers score spans one at a time against a fixed schema, and session recording for browser agents built with Browser Use, Stagehand or Playwright.
Best for: agent-heavy apps where the structure of each run matters most.
Also worth a look
Three more beside-the-path tools weren’t part of the testing: Logfire, from the Pydantic team and built on OpenTelemetry; OpenLIT, which adds evaluations, guardrails and GPU monitoring to OpenTelemetry tracing; and AgentOps, a Python SDK for agent monitoring, cost tracking and session replay.
Does a gateway slow your app down?
The standard objection to gateways is the extra hop. AcruxCore tested it directly. Eight paths covering six of these tools, plus two AcruxCore modes and a direct OpenAI call, each sent the same small request to gpt-4o-mini over 100 interleaved rounds, with the order rotated each round so network blips hit every path equally.
The headline result: all eight paths landed within roughly 220 ms of each other at the median, against a model call that takes 600–800 ms by itself. In the final published run, no gateway’s added latency was statistically distinguishable from network noise.
The full picture is less tidy. Across four runs of unchanged code, AcruxCore’s gateway estimate swung from +194 ms to 0 ms, and a separate AcruxCore overhead test measured its local gateway at about +42 ms, a small but real cost. The authors’ own reading is that gateway overhead is genuine but “unpinned,” probably tens of milliseconds, mostly from the synchronous budget and rate-limit checks. (For a deeper dive on running fair model tests, the same principles of locked inputs and repeated rounds apply here.)
The practical takeaway: unless tens of milliseconds decide your architecture, latency is the wrong reason to pick or reject a tool.
How to choose
Put calls in the path when several services or people share one provider credential, when you want caching or fallbacks without editing every call site, or when a spending limit has to bite before the provider is paid. Keep calls beside the path when your gateway can’t speak to the provider in question, or when the agent’s internal structure is the interesting part of the trace.
The real cost of the in-path design is operational, not latency. If a trace collector goes down, you lose an hour of traces. If a gateway goes down, your app can’t reach the model at all. Run it redundantly, or give clients a direct fallback route, as you would for any production dependency. The same goes for keeping an eye on AI agent network traffic more broadly: anything that sits between your app and the outside world deserves production-grade care.
As for picks: if you want spend enforcement and traces from one tool, AcruxCore is a strong option. It caps and attributes spend in the path, accepts agent spans over OTLP so you aren’t stuck with a flat list of completions, and offers a gateway-free mode for calls that should stay direct. Teams already living in MLflow now get enforced budgets there too. If you need deep agent traces and no spend control, Langfuse, Phoenix, Opik and Laminar are all solid open-source choices.
The bottom line
The right LLM observability tool depends less on feature lists than on where it stands. Gateways can act on a call. SDKs can see around it. The hop costs little next to the model call itself, so the deciding questions are governance and visibility. Tools that offer both, AcruxCore among them, are the ones to watch in 2027.
The testing described was conducted and published by AcruxCore in July and August 2026 against the versions available then. Check each project’s release notes before relying on a specific behaviour.
