AgentX Trace & Eval

A portable, self-hostable build of AgentX's Governance layer - Trace, Evaluate, and
Monitor for AI agents - as a single local install. No account, no multi-tenant billing, bring
your own LLM API keys.

curl -sSL https://raw.githubusercontent.com/AgentX-ai/AgentX-trace-eval/main/install.sh | bash
agentx-server --dev

--dev starts the engine (API server) on a local SQLite database, opens the dashboard in your
browser, and prints a local API key for the SDK. Prefer a container? See Docker.

Contents

• Features
• Docker
• Configuration
• SDK & OpenTelemetry
• What's in this repo
• Building from source
• Dashboard release process
• Project status
• Contributing
• License

Features

• Trace - ingest agent runs from the AgentX Python SDK
  or any OpenTelemetry-compatible exporter (see SDK & OpenTelemetry).
  Traces sharing a session_id are also browsable as Sessions: one row per conversation, with
  turn counts and a conversation-level coherence score.
• Evaluate - datasets and eval runs with judge scoring, 4 built-in similarity metrics
  (vector/Jaccard/BLEU/ROUGE), and sandboxed custom code scorers; version-scoped Prompt and
  Tool Schema registries for propose → human-approve → publish iteration, both fed by real
  evidence (worst-rated eval results and production failures); Model Portability to replay a
  captured trace's input against alternative models for a quick cost/latency/quality comparison;
  and a Playground grid for testing a prompt/model against dataset cases without a full run -
  tools included (from the Tool Schema registry or ad hoc; with no endpoint configured, calls are
  simulated so tool choice and argument formation are still testable). Tools failing in traffic
  that nobody registered yet are surfaced for one-click registration, drafted from the trace's
  own metadata or the observed calls.
• Monitor - pattern-based detection (phrase/regex/semantic) with per-agent scope and sampling,
  continuous Online Evaluators scoring live traffic against judge criteria (per trace, or per
  session: whole conversations judged automatically once they go idle), Custom Evaluators
  that delegate the verdict to a webhook you control, and Topics clustering of what your
  agents are actually being asked. Signals triage, KPI/trend dashboards, and outbound webhook
  notifications on failures.
• Close the loop - turn production into tests and fixes into proofs. Any trace or session
  becomes a golden dataset case in two clicks (multi-turn conversations included, with
  deduplication and provenance); prompt/tool-schema proposals are validated against those
  cases before a human publishes (candidate vs current, measured); real-world outcomes
  (client.outcomes.report(...)) and end-user feedback (client.feedback.report(...), a
  downvote raises a signal directly) feed Overview's Judge Calibration card, which measures
  how often the automated verdicts agreed with reality; and a Model Comparison card
  aggregates quality/cost/latency per model from real traffic.
• Judge tuning - the judges get judged: each online evaluator's verdicts are measured
  against recorded reality (human re-scores, outcomes, user votes), and its own grading criteria
  can be rewritten from the disagreements - validated by exact re-judging (fixes the cases it got
  wrong, preserves a control set it got right) before a human publishes.
• Improvement Inbox - the loop runs itself: a background sweep notices when a prompt or
  tool schema accumulates fresh failure evidence, generates the improvement proposal AND runs its
  baseline-vs-candidate validation automatically, then queues it under Improve -> Suggestions
  with the measured verdict attached. Humans keep the only pen: review, then publish or dismiss.
• CI gate - fail a build on eval regression: report.gate(fail_under=7, no_regression=True)
  after any SDK eval run returns an exit code, and every recorded gate lands in the dashboard's
  CI Gates tab (history plus a "would the latest run pass?" preview). See the docs for the
  copy-paste GitHub Actions workflow.
• Bring your own keys - OpenAI, Anthropic, and Gemini are all supported for judge scoring and
  model calls; nothing works without your own key, and nothing is billed through AgentX.
• Single binary - the engine compiles to a native executable (via Bun) and the CLI to a native
  Go binary; end users never need Node, Bun, or Go installed.

Docker

docker build -t agentx-selfhost .
docker run -d -p 4700:4700 -v agentx-data:/data agentx-selfhost
docker logs $(docker ps -lq) 2>&1 | grep "API key"   # the dashboard asks for this key

The dashboard (http://localhost:4700) asks for the Default project API key on first visit -
the engine prints it on every start, so with a detached (-d) container it's in docker logs.

/data is where the default SQLite database and config live (AGENTX_HOME, see
Configuration) - mount a named volume so state survives a container recreate, or
point AGENTX_DB_URL at your own Postgres instead and skip the volume entirely. Pass provider
keys with -e OPENAI_API_KEY=... -e ANTHROPIC_API_KEY=... -e GEMINI_API_KEY=..., or set them
later from the dashboard's Platform Settings. The image includes a /health HEALTHCHECK.

The build downloads the latest dashboard release (see
Dashboard release process) and re-checks it on every rebuild - the
download layer is an ADD from the release URL, which the builder revalidates against the remote
file, so a newly published dashboard bundle is picked up automatically with no --no-cache
needed. Pin a specific dashboard build instead with
--build-arg AGENTX_WEB_URL=https://github.com/AgentX-ai/AgentX-trace-eval/releases/download/v0.2.0/agentx-web.tar.gz.

Configuration

Set these in the environment before starting agentx-server:

Trace ingest and pattern matching on phrase/regex both work with no keys configured at all - only
judge scoring and semantic pattern detection need a provider key.

Dashboard authentication

By default there is no login: the engine trusts anything that can reach its port (one machine,
one operator), and the dashboard bootstraps its API key automatically. That stays the default.

AGENTX_AUTH=enabled turns on dashboard sign-in for shared deployments (a team server, or
hosting the dashboard on the internet):

• The first visit shows an owner-setup screen. The first account created becomes the owner of a
  default organization and takes over every existing project on the instance. Later signups join
  that organization as members.
• Signing in is what grants the dashboard its project list (and each project's API key); the
  unauthenticated bootstrap endpoint is disabled in this mode.
• SDK ingest is unchanged in both modes: it authenticates with project API keys, never sessions.
  Enabling or disabling auth never breaks a deployed integration.
• Identity is standard better-auth (email/password out of the box),
  stored in the same database as everything else - works on both SQLite and Postgres.

SDK & OpenTelemetry

AgentX Python SDK - already supports pointing at any base URL, so it works unmodified against
a self-host instance:

export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
export AGENTX_API_KEY=<printed by agentx-server on first run>

OpenTelemetry - Trace also accepts real OTLP/HTTP traces directly, no AgentX SDK required:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4700/api/v1/otel
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=<printed by agentx-server on first run>"

Both OTLP/HTTP wire formats are supported (protobuf and application/json, via
OTEL_EXPORTER_OTLP_PROTOCOL=http/json). One incoming span becomes one AgentX trace row.
Attributes are mapped using the GenAI semantic conventions (gen_ai.*, both the older and newer
field names), OpenLLMetry's legacy indexed attributes, and OpenInference's input.value/
output.value/llm.model_name - see engine/src/otel/mapping.ts for the exact priority order.
Monitor and Online Evaluators run against every OTel-ingested span by default; set
AGENTX_OTEL_MONITOR=false to disable that.

OTel traffic is a first-class citizen of the full loop, via three attribute conventions:

Known gap: gRPC transport isn't supported (HTTP only).

What's in this repo

Building from source

Prerequisites: Go, Node.js + Yarn, and
Bun (only needed for the compiled single-binary path, not day-to-day dev).

git clone [email protected]:AgentX-ai/AgentX-trace-eval.git && cd AgentX-trace-eval
yarn install   # workspace install: engine/ + packages/judge-core together

Fastest dev loop

Runs the engine directly via tsx, no compile step, restarts on file changes:

cd engine && yarn dev --dev

That's the whole loop on a fresh clone: yarn dev first builds the @agentx/judge-core
workspace package automatically (a ~1s tsup step - the engine imports its dist/, which a
fresh clone doesn't have yet), .env is optional (judge features need provider keys,
tracing/ingest work without any), and if web/ is missing (it isn't committed, see
What's in this repo) dev mode downloads the prebuilt dashboard bundle
from this repo's releases on first boot. Offline, or to refresh it manually:

mkdir -p web && curl -fsSL https://github.com/AgentX-ai/AgentX-Trace-Eval/releases/latest/download/agentx-web.tar.gz | tar -xz -C web

The dashboard's source (AgentX-eval-front) is a private repo; if you're on the AgentX team and
have it checked out as a sibling directory, build it from source instead:

(cd ../AgentX-eval-front && yarn install && yarn build:selfhost) && rm -rf web && cp -r ../AgentX-eval-front/dist web

Iterating on the engine's API doesn't need a web/ rebuild; iterating on the dashboard itself
means re-running whichever step above got you web/ in the first place - there's no hot-reload
loop wired up between the two repos yet.

Tests

Unit and integration tests (vitest for the engine and judge-core, go test for the CLI). They run
against a temporary SQLite database - no API keys or network needed:

yarn test        # judge-core + engine + the Go CLI
yarn typecheck   # both TypeScript packages, tests included

Or one workspace at a time:

yarn workspace @agentx/engine test
yarn workspace @agentx/judge-core test

The engine's integration suites boot the real engine as a subprocess. The ones that exercise
Postgres skip unless you point them at a server:

docker run -d --name agentx-pg-test -e POSTGRES_PASSWORD=agentx -e POSTGRES_DB=agentx -p 55432:5432 postgres:16-alpine
AGENTX_TEST_DB_URL=postgres://postgres:agentx@localhost:55432/agentx yarn workspace @agentx/engine test

That is worth running before anything touching storage/db.ts or an ingest path: Postgres is
where concurrent writes actually interleave, so a race that SQLite hides shows up there.

Both suites run under tsx, which storage/db.ts serves with better-sqlite3. The released
binary takes its bun:sqlite branch instead, so nothing above says anything about it - smoke-test
that separately before touching either branch, web.ts, or the shutdown path:

bun build engine/src/index.ts --compile --outfile /tmp/agentx-engine
./scripts/smoke-binary.sh /tmp/agentx-engine

Full distribution build

Compiles the engine to a Bun-compiled binary and builds the Go CLI, laid out exactly the way the
curl | bash install would:

./build.sh
./dist/agentx-server --dev

Once it's up:

AgentX self-host engine listening on http://localhost:4700
Local API key: agtx_local_...

--dev opens the dashboard in your browser automatically.

End-to-end smoke test

Confirms everything actually works, including a real Python SDK round-trip:

OPENAI_API_KEY=sk-... ./scripts/smoke-test.sh

Runs against SQLite by default; pass AGENTX_DB_URL=postgres://... (e.g. a throwaway Dockerized
Postgres) to verify against Postgres instead - see the script's own header for the exact command.

Dashboard release process

AgentX-eval-front (the app that builds into web/) is AgentX's private frontend dedicated to
this Governance dashboard. Rather than open-sourcing it, only its build output is public: a
workflow in that repo builds it in self-host mode and uploads the result as agentx-web.tar.gz
onto this repo's own GitHub releases. install.sh, build.sh, and the Dockerfile all fetch it
from there - nobody installing or building this repo, including outside contributors, ever needs
access to that private repo.

Maintainers cutting a new dashboard build: in AgentX-eval-front's Actions tab, run "Publish
self-host web bundle" (optionally targeting a specific release tag here; defaults to whatever's
currently latest). One-time setup: a fine-grained PAT scoped to this repo with release/contents
write access, saved as AgentX-eval-front's SELFHOST_RELEASE_TOKEN secret.

Project status

Trace, Evaluate, and Monitor are wired end-to-end and verified against the real Python SDK, both
SQLite and Postgres, and the compiled single-binary distribution - including a real OTLP/HTTP
receiver verified against both protobuf and JSON payloads. The dashboard covers Governance's full
self-host surface: Trace ingest (traces and sessions, with end-user feedback chips), Monitor
(patterns, trace- and session-scoped online evaluators, custom evaluators, topics, signals
triage, KPIs), Evaluate (runs, datasets curated straight from production traffic, standalone
evaluator configs, version history, per-case run comparison, Model Portability, and Playground),
Improve (the Prompt and Tool Schema registries with validated proposals and unregistered-tool
surfacing), CI Gates (recorded gate history plus a latest-run preview), and Overview (KPIs,
trends, topic map, Model Comparison, and Judge Calibration against reported outcomes and
end-user feedback).

Known gaps:

• The self-host build ships the full frontend bundle rather than one trimmed to just Governance -
  tracked as a follow-up in AgentX-eval-front.
• No hot-reload loop between an AgentX-eval-front dev server and this engine yet.
• Autotune's candidate-branch creation/evaluation/merging is out of scope, not deferred: it's
  fundamentally tied to AgentX's native agent config-branching system, which self-host doesn't
  have. The version-comparison and Prompt Registry features are the self-host analogs.
• Guardrail hasn't been started.
• Evaluate's async whole-run analysis (analyze_run/get_report) is out of scope for now.

See CHANGELOG.md for the detailed, narrative build/verification history behind
every feature above.

Contributing

Issues and PRs are welcome. There's no separate CONTRIBUTING.md yet - for anything beyond a
small fix, please open an issue first to discuss the approach. Run yarn install at the repo root
(a workspace install covering engine/ and packages/judge-core/ together), and see
Building from source for the dev loop and smoke test.

License

Apache-2.0 (see LICENSE).