AI Voice Interviewer — a LangWatch case study

A self-contained, runnable demo of an AI voice interviewer, shown as a
naive → hardened prompt evolution and stress-tested with the
LangWatch Scenario SDK — then taken one layer
further: the interviewer's own self-emitted scored report is independently
judged on the LangWatch platform (a Saved Evaluator + Dataset + Experiment, run
server-side on app.langwatch.ai). The same realtime voice agent is exercised
against three simulated candidates by an LLM-backed JudgeAgent that grades
interview conduct — headless, in a way you can run in CI. No company or
product is named anywhere; this doubles as an anonymized case study.

The story: a bare prompt ("just conduct the interview") versus a hardened
one (AI disclosure, no illegal questions, full rubric coverage, CV-grounded
probing, anti-buzzword pushback, and a grounded scored report). Run the same
suite against each variant and watch the judge expose exactly what the naive
agent misses and the hardened one gets right.

The three scenarios

test_interview_evolution.py runs three realtime voice interviews, each a
real scenario.run against a live GA OpenAI Realtime interviewer
(build_hirevoice_interviewer). All three are graded by a single
scenario.JudgeAgent on the same SHARED_CRITERIA set (AI disclosure, no
legally prohibited questions, professionalism / no false cheating accusation,
full rubric coverage, CV-probing, anti-buzzword pushback, and four
report-integrity criteria), so the naive↔hardened delta is visible across
identical evaluation axes. The judge is given the candidate's CV
(JUDGE_SYSTEM_PROMPT embeds CANONICAL_CV) so it can check whether the
interviewer grounded its questions and whether the report's scores are warranted.

All three candidates are voiced (true voice-to-voice): each
scenario.UserSimulatorAgent gets a voice=, so the realtime interviewer
receives audio on every turn. The same file runs naive or hardened depending
on INTERVIEWER_VARIANT — one env var, no test edits — and each run posts to a
variant-suffixed set_id (set_id_for("interview-evolution") →
interview-evolution-naive / interview-evolution-hardened).

Run it

source .venv/bin/activate

# Collect only — verifies the suite imports and parametrizes, hits no keys:
python -m pytest --collect-only

# BEFORE — naive agent (bare prompt). No assert on judge success; the criteria
# failures ARE the demo's "before" contrast, and the run completes green.
INTERVIEWER_VARIANT=naive    python -m pytest tests/scenarios/test_interview_evolution.py -s

# AFTER — hardened agent (full guardrails). hardened is the default, so plain
# pytest also runs it.
INTERVIEWER_VARIANT=hardened python -m pytest tests/scenarios/test_interview_evolution.py -s

-s surfaces the Scenario verbose output and the judge's reasoning inline. Voice
runs are slow (real TTS + STT + realtime WebSocket + several LLM calls per turn);
each test carries @pytest.mark.timeout(240) and @pytest.mark.flaky(reruns=2).
If the realtime model is not accessible on your OPENAI_API_KEY, the tests
skip (graceful-degradation gate) rather than fail.

Run a single scenario:

INTERVIEWER_VARIANT=hardened python -m pytest tests/scenarios/test_interview_evolution.py::test_qualified_candidate -s

The naive → hardened evolution

The same agent (build_hirevoice_interviewer, shared by the tests and the live
CLI) ships in two variants, selected by INTERVIEWER_VARIANT:

• naive — build_system_prompt(..., variant="naive"): role + spoken-format
  instruction + a minimal what-to-ask directive, and nothing else. No AI
  disclosure, no prohibited-question rules, no manipulation resistance, no
  anti-buzzword pushback, no emit_report. The CV is appended with no
  usage directive, so the agent asks generic questions.
• hardened — variant="hardened": the full guardrail prompt plus a strong
  CV-grounding directive and the emit_report self-assessment instruction.

The ground-truth signal is agent_report.json. outputs.py writes it only
when the interviewer actually called the emit_report tool (surfaced to the
judge as an EMIT_REPORT: text part by harness/realtime_adapter.py). So a hardened run
produces agent_report.json with citation-backed competency scores; a naive run
produces none — and each test asserts that presence/absence directly, rather
than trusting the judge (which has no filesystem access). The judge verdicts and
the audio recordings in outputs/ carry the before/after story regardless of
which variant ran last.

Both prompt variants are also pushed to LangWatch as versioned prompts under the
handle ai-interviewer-demo (the Prompts product) so the evolution has a
real, browsable version history:

PYTHONPATH=. .venv/bin/python scripts/prompt_versions.py

v1 is the bare naive prompt; the later version is the hardened (guardrailed) one.
The script is idempotent — re-running appends new versions, never clobbers.

Watch it on LangWatch

conftest.py calls scenario.setup_scenario_tracing() at startup. With
LANGWATCH_API_KEY set, every scenario.run streams to the
LangWatch Simulations viewer
and prints a Follow it live: … URL.

• Simulations — the naive and hardened batches land in separate sets
  (interview-evolution-naive vs interview-evolution-hardened), so the
  before/after is unmistakable, conversation by conversation, with the audio
  attached.
• Prompt version history — the ai-interviewer-demo handle in the Prompts
  product shows v1 (naive) → hardened side by side.

Project slug in app URLs: hirevoice-demo-VHElmI
(https://app.langwatch.ai/hirevoice-demo-VHElmI/…).

Layer 2: the report-judge (platform dogfood)

report_judge.py takes the demo one layer deeper: instead of judging the
conversation, it has LangWatch judge the quality of the interviewer's own
self-emitted report — and the judging work persists on the platform, not
just in a local script. End to end it:

1. Creates a Saved Evaluator (langevals/llm_score) on app.langwatch.ai
   via raw REST (POST /api/evaluators). The judge prompt encodes four
   criteria: (a) citation faithfulness, (b) score-evidence consistency,
   (c) rubric coverage, (d) recommendation soundness.
2. Builds + pushes a Dataset from the local
   outputs/evolution_qualified/{transcript.json, agent_report.json}
   (POST /api/dataset/ then …/entries) — one row, transcript + agent_report
   columns.
3. Runs a batch Experiment over that dataset (langwatch.evaluation.Evaluation
   → POST /api/experiment/init, …/evaluations/evaluators/{id}/evaluate,
   …/evaluations/batch/log_results). The judge LLM call runs server-side;
   the script only seeds and triggers.

It also exercises the failure/consequence paths (auth 401, induced LLM error,
dataset-push failure, experiment-launch failure, idempotency, and a
negative-control contrast over degraded input) and prints a verification block
with every app URL and AC proof.

PYTHONPATH=. .venv/bin/python scripts/report_judge.py    # needs LANGWATCH_API_KEY (+ LANGWATCH_PROJECT_ID for scoping checks)

Design + the 15 acceptance criteria: docs/layer2/REPORT_JUDGE_PLAN.md.
The confirmed endpoints / field contracts it was built against:
docs/layer2/BUILD_NOTES.md.

Files

Install

uv venv
uv pip install -r requirements.txt

Environment variables

Copy .env.example to .env and fill it in. conftest.py (and the standalone
scripts) load .env via python-dotenv; .env is gitignored — never commit it.

Live runs hit paid voice + LLM providers — real TTS, STT, realtime
WebSocket, and multiple LLM calls per turn. Expect modest provider cost per run.

Talk to it with your voice

PYTHONPATH=. python scripts/talk_to_interviewer_voice.py

Run it on a machine with a working microphone and headphones (headphones are
required — raw mic capture has no echo cancellation, so over open speakers the
interviewer would hear itself and barge in on itself). It connects the
interviewer to the GA OpenAI Realtime API over a bidirectional WebSocket with
server-side VAD, so it speaks after you stop and you can talk over it to
interrupt (true barge-in). Because it uses hirevoice_system_prompt(), the live
agent is byte-identical to the tested one — set INTERVIEWER_VARIANT=naive to
hear the bare agent dive straight into questions, or hardened (default) to hear
it disclose it is an AI first. On macOS, grant your terminal Microphone access
(System Settings → Privacy & Security → Microphone) first. Ctrl-C to quit.

Honesty caveats

• The challenging scenario surfaces a real over-accommodation finding. Under
  high difficulty (heavy accent + background noise + interruptions), the hardened
  interviewer sometimes over-accommodates ("we can slow down", "I'll keep my
  volume moderate") and under-probes the competencies — so the conduct judge
  can fail it. That is product signal, not a test bug, so test_challenging_candidate
  records the verdict (and still asserts a grounded report) rather than asserting
  conduct success. Whether that over-accommodation is acceptable is left as an
  open product decision.
• The report-judge accretes objects on the live project. report_judge.py
  creates real evaluators / datasets / experiments on app.langwatch.ai (the
  create API issues a fresh nanoid every time — names are timestamp-stamped, not
  deduplicated), so repeated runs leave duplicate objects in the project that a
  teammate may need to clean up.