Repository
harness-claude-code-usage-repro
Repro: @ai-sdk/harness-claude-code finish-step reports the whole turn's cumulative usage on the last step, not that step's own
- input
- output
- Expected finish-step #2 usage
- Expected
- Actual finish-step #2 usage
- Actual
Specs
- inputoutput
- Expected finish-step #2 usageExpected
- Actual finish-step #2 usageActual
README
@ai-sdk/harness-claude-code: last-step finish-step usage is the whole turn's cumulative total, not that step's own usage
Minimal reproduction. npm install && npm run repro — no API key needed, it
drives the package's own bridge event-processing code against a synthetic
(but protocol-accurate) Claude Code CLI stream.
The bug
On any turn with two or more model calls (i.e. any turn with at least one
tool round trip before the final answer — subagents not required), the
finish-step part emitted for the last step reports the whole turn's
cumulative usage (every step summed) instead of that step's own usage.
Concretely, in packages/harness-claude-code/src/bridge/index.ts,
on the CLI's turn-ending result message:
const usage = msg.usage ?? msg.message?.usage; // cumulative, whole-turn total
const harnessUsage = mapUsage(usage);
if (harnessUsage) turnUsage = harnessUsage;
...
if (streamEventState.stepOpen) {
emitFinishStep({
state: streamEventState,
emit,
usage: harnessUsage ?? streamEventState.pendingStepUsage,
});
}
streamEventState.pendingStepUsage is already sitting there with the right
value — the final step's own assistant message set it correctly (see
create-emit-stream-event.ts).
But a text-only final response never runs through closeStepIfReady() (that
only fires from tool-result handling or finishApprovalStep), so the step is
still open when result arrives — and harnessUsage (the cumulative total)
wins over pendingStepUsage (the correct, final-step-only value) because of
the ?? precedence.
Expected vs. actual
Given a turn with:
- step 1 (a Bash tool call): 1000 input / 50 output tokens
- step 2 (the final text answer): 1800 input / 70 output tokens of its own
- CLI's
result.usage(cumulative): 2800 input / 120 output tokens
| input | output | |
|---|---|---|
Expected finish-step #2 usage |
1800 | 70 |
Actual finish-step #2 usage |
2800 | 120 |
Why it matters
Anything consuming finish-step/finish to answer "how many tokens did the
model just use" — most obviously, a live "how full is the context window
right now" indicator — overcounts by the sum of every prior step in the turn,
on every ordinary multi-tool-call turn. It's easy to miss in testing because
a single-model-call turn (no tools, or the harness's built-in tools handled
in one shot) never shows a discrepancy: cumulative-so-far and "this step's
usage" are numerically identical when there's only one step.
This is a different bug from #18618 (subagent Task-tool activity bleeding
into the parent step's usage) — that one is fixed as of
[email protected]. This one reproduces unchanged against the
latest published version as of this writing
(@ai-sdk/[email protected], see package.json).
What this reproduction does
repro.ts imports the package's actual shipped bridge source (by relative
filesystem path into node_modules, since the package's exports map only
publishes the top-level entry point — the npm tarball does ship the src/
directory, pinned to whatever version is installed) and feeds it a
protocol-accurate two-step Claude Code CLI stream-json sequence: one Bash
tool round trip, then a final text answer, then the terminal result
message. It asserts the second finish-step's usage equals that step's own
usage. It currently fails (exit code 1) — that's the bug. If/when this is
fixed upstream, it should print PASS and exit 0.