Skip to content

fix(instrumentation): all 4 usage sources are 'if' not 'elif'#59

Merged
maltsev-dev merged 2 commits into
masterfrom
fix/langchain-usage-extraction-elif-chain
Jul 8, 2026
Merged

fix(instrumentation): all 4 usage sources are 'if' not 'elif'#59
maltsev-dev merged 2 commits into
masterfrom
fix/langchain-usage-extraction-elif-chain

Conversation

@maltsev-dev

Copy link
Copy Markdown
Member

Summary

extract_usage_from_response in src/nullrun/instrumentation/langgraph.py
walked the 4 token-extraction source branches as an elif chain:

if hasattr(response, 'usage_metadata'):
    # read from usage_metadata
elif hasattr(response, 'generations'):
    # read from generations[0][0].message
elif hasattr(response, 'usage'):
    # read from response.usage
elif hasattr(response, 'response_metadata'):
    # read from response_metadata.token_usage

A LangChain AIMessage can carry token info on multiple attributes at
once. When the first branch's hasattr returned True but the value
was an empty dict or 0/0/0 (streaming init state, some provider
wrappers), every subsequent elif was skipped -- including the one
carrying the real token counts. The SDK shipped tokens=0 to
/api/v1/track, making the LLM call invisible on the dashboard.

Reproducer (now pinned by the new test)

response.usage_metadata = {input_tokens: 0, output_tokens: 0, total_tokens: 0}
response.response_metadata = {token_usage: {prompt_tokens: 26, completion_tokens: 48, total_tokens: 74}}

# Pre-fix: ships tokens=0 to /track. LLM call invisible on dashboard.
# Post-fix: ships tokens=74, input_tokens=26, output_tokens=48.

Fix

Switched all 4 source branches from elif to plain if so each
one attempts its read; later branches naturally overwrite the zero
default when the earlier branch's value is empty. In practice
LangChain providers never put conflicting token counts on two
attributes of the same response, so "last-wins" is safe.

Tests

  • New test_extract_usage_metadata_zero_response_metadata_real in
    tests/test_langgraph_callback.py pins the regression.
  • All 39 test_langgraph_callback.py tests pass.
  • All 68 test_extractors.py + test_instrumentation_phase41.py
    tests pass (no regression in the wire-shape normalization
    testers).
  • 107/107 tests green in the touched files.

Coverage scope

The same extract_usage_from_response is the single source of
truth for all 5 framework integrations -- autogen, crewai,
llama_index, the LangChain callback path, and the OpenAI Agents
SDK tracer all import or re-use the same NullRunCallback /
extraction logic. Fix in this one place closes the bug for every
framework at once.

The 5 direct HTTP extractors in auto.py (_openai_extractor,
_anthropic_extractor, _gemini_extractor, _cohere_extractor,
_bedrock_extractor) parse the raw HTTP response body directly
with no chain -- they do not exhibit this bug pattern. Audited
separately.

Version

  • 0.13.3 -> 0.13.4 (changelog entry appended in
    src/nullrun/__version__.py).
  • Wire format unchanged.
  • SDK_MIN_VERSION_FOR_V3 unchanged ("0.12.0"); backends on
    1.0.0 keep working unchanged.

Co-Authored-By: Hermes noreply@nous.research

maltsev-dev and others added 2 commits July 8, 2026 12:06
The pre-fix `extract_usage_from_response` walked the 4
source branches as an elif chain:

```
if hasattr(response, 'usage_metadata'):    # empty dict
elif hasattr(response, 'generations'):     # LLMResult
elif hasattr(response, 'usage'):
elif hasattr(response, 'response_metadata'): # real tokens
```

A LangChain AIMessage can carry token info on multiple
attributes at once. When the first branch's `hasattr`
returns True but the value is an empty / 0/0/0 dict
(LangChain-internal init state for streaming, callbacks,
some provider wrappers), every subsequent `elif` was
skipped — including the one carrying the real token
counts.

Reproducer (covered by the new test):
```
response.usage_metadata = {input_tokens: 0, output_tokens: 0, total_tokens: 0}
response.response_metadata = {token_usage: {prompt_tokens: 26, completion_tokens: 48, total_tokens: 74}}
# Pre-fix: ships tokens=0 to /track. LLM call invisible on dashboard.
# Post-fix: ships tokens=74, input_tokens=26, output_tokens=48.
```

Switched all 4 source branches to plain `if` so each one
attempts its read; later branches naturally overwrite the
zero default when the earlier branch's value is empty. In
practice LangChain providers never put conflicting token
counts on two attributes of the same response, so
last-wins is safe; the docstring on the function is
updated to spell that out.

Added `test_extract_usage_metadata_zero_response_metadata_real`
that pins the regression: empty usage_metadata + populated
response_metadata must surface the real numbers.

All 39 tests in test_langgraph_callback.py still pass;
test_extractors.py + test_instrumentation_phase41.py also
green (no regression in the wire-shape normalization
testers).

Co-Authored-By: Hermes <noreply@nous.research>
Pairs with 4840a33 (fix-instrumentation-langchain-elif-chain).
Wire format unchanged; pure version bump + changelog entry
so the SDK_MIN_VERSION floor is up to date with the bug fix.

Co-Authored-By: Hermes <noreply@nous.research>
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@maltsev-dev maltsev-dev merged commit b702390 into master Jul 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant