fix(instrumentation): all 4 usage sources are 'if' not 'elif'#59
Merged
Conversation
The pre-fix `extract_usage_from_response` walked the 4
source branches as an elif chain:
```
if hasattr(response, 'usage_metadata'): # empty dict
elif hasattr(response, 'generations'): # LLMResult
elif hasattr(response, 'usage'):
elif hasattr(response, 'response_metadata'): # real tokens
```
A LangChain AIMessage can carry token info on multiple
attributes at once. When the first branch's `hasattr`
returns True but the value is an empty / 0/0/0 dict
(LangChain-internal init state for streaming, callbacks,
some provider wrappers), every subsequent `elif` was
skipped — including the one carrying the real token
counts.
Reproducer (covered by the new test):
```
response.usage_metadata = {input_tokens: 0, output_tokens: 0, total_tokens: 0}
response.response_metadata = {token_usage: {prompt_tokens: 26, completion_tokens: 48, total_tokens: 74}}
# Pre-fix: ships tokens=0 to /track. LLM call invisible on dashboard.
# Post-fix: ships tokens=74, input_tokens=26, output_tokens=48.
```
Switched all 4 source branches to plain `if` so each one
attempts its read; later branches naturally overwrite the
zero default when the earlier branch's value is empty. In
practice LangChain providers never put conflicting token
counts on two attributes of the same response, so
last-wins is safe; the docstring on the function is
updated to spell that out.
Added `test_extract_usage_metadata_zero_response_metadata_real`
that pins the regression: empty usage_metadata + populated
response_metadata must surface the real numbers.
All 39 tests in test_langgraph_callback.py still pass;
test_extractors.py + test_instrumentation_phase41.py also
green (no regression in the wire-shape normalization
testers).
Co-Authored-By: Hermes <noreply@nous.research>
Pairs with 4840a33 (fix-instrumentation-langchain-elif-chain). Wire format unchanged; pure version bump + changelog entry so the SDK_MIN_VERSION floor is up to date with the bug fix. Co-Authored-By: Hermes <noreply@nous.research>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extract_usage_from_responseinsrc/nullrun/instrumentation/langgraph.pywalked the 4 token-extraction source branches as an
elifchain:A LangChain AIMessage can carry token info on multiple attributes at
once. When the first branch's
hasattrreturned True but the valuewas an empty dict or 0/0/0 (streaming init state, some provider
wrappers), every subsequent
elifwas skipped -- including the onecarrying the real token counts. The SDK shipped
tokens=0to/api/v1/track, making the LLM call invisible on the dashboard.Reproducer (now pinned by the new test)
Fix
Switched all 4 source branches from
elifto plainifso eachone attempts its read; later branches naturally overwrite the zero
default when the earlier branch's value is empty. In practice
LangChain providers never put conflicting token counts on two
attributes of the same response, so "last-wins" is safe.
Tests
test_extract_usage_metadata_zero_response_metadata_realintests/test_langgraph_callback.pypins the regression.test_langgraph_callback.pytests pass.test_extractors.py+test_instrumentation_phase41.pytests pass (no regression in the wire-shape normalization
testers).
Coverage scope
The same
extract_usage_from_responseis the single source oftruth for all 5 framework integrations --
autogen,crewai,llama_index, the LangChain callback path, and the OpenAI AgentsSDK tracer all import or re-use the same
NullRunCallback/extraction logic. Fix in this one place closes the bug for every
framework at once.
The 5 direct HTTP extractors in
auto.py(_openai_extractor,_anthropic_extractor,_gemini_extractor,_cohere_extractor,_bedrock_extractor) parse the raw HTTP response body directlywith no chain -- they do not exhibit this bug pattern. Audited
separately.
Version
src/nullrun/__version__.py).SDK_MIN_VERSION_FOR_V3unchanged ("0.12.0"); backends on1.0.0 keep working unchanged.
Co-Authored-By: Hermes noreply@nous.research