Skip to content

Fix 4-bit split QKV projection#1502

Open
abhinav-bellapu wants to merge 2 commits into
TransformerLensOrg:mainfrom
abhinav-bellapu:codex/fix-4bit-split-qkv
Open

Fix 4-bit split QKV projection#1502
abhinav-bellapu wants to merge 2 commits into
TransformerLensOrg:mainfrom
abhinav-bellapu:codex/fix-4bit-split-qkv

Conversation

@abhinav-bellapu

Copy link
Copy Markdown
Contributor

Fixes #737.

Summary

  • Add a shared 4-bit Q/K/V projection helper for regular attention.
  • Preserve the existing flattened 4-bit projection behavior for normal [batch, pos, d_model] inputs.
  • Handle split Q/K/V inputs by selecting the matching input-head/output-head projection slice, matching complex_attn_linear.
  • Add CPU-safe regression coverage with fake bitsandbytes matmul, so the split 4-bit path is tested without requiring a quantized 7B model locally.

Why

The old 4-bit path projected split inputs to [batch, pos, input_head, d_model] and then tried to reshape directly to [batch, pos, head, d_head], which reproduces the shape error in #737. Split Q/K/V inputs need head-specific projections: each input head should keep only the corresponding output-head slice.

Validation

  • make format
  • make check-format
  • uv run --python 3.11 pytest tests/unit/components/test_attention.py -q
  • uv run --python 3.11 mypy .
  • make unit-test

@abhinav-bellapu abhinav-bellapu marked this pull request as ready for review July 10, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug Report] Q cannot be reshaped correctly when model is loaded in 4bit

1 participant