Skip to content

fix(gpt-oss): route packed attention through THD#3226

Draft
akoumpa wants to merge 1 commit into
mainfrom
akoumpa/fix/amint28-gpt-oss-qkv-bshd
Draft

fix(gpt-oss): route packed attention through THD#3226
akoumpa wants to merge 1 commit into
mainfrom
akoumpa/fix/amint28-gpt-oss-qkv-bshd

Conversation

@akoumpa

@akoumpa akoumpa commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Infer GPT-OSS THD/packed attention from cu_seqlens metadata when qkv_format is not explicitly set.
  • Preserve singleton-batch packed outputs while passing TE attention qkv_format=thd and clearing the padding mask before backend preprocessing.
  • Add GPT-OSS regression coverage for packed metadata routing and TE backend preprocessing arguments.

Root Cause

The packed recipe passed THD metadata through cu_seqlens, but GptOssForCausalLM.forward only treated the input as packed when attn_kwargs["qkv_format"] == "thd". That left TE preprocessing able to route flattened packed tensors through the BSHD path, which produced the runtime assertion from AMINT-28: Queries, keys and values must be 4D tensors when qkv_format='bshd'.

Validation

Note: the successful Nemo-CI run used the x86 EOS/H100 lane and INSTALL_FA3=false to bypass unrelated pre-test flash_attn_3 container build failures observed in earlier attempts.

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@akoumpa

akoumpa commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f915978

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant