Skip to content

fix(recipe): MB-923 verify VR200 FP8-MX environments#5073

Closed
svcnemo-autobot wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/vr200-fp8-mx-env-vars
Closed

fix(recipe): MB-923 verify VR200 FP8-MX environments#5073
svcnemo-autobot wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/vr200-fp8-mx-env-vars

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Background

The two VR200 FP8-MX recipes require explicit process environments owned by their recipe builders.

What changed

  • Keep both environment mappings directly on the VR200 recipes, matching all other flat recipes.
  • Clarify that these builders are VR200-specific rather than pure GB300 aliases.
  • Add exact recipe-level assertions for the required DeepSeek-V3 and Qwen3 values.

Details

No source overlay and no runner-owned environment fallback remain. The tests lock HybridEP NVL72 topology, TE fused-MLP/LayerNorm settings, and DeepSeek deterministic-kernel selection.

Tested

  • git diff --check
  • CI.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Comment thread scripts/performance/run_script.py Outdated
Comment on lines +27 to +37
VR200_FP8_MX_ENV_VARS = {
"CUDA_DEVICE_MAX_CONNECTIONS": 32,
"CUDNNFE_CLUSTER_OVERLAP_MARGIN": 8,
"NCCL_GRAPH_REGISTER": 0,
"NCCL_NVLS_ENABLE": 0,
"NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN": 32,
"NUM_OF_TOKENS_PER_CHUNK_COMBINE_API": 128,
"NVLINK_DOMAIN_SIZE": 72,
"NVTE_BWD_LAYERNORM_SM_MARGIN": 20,
"NVTE_CUTEDSL_FUSED_GROUPED_MLP": 1,
"NVTE_FWD_LAYERNORM_SM_MARGIN": 20,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@svcnemo-autobot i dont think this is the right place for env-vars.. can we check how we handle it for other recipes?

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test 0d00e35

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@yaoyu-33 yaoyu-33 added area:perf Performance optimizations and benchmarking bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Jul 24, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test 7ef11bc

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot svcnemo-autobot changed the title fix(perf): MB-923 restore VR200 FP8-MX environments fix(perf): MB-923 overlay flat recipe definitions Jul 24, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test 257e11c

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot svcnemo-autobot changed the title fix(perf): MB-923 overlay flat recipe definitions fix(recipe): MB-923 verify VR200 FP8-MX environments Jul 24, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test d531215

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

Validated the latest recipe-owned change locally on the exact PR CI merge image 75a36827cb9efd457596bb1011c75fe1f526d228:

  • ruff format --check on both VR200 recipe files and the recipe environment test: passed
  • ruff check on the same files: passed
  • focused test_perf_recipe_environment.py selection: 7 passed, 3 deselected

Full PR CI is also green with no failed checks. Ready for re-review.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:perf Performance optimizations and benchmarking bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants