NCCL timeout when GRPO training with vllm

[edwardzjl]

I'm working on reproducing the experiments from this awesome blog. However, I’m encountering an issue when enabling vLLM.

When I run the training without vLLM, everything works fine, and I get an approximately training speed of 115.96s/it.

part of the training log:

INFO:__main__:*** Starting training 2025-02-21 08:08:48 for 3.0 epochs***
INFO:__main__:*** Starting training 2025-02-21 08:08:48 for 3.0 epochs***
2025-02-21 08:08:48,573 - __main__ - INFO - *** Starting training 2025-02-21 08:08:48 for 3.0 epochs***
2025-02-21 08:08:48,574 - __main__ - INFO - nccl_cumem_enable: 0
INFO:__main__:nccl_cumem_enable: 0
2025-02-21 08:08:48,574 - __main__ - INFO - nccl_cumem_enable: 0
INFO:__main__:nccl_cumem_enable: 0
INFO:__main__:*** Starting training 2025-02-21 08:08:48 for 3.0 epochs***
2025-02-21 08:08:48,574 - __main__ - INFO - nccl_cumem_enable: 0
INFO:__main__:nccl_cumem_enable: 0
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
  0%|                                                                                                                                                                                                | 0/450 [00:00<?, ?it/s]`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
{'loss': 0.0, 'grad_norm': 1.5064643621444702, 'learning_rate': 7.142857142857142e-08, 'rewards/format_reward_func': 0.234375, 'rewards/equation_reward_func': 0.015625, 'reward': 0.25, 'reward_std': 0.4207531735301018, 'completion_length': 301.15625, 'kl': 0.0, 'epoch': 0.0}
{'loss': 0.0, 'grad_norm': 2.2298247814178467, 'learning_rate': 1.4285714285714285e-07, 'rewards/format_reward_func': 0.3125, 'rewards/equation_reward_func': 0.0, 'reward': 0.3125, 'reward_std': 0.3221687823534012, 'completion_length': 313.96875, 'kl': 3.722056598576273e-05, 'epoch': 0.0}
{'loss': 0.0, 'grad_norm': 2.245654344558716, 'learning_rate': 2.1428571428571426e-07, 'rewards/format_reward_func': 0.40625, 'rewards/equation_reward_func': 0.015625, 'reward': 0.421875, 'reward_std': 0.49775634706020355, 'completion_length': 247.390625, 'kl': 0.0007590312125103083, 'epoch': 0.0}
{'loss': 0.0, 'grad_norm': 3.145155429840088, 'learning_rate': 2.857142857142857e-07, 'rewards/format_reward_func': 0.484375, 'rewards/equation_reward_func': 0.03125, 'reward': 0.515625, 'reward_std': 0.5373184233903885, 'completion_length': 173.875, 'kl': 0.002920848364738049, 'epoch': 0.0}
  2%|███▌                                                                                                                                                                                | 9/450 [21:35<14:12:16, 115.96s/it] 

When I enable vLLM, I hit an NCCL timeout before training even starts.

Here are some extra information:

• I set NCCL_CUMEM_ENABLE=0 following the vLLM's documentation: https://docs.vllm.ai/en/v0.7.3/getting_started/troubleshooting.html#known-issues.
• I'm using first 4 GPUs for training and the 5th GPU for vllm. So CUDA_VISIBLE_DEVICES="0,1,2,3,4", torchrun --nproc_per_node 4 and --vllm_device "cuda:4".
• As I am running on V100, and FlashAttention does not supports V100, my vllm fallbacks to xformers backend. There's a small issue of the xformers backend when not running on the first GPU (cuda:0), so I made some modifications to vLLM ([Bugfix] Initialize attention bias on the same device as Query/Key/Value vllm-project/vllm#13468).
• Also related to V100, I cannot use prefix caching on V100 ([Bug]: Crash with --enable-prefix-caching enabled vllm-project/vllm#3944), so I turned enable_prefix_caching=False when initializing the LLM.
• I’ve also tried enforcing IPv4 networking by setting NCCL_SOCKET_FAMILY=AF_INET, adjusting the NCCL P2P level with NCCL_P2P_LEVEL=NVL, and even disabling P2P with NCCL_P2P_DISABLE=1, but none of these solutions resolved the issue.

part of the error log:

[rank3]:[I221 08:35:07.859408571 ProcessGroupNCCL.cpp:914] [PG ID 1 PG GUID 1 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.21.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 1, TORCH_NCCL_DUMP_ON_TIMEOUT: 1, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 1, TORCH_NCCL_ENABLE_TIMING: 1, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: DETAIL, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 480, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0, TORCH_NCCL_CUDA_EVENT_CACHE: 0, TORCH_NCCL_LOG_CPP_STACK_ON_UNCLEAN_SHUTDOWN: 1
[rank2]:[I221 08:35:07.861143314 ProcessGroupWrapper.cpp:587] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[151936, 1536], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[rank3]:[I221 08:35:07.861209797 ProcessGroupWrapper.cpp:587] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[151936, 1536], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[rank0]:[W221 08:45:06.568395840 socket.cpp:462] [c10d] waitForInput: poll for socket SocketImpl(fd=42, addr=[localhost]:34988, remote=[localhost]:29500) returned 0, likely a timeout
[rank0]:[W221 08:45:06.569851551 socket.cpp:487] [c10d] waitForInput: socket SocketImpl(fd=42, addr=[localhost]:34988, remote=[localhost]:29500) timed out after 600000ms
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/jovyan/part4/grpo.py", line 277, in <module>
[rank0]:     grpo_function(model_args, script_args, training_args)
[rank0]:   File "/home/jovyan/part4/grpo.py", line 201, in grpo_function
[rank0]:     trainer = GRPOTrainer(
[rank0]:               ^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 444, in __init__
[rank0]:     self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)
[rank0]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/trl/models/utils.py", line 247, in prepare_deepspeed
[rank0]:     model, *_ = deepspeed.initialize(model=model, config=config_kwargs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/__init__.py", line 193, in initialize
[rank0]:     engine = DeepSpeedEngine(args=args,
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 271, in __init__
[rank0]:     self._configure_distributed_model(model)
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1200, in _configure_distributed_model
[rank0]:     self.data_parallel_group = groups._get_data_parallel_group()
[rank0]:                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/utils/groups.py", line 410, in _get_data_parallel_group
[rank0]:     return _clone_world_group()
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/utils/groups.py", line 367, in _clone_world_group
[rank0]:     _WORLD_GROUP = dist.new_group(ranks=range(dist.get_world_size()))
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 185, in new_group
[rank0]:     return cdb.new_group(ranks)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 357, in new_group
[rank0]:     return torch.distributed.new_group(ranks)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
[rank0]:     func_return = func(*args, **kwargs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4565, in new_group
[rank0]:     return _new_group_with_tag(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4648, in _new_group_with_tag
[rank0]:     pg, pg_store = _new_process_group_helper(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1840, in _new_process_group_helper
[rank0]:     backend_class = _create_process_group_wrapper(
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4255, in _create_process_group_wrapper
[rank0]:     helper_pg = ProcessGroupGloo(store, rank, world_size, timeout=timeout)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.distributed.DistStoreError: wait timeout after 600000ms, keys: //worker/attempt_0/default_pg/0//7//cuda//pg_wrapper:7/0/1
[rank3]:[E221 08:45:07.886838878 ProcessGroupGloo.cpp:143] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank2]:[E221 08:45:07.887390966 ProcessGroupGloo.cpp:143] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank3]: Traceback (most recent call last):
[rank3]:   File "/home/jovyan/part4/grpo.py", line 277, in <module>
[rank3]:     grpo_function(model_args, script_args, training_args)
[rank3]:   File "/home/jovyan/part4/grpo.py", line 201, in grpo_function
[rank3]:     trainer = GRPOTrainer(
[rank3]:               ^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 444, in __init__
[rank3]:     self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)
[rank3]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/trl/models/utils.py", line 247, in prepare_deepspeed
[rank3]:     model, *_ = deepspeed.initialize(model=model, config=config_kwargs)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/__init__.py", line 193, in initialize
[rank3]:     engine = DeepSpeedEngine(args=args,
[rank3]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 271, in __init__
[rank3]:     self._configure_distributed_model(model)
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1213, in _configure_distributed_model
[rank3]:     self._broadcast_model()
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1131, in _broadcast_model
[rank3]:     dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group)
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 224, in broadcast
[rank3]:     return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 206, in broadcast
[rank3]:     return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2421, in broadcast
[rank3]:     work = group.broadcast([tensor], opts)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: RuntimeError: Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank3]:  Original exception: 
[rank3]: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 600000ms for send operation to complete
[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/jovyan/part4/grpo.py", line 277, in <module>
[rank2]:     grpo_function(model_args, script_args, training_args)
[rank2]:   File "/home/jovyan/part4/grpo.py", line 201, in grpo_function
[rank2]:     trainer = GRPOTrainer(
[rank2]:               ^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 444, in __init__
[rank2]:     self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)
[rank2]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/trl/models/utils.py", line 247, in prepare_deepspeed
[rank2]:     model, *_ = deepspeed.initialize(model=model, config=config_kwargs)
[rank2]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/__init__.py", line 193, in initialize
[rank2]:     engine = DeepSpeedEngine(args=args,
[rank2]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 271, in __init__
[rank2]:     self._configure_distributed_model(model)
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1213, in _configure_distributed_model
[rank2]:     self._broadcast_model()
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1131, in _broadcast_model
[rank2]:     dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group)
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 224, in broadcast
[rank2]:     return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 206, in broadcast
[rank2]:     return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2421, in broadcast
[rank2]:     work = group.broadcast([tensor], opts)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank2]:  Original exception: 
[rank2]: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 600000ms for send operation to complete

(I can provide the full log if that's helpful. But I did not find any information there.)

Here's the trl env output:

$ trl env
[2025-02-21 08:58:07,927] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/jovyan/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
INFO 02-21 08:58:10 __init__.py:190] Automatically detected platform cuda.

Copy-paste the following information when reporting an issue:

- Platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35
- Python version: 3.11.11
- PyTorch version: 2.5.1+cu121
- CUDA device(s): Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB, Tesla V100-SXM2-32GB
- Transformers version: 4.50.0.dev0
- Accelerate version: 1.3.0
- Accelerate config: 
  - compute_environment: LOCAL_MACHINE
  - distributed_type: FSDP
  - mixed_precision: fp16
  - use_cpu: False
  - debug: False
  - num_processes: 4
  - machine_rank: 0
  - num_machines: 1
  - rdzv_backend: static
  - same_network: True
  - main_training_function: main
  - enable_cpu_affinity: False
  - fsdp_config: {'fsdp_auto_wrap_policy': 'NO_WRAP', 'fsdp_backward_prefetch_policy': 'BACKWARD_PRE', 'fsdp_cpu_ram_efficient_loading': False, 'fsdp_offload_params': False, 'fsdp_sharding_strategy': 1, 'fsdp_state_dict_type': 'FULL_STATE_DICT', 'fsdp_use_orig_params': True}
  - downcast_bf16: no
  - tpu_use_cluster: False
  - tpu_use_sudo: False
  - tpu_env: []
- Datasets version: 3.2.0
- HF Hub version: 0.27.1
- TRL version: 0.15.1
- bitsandbytes version: not installed
- DeepSpeed version: 0.16.3
- Diffusers version: not installed
- Liger-Kernel version: 0.5.2
- LLM-Blender version: not installed
- OpenAI version: 1.63.2
- PEFT version: 0.14.0

Has anyone encountered a similar problem or have any advice on how to resolve this? Any help would be greatly appreciated!

[Christoph-XJ]

I ran into the exact same problem when training a 1.5B model with 8 A100s. Can't do anything about it now.

[haizhian]

我正在努力复制这个很棒的博客中的实验。但是，我在启用 vLLM 时遇到问题。

当我在没有 vLLM 的情况下运行训练时，一切正常，我得到的训练速度约为 115.96 秒/它。

训练日志的一部分：
启用 vLLM 后，我甚至在训练开始之前就达到了 NCCL 超时。

以下是一些额外信息：

• 我按照 vLLM 的文档设置：https://docs.vllm.ai/en/v0.7.3/getting_started/troubleshooting.html#known-issues。`NCCL_CUMEM_ENABLE=0`
• 我使用前 4 个 GPU 进行训练，使用第 5 个 GPU 进行 vllm。所以 和 .CUDA_VISIBLE_DEVICES="0,1,2,3,4"``torchrun --nproc_per_node 4``--vllm_device "cuda:4"
• 由于我在 V100 上运行，而 FlashAttention 不支持 V100，因此我的 vllm 回退到后端。当不在第一个 GPU （cuda：0） 上运行时，后端有一个小问题，所以我对 vLLM （xformers``xformers[错误修正]在与 Query/Key/Value 相同的设备上初始化注意力偏差 vllm-project/vllm#13468).
• 同样与 V100 相关，我无法在 V100 上使用前缀缓存 （[Bug]：启用 --enable-prefix-caching vllm-project/vllm#3944 时崩溃），所以我在初始化 LLM 时转动了。enable_prefix_caching=False
• 我还尝试过通过设置 来强制实施 IPv4 网络，使用 调整 NCCL P2P 级别，甚至使用 禁用 P2P，但这些解决方案都没有解决问题。NCCL_SOCKET_FAMILY=AF_INET``NCCL_P2P_LEVEL=NVL``NCCL_P2P_DISABLE=1

部分错误日志：
[rank3]:[I221 08:35:07.859408571 ProcessGroupNCCL.cpp:914] [PG ID 1 PG GUID 1 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.21.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 1, TORCH_NCCL_DUMP_ON_TIMEOUT: 1, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 1, TORCH_NCCL_ENABLE_TIMING: 1, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: DETAIL, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 480, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0, TORCH_NCCL_CUDA_EVENT_CACHE: 0, TORCH_NCCL_LOG_CPP_STACK_ON_UNCLEAN_SHUTDOWN: 1
[rank2]:[I221 08:35:07.861143314 ProcessGroupWrapper.cpp:587] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[151936, 1536], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[rank3]:[I221 08:35:07.861209797 ProcessGroupWrapper.cpp:587] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[151936, 1536], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[rank0]:[W221 08:45:06.568395840 socket.cpp:462] [c10d] waitForInput: poll for socket SocketImpl(fd=42, addr=[localhost]:34988, remote=[localhost]:29500) returned 0, likely a timeout
[rank0]:[W221 08:45:06.569851551 socket.cpp:487] [c10d] waitForInput: socket SocketImpl(fd=42, addr=[localhost]:34988, remote=[localhost]:29500) timed out after 600000ms
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/jovyan/part4/grpo.py", line 277, in
[rank0]: grpo_function(model_args, script_args, training_args)
[rank0]: File "/home/jovyan/part4/grpo.py", line 201, in grpo_function
[rank0]: trainer = GRPOTrainer(
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 444, in init
[rank0]: self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/trl/models/utils.py", line 247, in prepare_deepspeed
[rank0]: model, *_ = deepspeed.initialize(model=model, config=config_kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize
[rank0]: engine = DeepSpeedEngine(args=args,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 271, in init
[rank0]: self._configure_distributed_model(model)
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1200, in _configure_distributed_model
[rank0]: self.data_parallel_group = groups._get_data_parallel_group()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/utils/groups.py", line 410, in _get_data_parallel_group
[rank0]: return _clone_world_group()
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/utils/groups.py", line 367, in _clone_world_group
[rank0]: _WORLD_GROUP = dist.new_group(ranks=range(dist.get_world_size()))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 185, in new_group
[rank0]: return cdb.new_group(ranks)
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 357, in new_group
[rank0]: return torch.distributed.new_group(ranks)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
[rank0]: func_return = func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4565, in new_group
[rank0]: return _new_group_with_tag(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4648, in _new_group_with_tag
[rank0]: pg, pg_store = _new_process_group_helper(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1840, in _new_process_group_helper
[rank0]: backend_class = _create_process_group_wrapper(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4255, in create_process_group_wrapper
[rank0]: helper_pg = ProcessGroupGloo(store, rank, world_size, timeout=timeout)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.distributed.DistStoreError: wait timeout after 600000ms, keys: //worker/attempt_0/default_pg/0//7//cuda//pg_wrapper:7/0/1
[rank3]:[E221 08:45:07.886838878 ProcessGroupGloo.cpp:143] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank2]:[E221 08:45:07.887390966 ProcessGroupGloo.cpp:143] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank3]: Traceback (most recent call last):
[rank3]: File "/home/jovyan/part4/grpo.py", line 277, in
[rank3]: grpo_function(model_args, script_args, training_args)
[rank3]: File "/home/jovyan/part4/grpo.py", line 201, in grpo_function
[rank3]: trainer = GRPOTrainer(
[rank3]: ^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 444, in init
[rank3]: self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/trl/models/utils.py", line 247, in prepare_deepspeed
[rank3]: model, * = deepspeed.initialize(model=model, config=config_kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize
[rank3]: engine = DeepSpeedEngine(args=args,
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 271, in init
[rank3]: self._configure_distributed_model(model)
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1213, in _configure_distributed_model
[rank3]: self._broadcast_model()
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1131, in _broadcast_model
[rank3]: dist.broadcast(p.data, groups.get_broadcast_src_rank(), group=self.seq_data_parallel_group)
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 224, in broadcast
[rank3]: return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 206, in broadcast
[rank3]: return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2421, in broadcast
[rank3]: work = group.broadcast([tensor], opts)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: RuntimeError: Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank3]: Original exception:
[rank3]: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 600000ms for send operation to complete
[rank2]: Traceback (most recent call last):
[rank2]: File "/home/jovyan/part4/grpo.py", line 277, in
[rank2]: grpo_function(model_args, script_args, training_args)
[rank2]: File "/home/jovyan/part4/grpo.py", line 201, in grpo_function
[rank2]: trainer = GRPOTrainer(
[rank2]: ^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 444, in init
[rank2]: self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/trl/models/utils.py", line 247, in prepare_deepspeed
[rank2]: model, * = deepspeed.initialize(model=model, config=config_kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize
[rank2]: engine = DeepSpeedEngine(args=args,
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 271, in init
[rank2]: self._configure_distributed_model(model)
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1213, in _configure_distributed_model
[rank2]: self._broadcast_model()
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1131, in _broadcast_model
[rank2]: dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group)
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper
[rank2]: return func(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 224, in broadcast
[rank2]: return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 206, in broadcast
[rank2]: return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank2]: return func(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/opt/conda/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2421, in broadcast
[rank2]: work = group.broadcast([tensor], opts)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank2]: Original exception:
[rank2]: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 600000ms for send operation to complete
（如果有帮助，我可以提供完整的日志。但我在那里没有找到任何信息。

这是 trl env 输出：
有没有人遇到过类似的问题或对如何解决这个问题有任何建议？任何帮助将不胜感激！

Does TRL currently support multi-modal GRPO training?

[tchang1997]

I don't think that vLLM 0.7.3 is supported — there is a known issue with hangs (e.g., see here). I use vLLM 0.7.2 and additionally set NCCL_P2P_DISABLE=1 (and no other NCCL env variables) and have had no further issues.

[edwardzjl]

I don't think that vLLM 0.7.3 is supported — there is a known issue with hangs (e.g., see here). I use vLLM 0.7.2 and additionally set NCCL_P2P_DISABLE=1 (and no other NCCL env variables) and have had no further issues.

I've downgraded vllm to 0.7.2 and also set NCCL_P2P_DISABLE=1, however still facing the same error.

[yhsong-nxt]

I am experiencing the same problem. Have you solved it?

[edwardzjl]

I am experiencing the same problem. Have you solved it?

Nope

[qgallouedec]

vLLM manipulates environment variables internally and because of that, it's pretty hard to get accelerate to work with vLLM. Especially as the way to modify these variables seems to vary from one patch version to another. This is why we have pinned vLLM to 0.7.2.

Even with this version, in some cases, initialization, or the first step hangs, and I don't know why. This is also why we don't support multiple GPUs for vLLM, as we encounter the same challenges. Help from the community would be greatly appreciated.

[qinqinqaq]

same quetion！

[jiangzizi]

Same problem here. I use 3 A100 for GRPO training and 1 A100 for vllm serving. Below is my error output

  File "/data/jiangdazhi/code/research/llama-trl/archive/full.py", line 82, in main
    trainer.train(resume_from_checkpoint="/data/jiangdazhi/code/research/llama-trl/outputs/full/0402-00/checkpoint-500")
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/transformers/trainer.py", line 2245, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/transformers/trainer.py", line 2556, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/transformers/trainer.py", line 3712, in training_step
    inputs = self._prepare_inputs(inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/trl/extras/profiling.py", line 87, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py", line 653, in _prepare_inputs
    inputs = self._generate_and_score_completions(inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py", line 683, in _generate_and_score_completions
    self._move_model_to_vllm()
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/trl/extras/profiling.py", line 87, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py", line 642, in _move_model_to_vllm
    self.vllm_client.update_named_param(name, param.data)
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/trl/extras/vllm_client.py", line 235, in update_named_param
    self.pynccl_comm.group.barrier()
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/vllm/distributed/utils.py", line 212, in barrier
    self.broadcast_obj(None, src=i)
  File "/data/jiangdazhi/miniconda3/envs/jdz/lib/python3.12/site-packages/vllm/distributed/utils.py", line 190, in broadcast_obj
    recv_obj = pickle.loads(self.store.get(key))
                            ^^^^^^^^^^^^^^^^^^^
torch.distributed.DistStoreError: wait timeout after 300000ms, keys: /broadcast_from/0/4426

[ricardorei]

I am getting the same error using 1 GPU for serving and 3 for training. After 8 batches I run into torch.distributed.DistStoreError: wait timeout after 300000ms

[qgallouedec]

I am getting the same error using 1 GPU for serving and 3 for training. After 8 batches I run into torch.distributed.DistStoreError: wait timeout after 300000ms

The issue was initially opened with system info TRL 0.15. There has been a major refactor in 0.16, so I don't know if you're experiencing the same error.

Can you confirm that you are using 0.16? Can you share the server and train script logs?

[yuantuo666]

I am getting the same error using 1 GPU for serving and 3 for training. After 8 batches I run into torch.distributed.DistStoreError: wait timeout after 300000ms

The issue was initially opened with system info TRL 0.15. There has been a major refactor in 0.16, so I don't know if you're experiencing the same error.

Can you confirm that you are using 0.16? Can you share the server and train script logs?

I was using trl 0.16 with same issue.

Using 1x H100 GPU running vllm and 7x for training GRPO. After running ~3 hours, the exception is raised:
torch.distributed.DistStoreError: wait timeout after 300000ms, keys: /broadcast_from/0/553755

I've tested 3 times, it will always occur after running several hours.

Envs:

$ python
Python 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] on linux
...
>>> torch.__version__
'2.5.1+cu124'
>>> vllm.__version__
'0.7.2'
>>> trl.__version__
'0.16.0'

Logs:

 65%|████████████████████████████████████████████████████████▊                              | 4339/6642 [2:40:27<2:11:03,  3.41s/it][rank0]:[W403 00:25:54.341099763 s
ocket.cpp:462] [c10d] waitForInput: poll for socket SocketImpl(fd=80, addr=[localhost]:44084, remote=[::ffff:0.0.0.0]:51216) returned 0, likely a timeout             
[rank0]:[W403 00:25:54.341388480 socket.cpp:487] [c10d] waitForInput: socket SocketImpl(fd=80, addr=[localhost]:44084, remote=[::ffff:0.0.0.0]:51216) timed out after 
300000ms                                                                                                                                                              
[rank0]: Traceback (most recent call last):                                                                                                                           
[rank0]:   File "/path/***/grpo.py", line 241, in <module>                                                      
[rank0]:     trainer.train(                                                                                                                                           
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 2164, in train                                  
[rank0]:     return inner_training_loop(                                                                                                                              
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 2524, in _inner_training_loop                   
[rank0]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)                                                                                     
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 3648, in training_step                          
[rank0]:     inputs = self._prepare_inputs(inputs)                                                                                                                    
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/trl/extras/profiling.py", line 87, in wrapper                                  
[rank0]:     return func(self, *args, **kwargs)                                                                                                                       
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/trl/trainer/grpo_trainer.py", line 647, in _prepare_inputs                     
[rank0]:     inputs = self._generate_and_score_completions(inputs)                                                                                                    
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/trl/trainer/grpo_trainer.py", line 677, in _generate_and_score_completions     
[rank0]:     self._move_model_to_vllm()                                                                                                                               
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/trl/extras/profiling.py", line 87, in wrapper                                  
[rank0]:     return func(self, *args, **kwargs)                                                                                                                       
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/trl/trainer/grpo_trainer.py", line 636, in _move_model_to_vllm
[rank0]:     self.vllm_client.update_named_param(name, param.data)                                                                  
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/trl/extras/vllm_client.py", line 235, in update_named_param
[rank0]:     self.pynccl_comm.group.barrier()                                                                                                                         
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/vllm/distributed/utils.py", line 193, in barrier
[rank0]:     self.broadcast_obj(None, src=i)                                                                                                                          
[rank0]:   File "/path/***/miniconda3/envs/trl/lib/python3.10/site-packages/vllm/distributed/utils.py", line 171, in broadcast_obj
[rank0]:     recv_obj = pickle.loads(self.store.get(key))                                                                           
[rank0]: torch.distributed.DistStoreError: wait timeout after 300000ms, keys: /broadcast_from/0/553755                              
 65%|██████▌   | 4339/6642 [2:45:28<1:27:49,  2.29s/it]                                                                                                               
[rank0]:[W403 00:25:55.341065896 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal prog
ram exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can e
xit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added si
nce PyTorch 2.4 (function operator())
W0403 00:25:56.537000 3353905 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3354576 closing signal SIGTERM

Server and train bash:

...
CUDA_VISIBLE_DEVICES="0" trl vllm-serve --model <path> &
CUDA_VISIBLE_DEVICES="1,2,3,4,5,6,7" accelerate launch \
    --main_process_port 13443 \
    --mixed_precision bf16 \
    "${work_dir}"/grpo.py \
    --config=$exp_config \
    --exp_name=$exp_name

Part of the train script:

...
print("====== init GRPOConfig ======")
training_args = GRPOConfig(
    output_dir=f"{args.exp_name}-vllm-GRPO",
    logging_steps=10,
    use_vllm=True,
    top_p=0.98,
    top_k=20,
    temperature=1.0,
    # max_completion_length=400, # default 256
    num_generations=2,  # default 8
    per_device_train_batch_size=4,
    num_iterations=1,  # default 1
)

print("====== init GRPOTrainer ======")
trainer = GRPOTrainer(
    model=model, reward_funcs=reward_speech, args=training_args, train_dataset=dataset
)

print("====== GRPOTrainer.train() ======")
trainer.train()