Feat: Clip Higher

[SabaPivot]

No description provided.

[SabaPivot]

1. set beta = 0 disabling kv_divergence
2. add Clip Higher from DAPO

[SZhanZ]

Hello，could you explain the update parameters?

[SabaPivot]

1. set beta = 0 disabling kv_divergene.

• Setting beta = 0 effectively removes this KL divergence penalty (as I see in your blog posts).
• So, I have enabled to set beta = 0.0 in .sh file, and when beta is set to 0.0, no reference model is initialized (which significantly reduces GPU usage during the GRPO training).

2. Add Clip Higher from DAPO:

• This change directly implements one of the core techniques introduced in the DAPO paper, specifically designed to promote exploration and prevent "entropy collapse" (where the model becomes too conservative and produces non-diverse outputs).

• Standard policy gradient methods often "clip" the objective function to limit how large the policy updates can be.

• DAPO modifies this by using a higher upper clipping threshold (ε_high).

The "Clip Higher" technique allows the probability ratio (new policy / old policy) to increase more for advantageous actions before being clipped(just like kl_beta=0.0).

[SabaPivot]

Hello，could you explain the update parameters?

Hope my reply helps, during my experiment, these improvements had contributed to achieve higher performance.

[SZhanZ]

1. set beta = 0 disabling kv_divergene.

• Setting beta = 0 effectively removes this KL divergence penalty (as I see in your blog posts).
• So, I have enabled to set beta = 0.0 in .sh file, and when beta is set to 0.0, no reference model is initialized (which significantly reduces GPU usage during the GRPO training).

2. Add Clip Higher from DAPO:

• This change directly implements one of the core techniques introduced in the DAPO paper, specifically designed to promote exploration and prevent "entropy collapse" (where the model becomes too conservative and produces non-diverse outputs).
• Standard policy gradient methods often "clip" the objective function to limit how large the policy updates can be.
• DAPO modifies this by using a higher upper clipping threshold (ε_high).

The "Clip Higher" technique allows the probability ratio (new policy / old policy) to increase more for advantageous actions before being clipped(just like kl_beta=0.0).

Thanks for your reply. We will merge it. Could you please further provide an example script in src/open-r1-multimodal/run_script folder?

[SabaPivot]

1. set beta = 0 disabling kv_divergene.

• Setting beta = 0 effectively removes this KL divergence penalty (as I see in your blog posts).
• So, I have enabled to set beta = 0.0 in .sh file, and when beta is set to 0.0, no reference model is initialized (which significantly reduces GPU usage during the GRPO training).

2. Add Clip Higher from DAPO:

• This change directly implements one of the core techniques introduced in the DAPO paper, specifically designed to promote exploration and prevent "entropy collapse" (where the model becomes too conservative and produces non-diverse outputs).
• Standard policy gradient methods often "clip" the objective function to limit how large the policy updates can be.
• DAPO modifies this by using a higher upper clipping threshold (ε_high).

The "Clip Higher" technique allows the probability ratio (new policy / old policy) to increase more for advantageous actions before being clipped(just like kl_beta=0.0).

Thanks for your reply. We will merge it. Could you please further provide an example script in src/open-r1-multimodal/run_script folder?

Sure. My bad.

Just added two lines below the .sh files.

src/open-r1-multimodal/run_script/run_grpo_rec.sh

cd /workspace/VLM-R1-DAPO/src/open-r1-multimodal

export DEBUG_MODE="true"
export CUDA_VISIBLE_DEVICES=1,2

RUN_NAME="Qwen2.5-VL-3B-GRPO-REC-SFT"
export LOG_PATH="./debug_log_$RUN_NAME.txt"

torchrun --nproc_per_node="2" \
    --nnodes="1" \
    --node_rank="0" \
    (..your code..)
    --beta 0.0 \ # disable reference model
    --epsilon_high 0.28 # Clip higher, epsilon_high=0.28 as recommended in the paper DAPO

[SabaPivot]

I have also updated a new commit.

Thank you.

[SZhanZ]

I have also updated a new commit.

Thank you.

okay~ thanks for PR

[Shengqi77]

Hello @SabaPivot How do I implement DAPO's dynamic sampling in this code?