[model] Support Kimi_VL thinking/instruct

[Kuangdd01]

What does this PR do?

add kimi_vl
Fixes #7680 (issue)

Have tested

1. Single GPU LoraFinetuning ~50GB VRAM (✔️)
2. Full Finetuning with zero2 @chenllliang (✔️)
3. Multiple GPUs inference in webui w/o thinking (✔️)
4. Multiple GPUs Lora Finetuning with zero2 (✔️)
5. Failed in Multiple GPUs Lora Finetuning without zero2 (❌) not recommended

Exp configs

Lora

### model
model_name_or_path: moonshotai/Kimi-VL-A3B-Instruct
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj # ref deepseek ft
freeze_vision_tower: true
freeze_multi_modal_projector: true
freeze_language_model: false
flash_attn: auto

### dataset
dataset: identity, mllm_demo
template: kimi_vl
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 1
dataloader_num_workers: 4

### output
output_dir: saves/kimi-vl/lora/
logging_steps: 1
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 4
gradient_accumulation_steps: 2
learning_rate: 1.0e-4
num_train_epochs: 10.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

# for ddp
# ddp_find_unused_parameters: true

full

### model
model_name_or_path: moonshotai/Kimi-VL-A3B-Instruct
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: full
freeze_vision_tower: true
freeze_multi_modal_projector: true
freeze_language_model: false
deepspeed: examples/deepspeed/ds_z2_config.json # 8xh20 gpu

### dataset
dataset: identity, mllm_demo, alpaca_en_demo
template: kimi_vl
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 1
dataloader_num_workers: 4

### output
output_dir: saves/kimi-vl/full
logging_steps: 1
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 2
learning_rate: 1.0e-4
num_train_epochs: 10.0
lr_scheduler_type: cosine
warmup_ratio: 0
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

Support

Thanks to Kimi_VL's team's help. @chenllliang @zhouzaida

TODO

• update modeling_kimi_vl.py for training @zhouzaida

Before submitting

• Did you read the contributor guideline?
• Did you write any new necessary tests?

[chenllliang]

script for full SFT with zero2

### model
model_name_or_path: moonshotai/Kimi-VL-A3B-Instruct
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: full
freeze_vision_tower: true
freeze_multi_modal_projector: true
freeze_language_model: false
deepspeed: examples/deepspeed/ds_z2_config.json # 8xh20 gpu

### dataset
dataset: identity, mllm_demo, alpaca_en_demo
template: kimi_vl
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 1
dataloader_num_workers: 4

### output
output_dir: saves/kimi-vl/full
logging_steps: 1
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 2
learning_rate: 1.0e-4
num_train_epochs: 10.0
lr_scheduler_type: cosine
warmup_ratio: 0
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

[hiyouga]

LGTM

[Himanshunitrr]

@chenllliang will the script for full SFT with zero2 work for Thinking model too?

[Gaoyg]

Have a quesiton: why change the topk_method to greedy during training?

[Kuangdd01]

Have a quesiton: why change the topk_method to greedy during training?

We can not use the origin top_k method in MOEGate, because it only supports the inference stage.
ref: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L437-L461
So we refer to DeepSeekVL2 and add a greedy method to the modeling and configurations.
ref: https://github.com/deepseek-ai/DeepSeek-VL2/blob/ef9f91e2b6426536b83294c11742c27be66361b1/deepseek_vl2/models/modeling_deepseek.py#L441-L466

[Kuangdd01]

Himanshunitrr

Sure, but the training data for the thinking version should be well-organized with thinking format.

[Gaoyg]

Have a quesiton: why change the topk_method to greedy during training?

We can not use the origin top_k method in MOEGate, because it only supports the inference stage. ref: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L437-L461 So we refer to DeepSeekVL2 and add a greedy method to the modeling and configurations. ref: https://github.com/deepseek-ai/DeepSeek-VL2/blob/ef9f91e2b6426536b83294c11742c27be66361b1/deepseek_vl2/models/modeling_deepseek.py#L441-L466

Get, thanks