[Bug] AttributeError: 'LoraModel' object has no attribute 'vllm_engine'

[Aaronhuang-778]

Did you update? pip install --upgrade unsloth unsloth_zoo: Yes
Colab or Kaggle or local / cloud: Kaggle
Number GPUs used, use nvidia-smi:1
Which notebook? Please link!: link
Which Unsloth version, TRL version, transformers version, PyTorch version?:
Torch = 2.6.0+cu124
Unsloth = 2025.8.5
Transformers = 4.53.2
TRL = 0.21.0

🦥 You can also ask via our Reddit page: https://www.reddit.com/r/unsloth/

Hi, when I set load_in_8bit, i will get this error

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. INFO 08-14 10:06:22 [__init__.py:244] Automatically detected platform cuda. 🦥 Unsloth Zoo will now patch everything to make training faster! Unsloth: We now expect per_device_train_batch_sizeto be a multiple ofnum_generations. We will change the batch size of 1 to the num_generationsof 8 ==((====))== Unsloth 2025.8.5: Fast Qwen2 patching. Transformers: 4.52.4. vLLM: 0.9.2. \\ /| NVIDIA A100-SXM4-80GB. Num GPUs = 1. Max memory: 79.325 GB. Platform: Linux. O^O/ \_/ \ Torch: 2.7.1+cu126. CUDA: 8.0. CUDA Toolkit: 12.6. Triton: 3.3.1 \ / Bfloat16 = TRUE. FA [Xformers = 0.0.31.post1. FA2 = True] "-____-" Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.21s/it] Unsloth: Makingmodel.base_model.model.model` require gradients
[2025-08-14 10:06:57,666] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/peft/peft_model.py", line 856, in getattr
return super().getattr(name) # defer to nn.Module's logic
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1940, in getattr
raise AttributeError(
AttributeError: 'PeftModel' object has no attribute 'vllm_engine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 370, in getattr
return super().getattr(name) # defer to nn.Module's logic
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1940, in getattr
raise AttributeError(
AttributeError: 'LoraModel' object has no attribute 'vllm_engine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/lustre/fs12/portfolios/nvr/projects/nvr_lpr_nvgptvision/users/weihua/Q-RL/open-r1/src/open_r1/grpo_qlora_8.py", line 164, in
trainer = GRPOTrainer(
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/unsloth/trainer.py", line 209, in new_init
original_init(self, *args, **kwargs)
File "/lustre/fs12/portfolios/nvr/projects/nvr_lpr_nvgptvision/users/weihua/Q-RL/open-r1/src/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 2892, in init
super().init(
File "/lustre/fs12/portfolios/nvr/projects/nvr_lpr_nvgptvision/users/weihua/Q-RL/open-r1/src/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 1434, in init
self.llm = model.vllm_engine
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/peft/peft_model.py", line 860, in getattr
return getattr(self.base_model, name)
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 374, in getattr
return getattr(self.model, name)
File "/lustre/fsw/portfolios/nvr/users/weihua/miniconda3/envs/open-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1940, in getattr
raise AttributeError(
AttributeError: 'Qwen2ForCausalLM' object has no attribute 'vllm_engine'`

[Datta0]

Hey @Aaronhuang-778 , I'd suggest using vLLM for inference by setting fast_inference=True in FastLanguageModel.from_pretrained()

If you do not wish to use vllm, can you show me what your GRPOConfig is? Make sure that use_vllm is False there

[Aaronhuang-778]

Hey @Aaronhuang-778 , I'd suggest using vLLM for inference by setting fast_inference=True in FastLanguageModel.from_pretrained()

If you do not wish to use vllm, can you show me what your GRPOConfig is? Make sure that use_vllm is False there

Hi, this is my config: I set fast_inference=True in previous script

model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "Qwen2.5-7B-Instruct",
max_seq_length = max_seq_length,
load_in_4bit = False, # False for LoRA 16bit
load_in_8bit = True, # True for LoRA 8bit
fast_inference = True, # Enable vLLM fast inference
max_lora_rank = lora_rank,
gpu_memory_utilization = 0.5, # Reduce if out of memory
)

training_args = GRPOConfig( use_vllm = True, # use vLLM for fast inference! learning_rate = 5e-6, adam_beta1 = 0.9, adam_beta2 = 0.99, weight_decay = 0.1, warmup_ratio = 0.1, lr_scheduler_type = "cosine", logging_steps = 1, per_device_train_batch_size = 1, gradient_accumulation_steps = 4, # Increase to 4 for smoother training num_generations = 8, # Decrease if out of memory max_prompt_length = 256, max_completion_length = 512, num_train_epochs = 1, # Set to 1 for a full training run max_steps = 250, save_steps = 100, max_grad_norm = 0.1, report_to = "wandb", # Can use Weights & Biases output_dir = "model/Qwen2.5-7B-Instruct_grpo_8bit_lora_gsm8k", )

[Aaronhuang-778]

load_in_4bit = False, # False for LoRA 16bit

But it would be ok if I only set load_in_4bit = True

[Aaronhuang-778]

Hi @Datta0, Please check the new support in vision.py

Here, only pass kwargs["quantization_config"] = bnb_config when it is 4bit, but fail for 8bit

 if load_in_4bit:
            # Ignore load_in_4bit / load_in_8bit for MXFP4 - best to get config file
            if "gpt-oss" in model_name.lower():
                pass
            else:
                kwargs["quantization_config"] = bnb_config
        pass