CUDA_VISIBLE_DEVICES ignored by Chat interface in multi-GPU setup (defaults to GPU 0)

[PASiChen]

I am using Unsloth Studio in a multi-GPU environment (RTX 3070 at Index 0 and RTX 5090 at Index 1). While the Studio successfully detects the 5090 upon startup (verified via logs), the Chat interface consistently loads the model into the VRAM of the RTX 3070 (GPU 0) instead of the specified device.

I have tried the following to force the 5090 usage, but the Chat backend still defaults to the 3070:

• Inline Environment Variables: CUDA_VISIBLE_DEVICES=1 unsloth studio -H 0.0.0.0 -p 8888
• Exported Variables: Running export CUDA_VISIBLE_DEVICES=1 prior to launch.
• Backend Specific Flags: Setting export GGML_CUDA_VISIBLE_DEVICES=1.
• .env File: Created a .env file in the root directory with CUDA_VISIBLE_DEVICES=1.

The best result I got is that the startup log correctly says "Hardware detected: RTX 5090", but clicking "Load" in the Chat UI still populates the RTX 3070 VRAM.

I installed with Developer installs for Linux. I'm on Ubuntu 24.04.4 LTS

[rolandtannous]

This is very odd.
I am running this on a B200 with 8 GPUs and it respects the CUDA_VISIBLE_DEVICES setting every single time.

[hackedpassword]

Seeing the same issue.

CUDA_VISIBLE_DEVICES 0 = 3050 6gb, windows primary, display
CUDA_VISIBLE_DEVICES 1 = 5060 8gb, zero use, total resource availability for compute

As an env var, CUDA_VISIBLE_DEVICES=0,1

nm the memory smash in the screenshot, different issue, you'll see the gpu monitor is looking at the wrong GPU. Looking at taskmon, the 5060 gets loaded up as expected with cuda vars set correctly for this system.

I dealt with a similar issue once after I had swapped the PCI-E gpu card slots. Pytorch got confused about this but only deeply within execution, took some doing to isolate. I believe the solution was setting CUDA_DEVICE_ORDER.

If I set these vars using CUDA_VISIBLE_DEVICES=0 or 0,1:
set CUDA_DEVICE_ORDER=FASTEST_FIRST -> Hardware detected: CUDA -- NVIDIA GeForce RTX 5060
set CUDA_DEVICE_ORDER=PCI_BUS_ID -> Hardware detected: CUDA -- NVIDIA GeForce RTX 3050

If I reverse orders of: CUDA_VISIBLE_DEVICES=1,0 and CUDA_DEVICE_ORDER=PCI_BUS_ID, same compute/monitor behavior as above which is consistent. Same also if reducing to CUDA_VISIBLE_DEVICES=1.

..compute shows 5060 then monitor shows 3050. Configured incorrectly, compute goes to the 3050 and monitor remains the 3050. The second screenshot shows 7.51GB which is being put to the 5060 correctly, but will result in the error seen in the first screenshot that is showing 3050 hardware.

imo, studio's hardware monitor is fixated on cuda:0 regardless of cuda env vars. Being that this is my first llm training, unsure if the issue is affecting training.. will know better after finding a smaller model to fit on the 5060.

[hackedpassword]

[danielhanchen]

This should be fixed now - sorry on the delay - please update Unsloth Studio to the latest via curl -fsSL https://unsloth.ai/install.sh | sh for Linux, Mac, WSL or irm https://unsloth.ai/install.ps1 | iex for Windows - if it's not fixed yet, it'll be in the next Studio release - if it's still not working after that, please open another issue - appreciate it again!

The chat interface now pins CUDA_DEVICE_ORDER=PCI_BUS_ID at startup, so it uses the GPU you selected.