Fails to load (OOM) models that I can load fine in LMStudio - probably related to worse (or potentially nonexistent?) automatic handling of how many GPU layers to offload

[slightlyoutofphase]

TLDR I straight up cannot load any of the models I can load in LM Studio, with this, and there appears to simply be no way whatsoever of even altering the count of GPU layers to offload (or any other load-time setting at all).

[Imagineer99]

This seems valid. For GGUF specifically, it may make sense to expose GPU offloading in Configuration so we can support partial layer offload, not just the current automatic behavior. That could explain why some models load in LM Studio but OOM here.

I’m not the owner of this backend path, so I can’t commit to implementation/fix, but it seems like a reasonable direction,
thoughts @rolandtannous ?

[Datta0]

Hey @slightlyoutofphase the studio is meant to be easy for people to use. So we do not expose every knob to the user. But if you do have multiple GPUs, we directly shard the model between them. Unfortunately, there is no way to control it via the UI/API.

[Kacper-Kondracki]

Working is not the enemy of easy, if it is not working, it is easily unusable. I can't load gemma 4 26B on my RTX 4080 in unsloth studio. In LM Studio it was working reasonably well with offloading.

[Datta0]

We're definitely improving things. This is not end state.
But if you feel like something is missing, the project is open source and we do welcome contributions :)