PR #6767 staging CI#83
Open
wasimysaid wants to merge 24 commits into
Open
Conversation
for more information, see https://pre-commit.ci
…LoRA, source parity Export page overhaul on top of the formats dropdown: - Unify merged precision into one sorted multi-select list (16-bit first, then 8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16), INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live in a multi-select "More formats" dropdown, so several formats export in one run. - Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig / Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM. FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and _unsloth_save_torchao, parallel to the compressed-tensors path. - Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep 16-bit and portable FP8/INT8. The backend also rejects a compressed request on non-NVIDIA hardware so it stays authoritative. - Relax merged export to non-PEFT models so Local Model and Hugging Face sources get the same 16-bit / compressed / portable options. - GGUF: send the whole quant list in one call (merge once, quantize many). - LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter. - Thread the new fields through models, routes, orchestrator, and worker; extend the export tests.
for more information, see https://pre-commit.ci
…ort save dir naming Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even with PyTorch installed. Add export_capability() in utils/hardware that reports export_supported plus a precise reason so the UI stops showing a generic "no GPU": - pytorch_not_installed: a --no-torch install (even a physical GPU is unusable) - no_accelerator: PyTorch present but no supported accelerator (bare CPU) - mlx_unavailable: Apple Silicon where the MLX stack is missing or too old Expose the fields on /api/system/hardware and /api/system, and guard the mutating export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the reason, leaving read-only endpoints usable so the Export page still renders. Make core/export/export.py import without PyTorch and without a usable accelerator (the Unsloth import is caught) so the export worker degrades to a clear message instead of crashing at import. Frontend: keep /export reachable on chat-only hosts and gray out the method and format options with the backend reason (Alert plus disabled MethodPicker) instead of silently redirecting to /chat, so users see why export is unavailable. Also fix the export save directory producing "model/null" for Local Model and Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS, that hardware.export_capability() reports the right decision and reason (pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export backend imports without PyTorch and degrades to a clear message instead of crashing. Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why" path a Mac/Windows user without an accelerator sees; a real accelerator export is validated separately. The job installs only a CPU PyTorch plus the backend import deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
…oken/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
… import) - studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to the torch install so torch's transitive deps still resolve; --index-url alone replaces PyPI with only the CPU wheel index, which does not serve all of them. - export-page handleStart: reject an empty merged selection (mirrors canExport), so clicking the panel's Start button with every precision pill deselected no longer submits mergedSelections: [] and launches an unintended default 16-bit export. - test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py as text (like the other ast/string checks) instead of `import unsloth.save`, which raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth installed.
… push torchao (_unsloth_save_torchao): - merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted - narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted - forward trust_remote_code (from auto_map) to the reload so custom-code models export Export UI: - hide portable torchao formats on macOS/MLX (backend rejects quantized export there) - restrict a Hub merged export to a single format (each writes to the repo root)
…F timeout torchao (_unsloth_save_torchao): - honor auto_map in the staged tokenizer/processor configs (not just model.config) when deriving trust_remote_code, so custom-code tokenizers reload after the merge - offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy Export orchestrator: - scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a large model does not time out at a flat 3600s
for more information, see https://pre-commit.ci
… hosts Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path. On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and continue hiding them on macOS/MLX.
- Multi-format merged export now collects every sibling output directory (one per selected precision) instead of only the last; the success banner lists them all. - Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations), so the panel says what is being exported rather than just 'Merged Model'. - Persist the selected formats in the run summary and seed them on mount, so navigating away and back (or toggling the export method) restores the selection instead of resetting to 16-bit.
… Q8_0, auto-select newest checkpoint - Progress/summary panel now shows a Formats row with the selected merged formats, and the success banner lists every output folder a multi-format merged run creates (one line per format) instead of only the last one. - Merged format selection is seeded from the active run, so navigating away and back (or switching method cards) no longer resets it to 16-bit. - GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA adapter) for adapter checkpoints, reusing the LoRA GGUF export path. - Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI, the request model, and the backend defaults; the outtype list is now Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers. - When a finetune has no checkpoint selected, auto-select the newest one.
Two fixes to the portable torchao FP8/INT8 export reload, from review of the narrowed VLM detection: - Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs. With the narrowed is_vlm test they now correctly skip the image-text class, but fell through to AutoModelForCausalLM and failed to reload after the merge. Reload them with their own architecture class from the config instead. - AutoModelForImageTextToText was imported unconditionally at the top of the torchao path, so on Transformers builds without that class the import aborted every torchao export (even text-only). Import it lazily only for a VLM, with the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS. Run the quantization against a dedicated llm-compressor-main "shadow": a --target package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered over the existing torch. It installs --no-deps so torch is never touched (works on any Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN. - transformers_version.py: provision + validate .venv_llmcompressor. - export.py: route all compressed exports through the shadow when available; else keep the workspace 0.10.x path and fail fast past its transformers ceiling. - save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow. - _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the RedHatAI and NVIDIA reference quants, and is required by the grouped schemes). Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and fp8 on Gemma-4, end to end through Studio.
…-dropdown # Conflicts: # studio/backend/main.py # unsloth/save.py
for more information, see https://pre-commit.ci
…nslothai/unsloth into pr-6767-ci-202607031642
|
Warning Review limit reached
Next review available in: 38 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (23)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
…nslothai/unsloth into pr-6767-ci-202607031642
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staging-only CI validation for unslothai#6767.
This PR exists only to run fork GitHub Actions checks and should be closed after validation.