Studio: multi-select export formats, portable FP8/INT8, GGUF LoRA, and source parity#6767
Conversation
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for a broader range of vLLM compressed-tensors schemes by adding a compressed_method parameter to the model merging and export pipeline. This allows the frontend to expose a 'More formats' dropdown (including FP8, INT, MXFP, and NVFP4) that overrides the default format type. The backend has been updated to resolve these schemes dynamically against the Unsloth registry. A critical issue was identified in the backend export logic where the unresolved compressed_alias is assigned to save_method instead of the resolved base save method, which would cause the export to fail.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| elif is_compressed: | ||
| save_method = compressed_alias |
There was a problem hiding this comment.
The save_method should be set to the resolved base save method (info[0]) from the registry lookup rather than the unresolved compressed_alias (which could be an alias like 'fp8_static' or 'w4a16'). Passing the unresolved alias directly to save_pretrained_merged will cause the export to fail as it is not a recognized save method in Unsloth.
| elif is_compressed: | |
| save_method = compressed_alias | |
| elif is_compressed: | |
| save_method = info[0] |
…LoRA, source parity Export page overhaul on top of the formats dropdown: - Unify merged precision into one sorted multi-select list (16-bit first, then 8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16), INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live in a multi-select "More formats" dropdown, so several formats export in one run. - Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig / Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM. FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and _unsloth_save_torchao, parallel to the compressed-tensors path. - Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep 16-bit and portable FP8/INT8. The backend also rejects a compressed request on non-NVIDIA hardware so it stays authoritative. - Relax merged export to non-PEFT models so Local Model and Hugging Face sources get the same 16-bit / compressed / portable options. - GGUF: send the whole quant list in one call (merge once, quantize many). - LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter. - Thread the new fields through models, routes, orchestrator, and worker; extend the export tests.
for more information, see https://pre-commit.ci
danielhanchen
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b3e52f025
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if not push_to_hub and os.path.isdir(local_dir): | ||
| try: | ||
| shutil.rmtree(local_dir) |
There was a problem hiding this comment.
For local torchao exports, local_dir is the caller's save_directory, not an isolated temp dir. This cleanup therefore recursively deletes the user-selected output directory after quantization; if the user picked an existing directory, or selected 16-bit plus a portable torchao format in the new multi-format UI, it removes the 16-bit export and any pre-existing files. Use a temporary staging directory or only delete a directory that this path created solely for staging.
Useful? React with 👍 / 👎.
| quantized_model = auto_model.from_pretrained( | ||
| local_dir, | ||
| device_map = "auto", | ||
| quantization_config = TorchAoConfig(quant_type = quant_type), | ||
| **dtype_kw, |
There was a problem hiding this comment.
When exporting a model whose config has auto_map/custom code, the UI can approve and load it with trust_remote_code=True, but the torchao path merges to local_dir and then reloads the staged checkpoint here without forwarding that trust. AutoModel...from_pretrained will reject those local staged checkpoints after the expensive merge, so portable FP8/INT8 exports fail for remote-code models; mirror the compressed path's trust handling for the model and processor reloads.
Useful? React with 👍 / 👎.
| const availableFormats = useMemo<MergedFormatOption[]>( | ||
| () => MERGED_FORMATS.filter((f) => hasNvidia || !f.needsNvidia), | ||
| [hasNvidia], |
There was a problem hiding this comment.
On macOS/MLX, hardware.cuda and hardware.rocm are both null, so this filter keeps every needsNvidia:false option, including torchao_fp8 and torchao_int8. The backend rejects is_torchao when _IS_MLX, so the page advertises portable FP8/INT8 formats that always fail on Apple Silicon; either detect MLX and hide them or allow the backend path.
Useful? React with 👍 / 👎.
| compressed_method: sel.compressedMethod, | ||
| push_to_hub: pushToHub, | ||
| repo_id: params.repoId, | ||
| hf_token: params.token, |
There was a problem hiding this comment.
When destination is Hub and a user selects more than one merged format, this loop sends each export to the same repo_id. All formats save root-level files like config.json and model shard names, so later uploads overwrite or mix with earlier ones and the Hub repo cannot contain all selected formats; use per-format repo IDs/subfolders or disable multi-select for Hub.
Useful? React with 👍 / 👎.
| : !isAdapter | ||
| ? ["lora"] |
There was a problem hiding this comment.
For sourceMode === 'checkpoint' and a full fine-tune (!isAdapter), this now leaves Merged enabled, but the checkpoint load path still omits load_in_4bit:false and therefore uses the backend default true. The non-PEFT merged save then serializes the 4-bit-loaded model rather than a true 16-bit export, so full-checkpoint merged exports can be quantized or wrong; load non-adapter merged checkpoints in full precision like local/HF sources.
Useful? React with 👍 / 👎.
| is_vlm = any( | ||
| x.endswith(("ForConditionalGeneration", "ForVisionText2Text")) for x in archs | ||
| ) |
There was a problem hiding this comment.
For portable torchao exports of text/audio encoder-decoder models such as T5, BART, or Whisper, their architecture names end in ForConditionalGeneration, so this marks them as VLMs and reloads the staged checkpoint with AutoModelForImageTextToText/AutoProcessor. Those models are not image-text models and the reload fails after the merge; match the narrower VLM test used by the compressed path (vision_config or explicit vision architecture) instead.
Useful? React with 👍 / 👎.
| import unsloth.save as _us | ||
| return hasattr(_us, "_normalize_torchao_method") |
There was a problem hiding this comment.
In environments where Studio intentionally has no real torchao package, such as Windows ROCm where the installer skips torchao and installs a runtime stub, unsloth.save still defines _normalize_torchao_method, so this reports portable export support and bypasses the NVIDIA compressed-tensors guard. The export then fails later when _unsloth_save_torchao imports or calls stubbed torchao classes; check for a real torchao.quantization implementation (or explicitly block the skipped/stubbed platforms) before classifying aliases as torchao.
Useful? React with 👍 / 👎.
…ort save dir naming Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even with PyTorch installed. Add export_capability() in utils/hardware that reports export_supported plus a precise reason so the UI stops showing a generic "no GPU": - pytorch_not_installed: a --no-torch install (even a physical GPU is unusable) - no_accelerator: PyTorch present but no supported accelerator (bare CPU) - mlx_unavailable: Apple Silicon where the MLX stack is missing or too old Expose the fields on /api/system/hardware and /api/system, and guard the mutating export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the reason, leaving read-only endpoints usable so the Export page still renders. Make core/export/export.py import without PyTorch and without a usable accelerator (the Unsloth import is caught) so the export worker degrades to a clear message instead of crashing at import. Frontend: keep /export reachable on chat-only hosts and gray out the method and format options with the backend reason (Alert plus disabled MethodPicker) instead of silently redirecting to /chat, so users see why export is unavailable. Also fix the export save directory producing "model/null" for Local Model and Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS, that hardware.export_capability() reports the right decision and reason (pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export backend imports without PyTorch and degrades to a clear message instead of crashing. Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why" path a Mac/Windows user without an accelerator sees; a real accelerator export is validated separately. The job installs only a CPU PyTorch plus the backend import deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ed2989636
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| <Button | ||
| type="button" | ||
| variant={loraAsGguf ? "default" : "outline"} | ||
| size="sm" | ||
| onClick={() => setLoraAsGguf(true)} | ||
| title="llama.cpp GGUF LoRA, loadable with `llama-cli --lora`." |
There was a problem hiding this comment.
On Apple Silicon/MLX, export_lora_adapter explicitly returns an error when gguf is true (studio/backend/core/export/export.py checks _IS_MLX before conversion), but this new option still lets users set loraAsGguf whenever export is otherwise supported. As a result, MLX users can choose the GGUF adapter path, wait through the load, and then always fail; gate this button on the same MLX condition or add backend support.
Useful? React with 👍 / 👎.
| : !isAdapter | ||
| ? ["lora"] | ||
| : [] |
There was a problem hiding this comment.
When the user switches to a Local/Hugging Face source after selecting a LoRA fine-tuned run, isAdapter still reflects the old selectedModelData, not the direct model being exported. This leaves LoRA enabled for an unrelated direct base model, but runExport loads that direct source with isAdapter: false and the backend then rejects it with “No adapter to export”; include sourceMode/the active source in this gate instead of relying on stale fine-tuned-run metadata.
Useful? React with 👍 / 👎.
| self.current_model.save_pretrained_gguf( | ||
| save_directory, | ||
| self.current_tokenizer, | ||
| save_method = "lora", | ||
| quantization_method = outtype, | ||
| ) |
There was a problem hiding this comment.
For GGUF LoRA exports whose adapter/base is gated or private, the load can succeed with the Studio-provided hf_token, but this new conversion call drops that token. The LoRA GGUF helper uses the token argument to set HF_TOKEN for convert_lora_to_gguf.py when it fetches the base config, so these exports fail even though the user supplied a valid token; pass token = hf_token here.
Useful? React with 👍 / 👎.
| : !isAdapter && isQuantized | ||
| ? ["merged", "lora", "gguf"] |
There was a problem hiding this comment.
If a user first selects a quantized full fine-tune run and then switches to a Local/Hugging Face source, isQuantized still comes from the old selectedModelData. This branch therefore disables every export method for an unrelated direct source, even when that source is exportable; only apply the quantized-checkpoint restriction while sourceMode === "checkpoint" (or derive it from the active source).
Useful? React with 👍 / 👎.
| f"Choose one of {', '.join(_GGUF_LORA_OUTTYPES)}.", | ||
| None, | ||
| ) | ||
| if not _supports_kwarg(self.current_model.save_pretrained_gguf, "save_method"): |
There was a problem hiding this comment.
On an older Unsloth model that lacks save_pretrained_gguf, this attribute access raises before _supports_kwarg can return False and before the method's try block starts. That turns the intended clean “does not support GGUF LoRA adapter export” response into a generic 500 from the route; use getattr or catch AttributeError in this guard.
Useful? React with 👍 / 👎.
…oken/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
|
Thanks, all five are real. Fixed in 5df1ad5.
tsc clean, eslint adds no new errors (8 -> 8 baseline), 50 backend export tests pass. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5df1ad54d0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - name: Install CPU PyTorch | ||
| # Pin to the CPU wheel index so every OS gets a CPU build (Windows | ||
| # would otherwise pull a CUDA wheel from PyPI; macOS has no CUDA wheel). | ||
| run: python -m pip install --index-url https://download.pytorch.org/whl/cpu "torch>=2.4,<2.13" |
There was a problem hiding this comment.
On fresh runners this replaces the default PyPI index with only the PyTorch CPU index. I checked python -m pip install --help: --index-url is the base package index, while --extra-index-url is for indexes used in addition, so torch's transitive dependencies are also resolved only from download.pytorch.org here. Other workflows in this repo add --extra-index-url https://pypi.org/simple; without that, this new capability workflow can fail during the torch install step before any tests run.
Useful? React with 👍 / 👎.
| quantLevels, | ||
| useImatrix: effectiveImatrix, | ||
| mergedFormat, | ||
| mergedSelections: selectedFormats.map((v) => mergedFormatPayload(v)), |
There was a problem hiding this comment.
If the panel is already open, a user can deselect every precision pill; canExport only disables the outer Export Model button, but the panel's Start Export button still calls handleStart. This sends mergedSelections: [], and export-runtime-store.ts falls back to a default 16-bit export for an empty list, so a user who selected no formats still launches an expensive 16-bit export; mirror the merged selectedFormats.length guard in handleStart or prevent unchecking the last format.
Useful? React with 👍 / 👎.
|
|
||
| def test_unsloth_save_has_torchao_registry_and_path(): | ||
| # The core normalizer + registry + save path must exist for the backend to route to them. | ||
| import unsloth.save as us |
There was a problem hiding this comment.
The regular studio-backend-ci.yml backend job runs python -m pytest tests/ from studio/backend without installing the repo package or adding the repo root to PYTHONPATH; I also checked that import unsloth.save from that working directory raises ModuleNotFoundError. This new test is therefore picked up by the CPU backend suite and fails during the torchao-registry check, even though the surrounding file otherwise uses AST/string checks to avoid the heavy Unsloth import path.
Useful? React with 👍 / 👎.
… import) - studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to the torch install so torch's transitive deps still resolve; --index-url alone replaces PyPI with only the CPU wheel index, which does not serve all of them. - export-page handleStart: reject an empty merged selection (mirrors canExport), so clicking the panel's Start button with every precision pill deselected no longer submits mergedSelections: [] and launches an unintended default 16-bit export. - test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py as text (like the other ast/string checks) instead of `import unsloth.save`, which raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth installed.
|
Thanks, all three fixed in 45f3df8.
tsc clean, eslint adds no new errors (8 -> 8 baseline). |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45f3df8af7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not push_to_hub and os.path.isdir(local_dir): | ||
| try: | ||
| shutil.rmtree(local_dir) |
There was a problem hiding this comment.
When a local torchao export runs, local_dir is the caller's save_directory, so this rmtree deletes the entire user-selected folder after producing the -torchao-* sibling. That wipes any pre-existing files in a custom export directory, and it also removes the 16-bit artifact when the new multi-select flow exports 16-bit before a portable FP8/INT8 format using the same base path. Use a temporary staging directory (or only remove files created for staging) instead of recursively deleting the requested output path.
Useful? React with 👍 / 👎.
| push_to_hub: pushToHub, | ||
| repo_id: params.repoId, |
There was a problem hiding this comment.
When Destination is Hub and more than one merged precision is selected, this loop submits every format to the same repo_id. The Hub upload paths write model files at the repository root, so a 16-bit upload followed by FP8/INT8 can overwrite config.json/weights and leave stale shards from the previous format, producing a repo that is inconsistent or unloadable. Use a per-format repo/path or restrict Hub merged exports to one selected format.
Useful? React with 👍 / 👎.
| """True if the installed unsloth build has the portable torchao FP8/INT8 export path.""" | ||
| try: | ||
| import unsloth.save as _us | ||
| return hasattr(_us, "_normalize_torchao_method") |
There was a problem hiding this comment.
This helper returns true as soon as unsloth.save contains the torchao registry, but the actual torchao package can still be unavailable or stubbed; for example, studio/install_python_stack.py skips torchao on Windows ROCm and the export worker installs the stub from core/_torchao_stub.py. In that environment the UI/backend accept torchao_fp8/torchao_int8, then _unsloth_save_torchao fails later when it tries to import/use the quantization classes. Verify the real torchao classes are importable (or disable these aliases on the stubbed platform) before classifying the alias as supported.
Useful? React with 👍 / 👎.
| except Exception as _unsloth_exc: # ImportError (e.g. missing torch) or a broken native load | ||
| FastLanguageModel = None | ||
| FastVisionModel = None | ||
| _IS_MLX = False |
There was a problem hiding this comment.
Because this broad catch also nulls FastLanguageModel/FastVisionModel for a broken Unsloth native load, a GPU host still passes the new export_capability() route guard, then load_checkpoint() later dereferences these None model classes and reports a cryptic load failure. Check _export_runtime_available() at the start of load_checkpoint() (or surface _UNSLOTH_IMPORT_ERROR) so a broken Unsloth install fails with the actual import problem instead of a NoneType error.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 299f40c36b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| hf_token: params.token, | ||
| private: params.privateRepo, | ||
| gguf: params.loraGguf ?? false, |
There was a problem hiding this comment.
When loraGguf is true for a local export of a gated/private adapter or base, this request still sends hf_token: params.token, but handleStart only populates params.token for Hub destinations while the load uses loadToken. Fresh evidence after the earlier backend fix: the backend now forwards hf_token into save_pretrained_gguf(... token=...), but this new GGUF LoRA call still supplies the hub-only token, so local GGUF LoRA conversion runs without the user’s token and fails when it fetches the gated base config; pass params.loadToken here when not pushing to Hub.
Useful? React with 👍 / 👎.
|
Agreed, valid P2. Fixed in d9409ba.
|
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9409ba63c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| hardware.exportUnsupportedMessage ?? | ||
| "Export requires a supported accelerator (NVIDIA, AMD, or Intel GPU, or Apple Silicon) with PyTorch or MLX installed."; | ||
| const availableFormats = useMemo<MergedFormatOption[]>( | ||
| () => MERGED_FORMATS.filter((f) => hasNvidia || !f.needsNvidia), |
There was a problem hiding this comment.
On an Apple Silicon/MLX host, exportUnsupported is false and hasNvidia is false, so this filter keeps every format with needsNvidia === false, including torchao_fp8 and torchao_int8. The backend rejects those same requests in export_merged_model when _IS_MLX is true (Quantized ... export is not supported on macOS/MLX), so the UI advertises portable FP8/INT8, lets the user start the export, and then fails after loading. Filter torchao formats out for MLX/macOS here, or make the backend support them.
Useful? React with 👍 / 👎.
| if not push_to_hub and os.path.isdir(local_dir): | ||
| try: | ||
| shutil.rmtree(local_dir) |
There was a problem hiding this comment.
When the user leaves the default 16-bit format selected and adds a portable FP8/INT8 format, the runtime writes the 16-bit export to save_directory first and then runs torchao with that same save_directory as its staging directory. This cleanup deletes local_dir after quantization, so the previously selected 16-bit artifact is removed even though multi-select promises each selected format as its own output. Use an isolated staging directory for torchao or avoid deleting save_directory when it may be a selected output.
Useful? React with 👍 / 👎.
| quantized_model = auto_model.from_pretrained( | ||
| local_dir, | ||
| device_map = "auto", | ||
| quantization_config = TorchAoConfig(quant_type = quant_type), | ||
| **dtype_kw, |
There was a problem hiding this comment.
For an HF/local source that required approved custom code to load, the initial load can succeed with trust_remote_code, but the portable torchao path reloads the staged checkpoint here without passing that flag. Checkpoints with auto_map are rejected by Transformers unless trust_remote_code=True, so torchao FP8/INT8 exports fail after the merge for exactly the custom-code models the UI already approved; derive the flag from model.config.auto_map as the compressed path does and pass it to this reload.
Useful? React with 👍 / 👎.
| repo_id: params.repoId, | ||
| hf_token: params.token, |
There was a problem hiding this comment.
When destination === "hub" and the user selects multiple merged formats, this loop calls the backend once per format but reuses the same repo_id every time. Each call uploads a full model/config to the repository root, so later formats overwrite or mix with earlier ones (for example 16-bit plus torchao INT8 leaves one repo with incompatible root-level artifacts) instead of producing separate outputs. Either restrict Hub export to one merged format or derive a per-format repo/path.
Useful? React with 👍 / 👎.
| is_vlm = any( | ||
| x.endswith(("ForConditionalGeneration", "ForVisionText2Text")) for x in archs | ||
| ) |
There was a problem hiding this comment.
For portable torchao exports, any architecture ending in ForConditionalGeneration is treated as a vision model and reloaded with AutoModelForImageTextToText. Text seq2seq/audio models such as T5/BART/Whisper also use *ForConditionalGeneration, so they will be sent through the image-text auto class and fail during FP8/INT8 export; the compressed path below already avoids this by checking vision_config or ForVisionText2Text only. Match that narrower VLM detection here.
Useful? React with 👍 / 👎.
… push torchao (_unsloth_save_torchao): - merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted - narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted - forward trust_remote_code (from auto_map) to the reload so custom-code models export Export UI: - hide portable torchao formats on macOS/MLX (backend rejects quantized export there) - restrict a Hub merged export to a single format (each writes to the repo root)
|
All five valid - fixed in fce8c9a. Backend (
Frontend (
Verified: |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fce8c9a9ba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| is_vlm = hasattr(model.config, "vision_config") or any( | ||
| x.endswith("ForVisionText2Text") for x in archs | ||
| ) | ||
| trust_remote_code = bool(getattr(model.config, "auto_map", None)) |
There was a problem hiding this comment.
For portable torchao exports where the approved custom code is declared only in the tokenizer/processor config, this derives trust_remote_code from model.config.auto_map and leaves it false. Fresh evidence after the earlier trust fix is that the reload below also calls auto_processor.from_pretrained(..., trust_remote_code=trust_remote_code), while the consent scanner covers tokenizer/processor auto_map too; those exports will fail after the merge even though the user approved the code. Carry the original approval through, or inspect the staged tokenizer/processor configs as well.
Useful? React with 👍 / 👎.
| torch.cuda.is_available() | ||
| and hasattr(model, "parameters") | ||
| and not getattr(model, "is_loaded_in_4bit", False) | ||
| and not getattr(model, "is_loaded_in_8bit", False) | ||
| and not getattr(model, "is_quantized", False) | ||
| ): |
There was a problem hiding this comment.
On Intel/XPU hosts, the portable torchao formats remain available because they do not need NVIDIA, but this memory-release branch only runs for torch.cuda.is_available(). The original XPU model therefore stays resident while the export reloads a second quantized copy with device_map="auto", so models that fit once on XPU can OOM during FP8/INT8 export; include the XPU cache/offload path or move any single-device accelerator model to CPU before reloading.
Useful? React with 👍 / 👎.
| const { outputPath } = await runRecoverableOp(() => | ||
| exportGGUF({ | ||
| save_directory: params.saveDirectory, | ||
| quantization_method: params.quantLevels, |
There was a problem hiding this comment.
When a user selects multiple GGUF quantizations for a large model, this sends the whole list as one export operation. The orchestrator still waits only 3600s per operation (core/export/orchestrator.py sets that timeout for export_gguf_done), so two or more 30B+ quantizations can time out even though the previous loop gave each quant its own hour. Either keep per-quant calls/recovery or scale the timeout for list exports.
Useful? React with 👍 / 👎.
…F timeout torchao (_unsloth_save_torchao): - honor auto_map in the staged tokenizer/processor configs (not just model.config) when deriving trust_remote_code, so custom-code tokenizers reload after the merge - offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy Export orchestrator: - scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a large model does not time out at a flat 3600s
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
All three valid - fixed in cdafa38.
Verified: |
for more information, see https://pre-commit.ci
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
… hosts Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path. On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and continue hiding them on macOS/MLX.
- Multi-format merged export now collects every sibling output directory (one per selected precision) instead of only the last; the success banner lists them all. - Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations), so the panel says what is being exported rather than just 'Merged Model'. - Persist the selected formats in the run summary and seed them on mount, so navigating away and back (or toggling the export method) restores the selection instead of resetting to 16-bit.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15d22c5d13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| auto_model = AutoModelForImageTextToText if is_vlm else AutoModelForCausalLM | ||
| auto_processor = AutoProcessor if is_vlm else AutoTokenizer |
There was a problem hiding this comment.
When the portable torchao path is used on supported non-causal checkpoints (for example Whisper, which the export load path handles with WhisperForConditionalGeneration), the narrowed VLM test now leaves is_vlm false and this branch reloads the staged checkpoint with AutoModelForCausalLM/AutoTokenizer. Fresh evidence after the earlier VLM fix is that T5/BART/Whisper no longer go to the image-text class, but they still hit the wrong causal-LM auto class here and fail after the merge; either carry the original model class/audio type through or hide torchao formats for those architectures.
Useful? React with 👍 / 👎.
|
|
||
| Returns {export_supported, export_unsupported_reason, export_unsupported_message}. | ||
| """ | ||
| if get_device() in (DeviceType.CUDA, DeviceType.XPU, DeviceType.MLX): |
There was a problem hiding this comment.
On a CPU-only host with PyTorch installed, this new capability check still returns unsupported because DeviceType.CPU is excluded, and _ensure_export_supported() uses it to reject /load-checkpoint before any export starts. That blocks the new portable torchao FP8/INT8 flow the UI/constants describe as device-agnostic/CPU-capable (and also prevents a plain 16-bit local save) even though the torchao path no longer requires NVIDIA; allow the CPU+torch case and gate only formats that truly need an accelerator.
Useful? React with 👍 / 👎.
| from transformers import ( | ||
| AutoModelForCausalLM, | ||
| AutoTokenizer, | ||
| AutoModelForImageTextToText, |
There was a problem hiding this comment.
On Transformers versions that do not expose AutoModelForImageTextToText, this unconditional import aborts every portable torchao export before the code even checks whether the model is a VLM, so plain text models fail too. Other Unsloth paths guard this class with an AutoModelForVision2Seq fallback; apply the same lazy/fallback import here so text-only FP8/INT8 exports continue to work on supported Transformers installs.
Useful? React with 👍 / 👎.
… Q8_0, auto-select newest checkpoint - Progress/summary panel now shows a Formats row with the selected merged formats, and the success banner lists every output folder a multi-format merged run creates (one line per format) instead of only the last one. - Merged format selection is seeded from the active run, so navigating away and back (or switching method cards) no longer resets it to 16-bit. - GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA adapter) for adapter checkpoints, reusing the LoRA GGUF export path. - Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI, the request model, and the backend defaults; the outtype list is now Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers. - When a finetune has no checkpoint selected, auto-select the newest one.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Pushed export UX fixes from re-testing the page on a real finetune:
Verified end to end on a Llama-3.2-1B LoRA: tsc, eslint (no new errors), and the Vite build are clean. A merged 16-bit + FP8 run produced two output folders, both listed in the success banner with a Formats row of "16-bit, FP8". A GGUF LoRA export with the default output type produced a valid Q8_0 adapter, and gguf_outtype=auto is now rejected by the request model. |
Two fixes to the portable torchao FP8/INT8 export reload, from review of the narrowed VLM detection: - Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs. With the narrowed is_vlm test they now correctly skip the image-text class, but fell through to AutoModelForCausalLM and failed to reload after the merge. Reload them with their own architecture class from the config instead. - AutoModelForImageTextToText was imported unconditionally at the top of the torchao path, so on Transformers builds without that class the import aborted every torchao export (even text-only). Import it lazily only for a VLM, with the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
|
Addressed the torchao review findings (pushed in 96d1a78):
On the CPU-hosts capability finding: declining that one. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96d1a78ebb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from models.export import ExportLoRAAdapterRequest | ||
|
|
||
| r = ExportLoRAAdapterRequest(save_directory = "/tmp/x") | ||
| assert r.gguf is False and r.gguf_outtype == "f16" |
There was a problem hiding this comment.
This assertion is stale: ExportLoRAAdapterRequest now defaults gguf_outtype to q8_0, and the export worker fallback also uses q8_0, so the backend pytest suite will fail here before exercising the new GGUF LoRA path; the string check a few lines below has the same f16 expectation. Update these assertions, or change the implementation defaults, so CI can pass.
Useful? React with 👍 / 👎.
| sourceMode === "checkpoint" | ||
| ? checkpoint ?? selectedModelIdx ?? sourceBaseModelName | ||
| : sourceBaseModelName; | ||
| return `${safePathSegment(rawName)}-${exportMethod === "lora" ? "adapter" : "merged"}`; |
There was a problem hiding this comment.
When sourceMode === "model" and the newly enabled Merged method is used, this default writes a flat exports/<model>-merged directory. The exported-model scanner in studio/backend/utils/models/model_config.py only recognizes top-level directories as GGUF, then looks for merged config.json/weights one level deeper under a run/checkpoint directory, so a successful default Local/HF merged export never appears in the exported models list or Chat picker; use a scanner-supported layout or teach the scanner about flat merged exports.
Useful? React with 👍 / 👎.
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS. Run the quantization against a dedicated llm-compressor-main "shadow": a --target package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered over the existing torch. It installs --no-deps so torch is never touched (works on any Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN. - transformers_version.py: provision + validate .venv_llmcompressor. - export.py: route all compressed exports through the shadow when available; else keep the workspace 0.10.x path and fail fast past its transformers ceiling. - save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow. - _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the RedHatAI and NVIDIA reference quants, and is required by the grouped schemes). Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and fp8 on Gemma-4, end to end through Studio.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Pushed 710e19e: FP8/FP4 compressed export now works for newer-transformers models (Qwen3.5, Gemma-4, Qwen3-Next), which previously failed because the shipped llm-compressor 0.10.x pins transformers<=4.57.6. The quantization now runs against a dedicated llm-compressor-main shadow: a
Verified end to end through Studio:
Outputs match the reference module ignore lists (Qwen3.5 w4a16 keeps the linear_attn projections in bf16, same as RedHatAI/Qwen3.5-9B-quantized.w4a16). 16-bit and LoRA export are unchanged. |
…-dropdown # Conflicts: # studio/backend/main.py # unsloth/save.py
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
for more information, see https://pre-commit.ci
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
for more information, see https://pre-commit.ci
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
2 participants
Summary
Reworks the Studio export page into a single, cross-platform export flow.
Float8WeightOnlyConfig/Int8WeightOnlyConfig) that quantizes on any device, including CPU, AMD, and Apple. FP8 serializes to safetensors and INT8 to.bin; both load in vLLM. This is wired intosave_pretrained_merged/push_to_hub_mergedthrough aTORCHAO_EXPORT_SCHEMESregistry and a new_unsloth_save_torchao, parallel to the existing compressed-tensors path.convert_lora_to_gguf.py, loadable withllama-cli --lora) with an output-type select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter.How it works
The precision pills and the "More formats" dropdown both toggle into one
selectedFormatslist. Each selected format maps to a backend payload: 16-bit is a plain save, a compressed alias (fp8,w4a16,mxfp4, ...) resolves viaunsloth.save.COMPRESSED_EXPORT_SCHEMES, and a portable alias (torchao_fp8,torchao_int8) resolves viaTORCHAO_EXPORT_SCHEMES. The store loops the merged selections (one call per format) and sends the GGUF quant list in a single call. NVIDIA presence comes fromuseHardwareInfo()(cuda != null && rocm == null) in the UI and fromDEVICE == CUDA and not IS_ROCMin the backend. The new fields are threaded through the request models, routes, orchestrator, and worker.Verification
ExportBackend, driven with a real Llama-3.2-1B LoRA: torchao FP8 merged (sibling-torchao-fp8, safetensors), the NVIDIA guard (compressed rejected on a simulated non-NVIDIA box, portable still allowed), the relaxed non-PEFT merged export, GGUF multi-quant (Q4_K_M + Q8_0 in one call), and GGUF LoRA (*-lora-f16.gguf) all succeed.tsc -bclean andeslintadds no new errors; Vite build clean; live UI drive confirms the multi-select pills, the "More formats" dropdown, the GGUF LoRA option, and GGUF multi-quant all render and toggle.Export gating (PyTorch / accelerator)
Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even with PyTorch installed (Unsloth calls
torch.cudaunconditionally at import). A newhardware.export_capability()reportsexport_supportedplus a precise reason so the UI stops showing a generic "no GPU":pytorch_not_installed: a--no-torchinstall, where even a physical GPU is unusable without PyTorch.no_accelerator: PyTorch present but no supported accelerator (bare CPU).mlx_unavailable: Apple Silicon where the MLX stack is missing or too old.The fields are exposed on
/api/system/hardwareand/api/system. The mutating export routes reject up front with HTTP 400 and the reason, while read-only endpoints stay usable so the Export page still renders.core/export/export.pynow imports without PyTorch and without a usable accelerator (the Unsloth import is caught), so the export worker degrades to a clear message instead of crashing at import. On the frontend the Export page stays reachable on chat-only hosts and grays out the method and format options with the backend reason (an alert plus disabled pickers) instead of silently redirecting to chat.This also fixes the export save directory producing "model/null" for Local Model and Hugging Face sources that have no run/checkpoint.
Cross-platform verification
export_capability()correctly reportsno_accelerator, and an export attempt returns the clear message without crashing. This confirms Unsloth cannot export on bare CPU (it fails attorch.cudainit), so the gate blocks it honestly instead of letting a user click into a runtime crash.studio-export-capability-ci.ymlruns the capability and import-safety tests on ubuntu-latest, windows-latest and macos-latest. Hosted runners have no GPU or MLX, so this validates the "export unavailable, here is why" path on real Windows and macOS; a real accelerator export is validated on GPU hardware.tsc -bclean;eslintadds no new errors.