PR #1946 forced OrtAccelerator::CpuOnly on macOS for Whispering's shipped ONNX engines because our current int8 Parakeet/Moonshine setup measured faster on CPU than through OrtAccelerator::Auto/CoreML.
Measured result from that investigation:
- CPU-only: ~3.5 us/sample
- Warm CoreML through Auto: ~12 us/sample
- Cold CoreML: ~24 us/sample
Upstream data point: cjpais/transcribe-rs#86
Downstream fix: #1946
This is not currently a Whispering bug. CPU-only is the measured correct default for our shipped macOS ONNX models.
Revisit this when one of these changes:
transcribe-rs adds model-aware provider selection
- Handy/transcribe-rs moves these models to a
transcribe.cpp/ggml backend
- Whispering ships different ONNX models or quantization formats
- CoreML EP support changes enough that int8 Parakeet/Moonshine should be re-benchmarked
Done when:
- Current macOS transcription models are re-benchmarked
- The CPU-only override is removed or narrowed only if Auto/CoreML is faster for the shipped models
PR #1946 forced
OrtAccelerator::CpuOnlyon macOS for Whispering's shipped ONNX engines because our current int8 Parakeet/Moonshine setup measured faster on CPU than throughOrtAccelerator::Auto/CoreML.Measured result from that investigation:
Upstream data point: cjpais/transcribe-rs#86
Downstream fix: #1946
This is not currently a Whispering bug. CPU-only is the measured correct default for our shipped macOS ONNX models.
Revisit this when one of these changes:
transcribe-rsadds model-aware provider selectiontranscribe.cpp/ggml backendDone when: