Skip to content

Revisit macOS ONNX accelerator selection for Whispering #1951

Description

@braden-w

PR #1946 forced OrtAccelerator::CpuOnly on macOS for Whispering's shipped ONNX engines because our current int8 Parakeet/Moonshine setup measured faster on CPU than through OrtAccelerator::Auto/CoreML.

Measured result from that investigation:

  • CPU-only: ~3.5 us/sample
  • Warm CoreML through Auto: ~12 us/sample
  • Cold CoreML: ~24 us/sample

Upstream data point: cjpais/transcribe-rs#86
Downstream fix: #1946

This is not currently a Whispering bug. CPU-only is the measured correct default for our shipped macOS ONNX models.

Revisit this when one of these changes:

  • transcribe-rs adds model-aware provider selection
  • Handy/transcribe-rs moves these models to a transcribe.cpp/ggml backend
  • Whispering ships different ONNX models or quantization formats
  • CoreML EP support changes enough that int8 Parakeet/Moonshine should be re-benchmarked

Done when:

  • Current macOS transcription models are re-benchmarked
  • The CPU-only override is removed or narrowed only if Auto/CoreML is faster for the shipped models

Metadata

Metadata

Assignees

No one assigned

    Labels

    mac osIssues specific to macOS build, features, or bugspriority-lowLow urgency; can be handled later

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions