Add OOTB tests for macos-m1-stable runner

[tom-arm]

• Separated out from Arm backend: Add OOTB tests #12392
• Make job names more specific

Signed-off-by: Tom Allsop [email protected]

Change-Id: Icdfba4bda4ed7021d6eed5698f529fee60919cdb

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13416

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 49 Unrelated Failures

As of commit 333926d with merge base 36d342c ():

NEW FAILURES - The following jobs have failed:

• Apple / build-frameworks-ios / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-huggingface-optimum-coreml (smollm|coreml_fp32_gpu|--quantize) / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 4e5d540d7c6e96fe522649d8498ae97f49b3ed69c234754a95b1654f2e86e302 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, xnnpack+custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh)
  RuntimeError: Command docker exec -t df4af2c89b906a0d43a61c1e32cd7c4d619eda6301b5c2cd7d38b1581d9a4c55 /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• trunk / test-arm-backend-ootb-macos / macos-job (gh) (similar failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-huggingface-optimum-coreml (distilbert|coreml_fp32_gpu|--quantize) / macos-job (gh) (similar failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / macos / macos-job (gh) (similar failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Apple / build-demo-ios / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• pull / unittest / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-buck / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-custom-ops-macos (cmake) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-huggingface-optimum-coreml (bert|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-huggingface-optimum-coreml (llama3|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-huggingface-optimum-coreml (olmo|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-huggingface-optimum-coreml (qwen3|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-torchao-lowbit / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-arm-zephyr (add) / linux-job (gh) (trunk failure)
  /pytorch/executorch/zephyr_scratch/modules/lib/executorch/kernels/portable/cpu/op__clone_dim_order.cpp:64:49: error: expected primary-expression before ‘>’ token
• trunk / test-models-arm-zephyr (mv2) / linux-job (gh) (trunk failure)
  /pytorch/executorch/zephyr_scratch/modules/lib/executorch/kernels/portable/cpu/op__clone_dim_order.cpp:64:49: error: expected primary-expression before ‘>’ token
• trunk / test-models-arm-zephyr (softmax) / linux-job (gh) (trunk failure)
  /pytorch/executorch/zephyr_scratch/modules/lib/executorch/kernels/portable/cpu/op__clone_dim_order.cpp:64:49: error: expected primary-expression before ‘>’ token
• trunk / test-models-macos-coreml (dl3) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (edsr) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (emformer_join) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (ic3) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (ic4) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (mobilebert) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (mv2) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (mv3) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (resnet50) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (vit) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (w2l) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (efficient_sam, portable) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (emformer_join, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (llama, portable) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (llama3_2_vision_encoder, portable) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (mv3, portable) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (mv3, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-mps / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-selective-build-macos (cmake) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-static-llama-ane / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.