MLContextOptions.deviceType seems unnecessary outside of conformance testing

[mwyrzykowski]

WebKit has a few objections to MLContextOptions.deviceType:

MLContextOptions.deviceType is currently unimplementable currently via CoreML

CoreML is the only public framework on macOS for running computations on the NPU / ANE. The reason as pointed out #623 (comment) and possibly elsewhere, CoreML allows you to specify MLComputeUnitsCPUOnly, MLComputeUnitsCPUAndGPU, MLComputeUnitsAll, and MLComputeUnitsCPUAndNeuralEngine, but not MLComputeUnitsNeuralEngine only for instance. Even assuming we could request CoreML to add support for only the ANE, that support would be limited to a future version of macOS and therefore could only be supported on currently unreleased versions of macOS.

Also note, MLComputeUnitsCPUAndNeuralEngine, is only available on macOS 13.0 and later and I'm not sure if some browser vendors want to support earlier versions of macOS.

MLContextOptions.deviceType, as currently specified, would lead to additional fragmentation, works on some devices, fails on others

Under Section 4 we see the wording:

if this type cannot be satisfied, an "OperationError" DOMException is thrown,

It would be preferable to fallback instead of failing, otherwise a website which specifies NPU and runs on a variety of NPUs, but maybe not the NPU of a certain device, the website would not run if the author does not handle fallback. This seems unnecessary, when the browser could fallback to the GPU or CPU themselves.

The browser has better insight into workloads than the website author

It was pointed out in #322 that a website author may want their models to not run on the GPU so they don't impact WebGPU rendering performance. While a good idea in theory, the browser implementation and native frameworks have much greater insight into the graphics chip utilization and the most appropriate resource to run a model on without impacting target frameworks.

The answer to this question, where to run the model, can not be easily decided by the website as the website does not know the underlying hardware it is running on due to privacy reasons and possibly other reasons. Additionally, while browser vendor can trivially compute the amount of time a GPUCommandBuffer takes to submit from WebGPU, such timing is not available to the website. For example, if all the GPUCommandBuffers in the WebGPU workload complete in <3ms on a Mac Studio, and the website calls requestAnimationFrame every ~16ms, then restricting the WebNN model from running on the GPU seems unnecessary. However on a less powerful device, like an iPhone 12 mini, if the WebGPU workload is falling behind the requested update rate, the browser implementation may move the model processing to the device least in use. To the website, a browser implementation may present both devices as having the same capabilities for privacy reasons, so there is no way for the website to make the correct decision which device to run on.

Often with graphics intensive WebGPU applications, we find the GPU is still the least utilized processor as the CPU overhead can be high. Long running shaders or complex compute kernels are two exceptions. In any case this is device dependent and not something a website author can easily predict.

To summarize it seems MLDeviceType is useful for testing in browser implementations, but seems counter-productive for framework authors and general websites / wasm applications.