Optional NPU-Accelerated Post-Processing (ONNX Runtime / DirectML) #633

Closed

opened 2026-11-08 15:09:27 +00:00 by makdiller · 2 comments

makdiller commented

2026-11-08 15:09:27 +00:00

(Migrated from github.com)

The idea

Summary
Add an optional ML-based post-processing filter pipeline (leveraging ONNX Runtime or DirectML execution providers) under the Video / Post-Processing settings. This allows users on mobile and integrated platforms to run neural upscaling models or motion interpolation using dedicated NPU hardware without consuming GPU compute resources or triggering Vulkan translation bottlenecks.

Motivation
Running heavy neural network operations (such as Real-ESRGAN spatial upscaling or frame interpolation) alongside the core Vulkan graphics loop can create significant rendering overhead and thermal throttling on integrated graphics (e.g., AMD Radeon 800-series iGPUs).

On modern mobile processors equipped with NPUs (such as AMD Ryzen AI / Strix Point APUs), offloading these post-processing passes to dedicated AI hardware keeps the primary iGPU focused strictly on rasterization, frame presentation, and core shader execution while keeping overall package power draw low.

Proposed Solution
NPU Execution Provider Integration: Integrate ONNX Runtime with support for NPU execution backends (such as Windows ML / DirectML or native NPU EP targets).

Post-Processing Pass Offload: Frame neural upscaling or spatial reconstruction models as an optional Post-Processing / Overlay Filter (similar to post-raster ML upscaling) that processes completed swapchain frames asynchronously.

Settings Toggle: Add an opt-in toggle under Video Settings (e.g., Enable NPU Acceleration for ML Filters) with fallback to standard spatial upscaling (FSR / Bilinear) if no supported NPU execution provider is detected.

Benefits
Power & Thermal Efficiency: NPUs operate at low wattage, reducing overall CPU/GPU package power draw and preventing thermal throttling on thin-and-light mobile workstations.

iGPU Overhead Relief: Frees up 100% of the integrated graphics compute units for the native Vulkan rendering thread.

Clean Architecture Boundary: Keeps standard GPU shader compilation and Vulkan pipeline logic completely isolated from experimental AI frameworks.

Hardware & OS Target
Platforms: Windows 11 (64-bit) / DirectML / Windows ML Execution Providers.

Target Hardware: AMD Ryzen AI / XDNA NPUs, Intel Core Ultra AI Boost, Qualcomm Hexagon NPU targets.

Anything else

No response

### The idea Summary Add an optional ML-based post-processing filter pipeline (leveraging ONNX Runtime or DirectML execution providers) under the Video / Post-Processing settings. This allows users on mobile and integrated platforms to run neural upscaling models or motion interpolation using dedicated NPU hardware without consuming GPU compute resources or triggering Vulkan translation bottlenecks. Motivation Running heavy neural network operations (such as Real-ESRGAN spatial upscaling or frame interpolation) alongside the core Vulkan graphics loop can create significant rendering overhead and thermal throttling on integrated graphics (e.g., AMD Radeon 800-series iGPUs). On modern mobile processors equipped with NPUs (such as AMD Ryzen AI / Strix Point APUs), offloading these post-processing passes to dedicated AI hardware keeps the primary iGPU focused strictly on rasterization, frame presentation, and core shader execution while keeping overall package power draw low. Proposed Solution NPU Execution Provider Integration: Integrate ONNX Runtime with support for NPU execution backends (such as Windows ML / DirectML or native NPU EP targets). Post-Processing Pass Offload: Frame neural upscaling or spatial reconstruction models as an optional Post-Processing / Overlay Filter (similar to post-raster ML upscaling) that processes completed swapchain frames asynchronously. Settings Toggle: Add an opt-in toggle under Video Settings (e.g., Enable NPU Acceleration for ML Filters) with fallback to standard spatial upscaling (FSR / Bilinear) if no supported NPU execution provider is detected. Benefits Power & Thermal Efficiency: NPUs operate at low wattage, reducing overall CPU/GPU package power draw and preventing thermal throttling on thin-and-light mobile workstations. iGPU Overhead Relief: Frees up 100% of the integrated graphics compute units for the native Vulkan rendering thread. Clean Architecture Boundary: Keeps standard GPU shader compilation and Vulkan pipeline logic completely isolated from experimental AI frameworks. Hardware & OS Target Platforms: Windows 11 (64-bit) / DirectML / Windows ML Execution Providers. Target Hardware: AMD Ryzen AI / XDNA NPUs, Intel Core Ultra AI Boost, Qualcomm Hexagon NPU targets. ### Anything else _No response_

👎 1

govt-boot-licker commented

2026-11-08 17:37:10 +00:00

(Migrated from github.com)

Yeah recent intel can do hardware upscale and framegen using XMX core, would be awesome to see

NotRexed commented

2026-11-09 05:52:09 +00:00

(Migrated from github.com)

Thanks for the idea. Feature requests are now collected in #1007, which lists this one with a link back here, so closing it in favour of that list (this is not a decision against the idea).

Commenting is not possible because the repository is archived.