Skip to content

OpenVINO

Jump to bottom
Disty0 edited this page Oct 26, 2023 · 34 revisions

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

  • Compiles models for your hardware.
  • Supports CPU/GPU/GNA/NPU.
  • Supports INTEL GPUs on both Linux and Windows.
  • Supports AMD GPUs on Windows with FP16 support. (Double performance over DirectML.)
  • Supports NVIDIA GPUs on both Linux and Windows.
  • Supports CPUs with FP16 support. (Double performance over PyTorch.)
  • Supports multiple devices at the same time using Hetero Device.

It is basically a TensorRT / Olive competitor that works with any hardware.

Custom Devices

Setting OPENVINO_TORCH_BACKEND_DEVICE environment value will override the device used for OpenVINO.
You can experiment with stuff like OPENVINO_TORCH_BACKEND_DEVICE=HETERO:GNA,GPU.1,GPU.0,CPU

Using --device-id cli argument with the WebUI will use a GPU with the specified Device ID.
Using --use-cpu openvino cli argument with the WebUI will use the CPU.

Hetero Device

OpenVINO use Hetero Device for single inference with multiple devices
This option will parse every device available for OpenVINO and use HETERO:Device0,Device1,Device2,(...),CPU as a single device.

OpenVINO remove CPU from Hetero Device option will remove CPU from the available devices for OpenVINO.
OpenVINO remove iGPU from Hetero Device option will remove GPU.0 from the available devices for OpenVINO.

Model Caching

OpenVINO will save compiled models to cache folder so you won't have to compile them again.
OpenVINO disable model caching option will disable caching.

Limitations

Same limitations with TensorRT / Olive applies here too.
Compilation takes a few minutes and any change to Resolution / Batch Size / LoRa will trigger recompilation.
Attention Slicing and HyperTile will not work.
OpenVINO will lock you in the Diffusers backend.

Installation

Preparations

  • Install the drivers for your device.
  • Install git and python.
  • Open CMD in a folder you want to install SD.Next.

Using SD.Next with OpenVINO

Install SD.Next from Github:

git clone https://github.com/vladmandic/automatic

Then enter into the automatic folder:

cd automatic

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino

© SD.Next

Clone this wiki locally