Skip to content

OpenVINO

Jump to bottom
Disty0 edited this page Jan 12, 2024 · 34 revisions

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

  • Compiles models for your hardware.
  • Supports Linux and Windows
  • Supports CPU/GPU/GNA/NPU.
  • Supports AMD GPUs on Windows with FP16 support. (Double performance over DirectML.)
  • Supports INTEL GPUs including iGPUs.
  • Supports NVIDIA GPUs.
  • Supports CPUs with FP16, INT8 and INT4 support. (Double performance over PyTorch.)
  • Supports multiple devices at the same time using Hetero Device.

It is basically a TensorRT / Olive competitor that works with any hardware.

INT8 / INT4

Currently, INT8 and INT4 is CPU only. Enable Compress X weights with NNCF options in Compute Settings to use it.
Select a 4 bit mode from OpenVINO compress mode for NNCF to use INT4.

Custom Devices

Setting OPENVINO_TORCH_BACKEND_DEVICE environment value will override the device used for OpenVINO.
You can experiment with stuff like OPENVINO_TORCH_BACKEND_DEVICE=HETERO:GNA,GPU.1,GPU.0,CPU

Using --device-id cli argument with the WebUI will use a GPU with the specified Device ID.
Using --use-cpu openvino cli argument with the WebUI will use the CPU.

Hetero Device

OpenVINO use Hetero Device for single inference with multiple devices in Compute Settings
This Option will parse every device available for OpenVINO and use HETERO:Device0,Device1,Device2,(...),CPU as a single device.

OpenVINO remove CPU from Hetero Device option in Compute Settings will remove CPU from the available devices for OpenVINO.
OpenVINO remove iGPU from Hetero Device option in Compute Settings will remove GPU.0 from the available devices for OpenVINO.

Model Caching

OpenVINO will save compiled models to cache folder so you won't have to compile them again.
OpenVINO disable model caching option in Compute Settings will disable caching.
Directory for OpenVINO cache option in System Paths will set a new location for saving OpenVINO caches.

Limitations

Same limitations with TensorRT / Olive applies here too.
Compilation takes a few minutes and any change to Resolution / Batch Size / LoRa will trigger recompilation.
Attention Slicing and HyperTile will not work.
OpenVINO will lock you in the Diffusers backend.

Installation

Preparations

  • Install the drivers for your device.
  • Install git and python.
  • Open CMD in a folder you want to install SD.Next.

Using SD.Next with OpenVINO

Install SD.Next from Github:

git clone https://github.com/vladmandic/automatic

Then enter into the automatic folder:

cd automatic

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino

© SD.Next

Clone this wiki locally