Skip to content

CPU Execution Provider

General

The CPU Execution Provider enables fast inference for HALCON deep learning models through the ONNX Runtime. Especially on aarch64-linux systems, it is significantly faster than the built-in cpu runtime of HALCON.

System requirements

OS Architecture Requirements
Windows x64 No additional requirements.
Linux x64 GLIBC_2.35, GLIBCXX_3.4.33*
Linux aarch64 GLIBC_2.34, GLIBCXX_3.4.30*

* On Ubuntu Linux the GLIBCXX requirements imply at least Ubuntu 24.04 LTS.

Installation

The CPU Execution Provider is included in the ONNX Runtime AI² package. No additional installation is required.

See Installation of ONNX Runtime AI² for more details.

Workflows

The CPU Execution Provider can be used to optimize a deep learning model for inference and run the optimized model on the selected device. The optimized model can also be serialized and used on another system.

Optimize and infer on the same system

If the model is optimized and used on the same system the workflow follows these three steps:

  1. Query the CPU Execution Provider device
  2. Optionally configure the device
  3. Create an optimized model

This example code illustrates the steps:

* 1.) Query the CPU Execution Provider device
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)

* 2.) Optionally configure the device
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
SettableDeviceParams.inter_op_threads := 8
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)

* 3.) Create an optimized model
get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)
* OptimizeForInferenceParams.graph_optimization_level := 99
optimize_dl_model_for_inference (DLModelHandle, DLDevice, 'float32', [], OptimizeForInferenceParams,\
                                 DLModelHandleConverted, ConversionReport)
* If the original full model is not used anymore, clean it up to save resources
clear_dl_model (DLModelHandle)

* DLModelHandleConverted is now ready to be used.
* The ONNX Runtime CPU Execution Provider device is already set.

Optimize on a development system and deploy on target systems for inference

In this scenario the optimized model has to be serialized and deserialized. This way, the optimization operation only needs to be performed once, reducing memory consumption and runtime on target systems.

On the development system:

  1. Query the CPU Execution Provider device
  2. Create an optimized model
  3. Serialize the optimized model
* 1.) Query the CPU Execution Provider device
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)

* 2.) Create an optimized model
get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)
* OptimizeForInferenceParams.graph_optimization_level := 99
optimize_dl_model_for_inference (DLModelHandle, DLDevice, 'float32', [], OptimizeForInferenceParams,\
                                 DLModelHandleConverted, ConversionReport)

* 3.) Serialize the optimized model
write_dl_model (DLModelHandleConverted, 'optimized.hdl')

On the target system:

  1. Read the optimized model
  2. Query the CPU Execution Provider device
  3. Optionally configure the device
  4. Assign the device
* 1.) Read the optimized model
read_dl_model ('optimized.hdl', DLModelHandleConverted)

* 2.) Query the CPU Execution Provider device
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)

* 3.) Optionally configure the device
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* SettableDeviceParams.inter_op_threads := 8
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)

* 4.) Assign the device
set_dl_model_param (DLModelHandleConverted, 'device', DLDevice)

* DLModelHandleConverted is now ready to be used.
* The ONNX Runtime CPU Execution Provider device is set.

Reference documentation

This section provides a detailed explanation of operations and parameters of the CPU Execution Provider.

Device query

Use query_available_dl_devices to query the ONNX Runtime AI² devices available on the system.

To select the CPU Execution Provider, query the device using the following parameters:

Parameter Value Type Description
ai2 onnxruntime string Identifies the ONNX Runtime AI² interface.
execution_provider CPU string Identifies the CPU Execution Provider.
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)

If DLDevice is empty, it means no device could be found. This can have multiple reasons:

Device configuration

The following parameter of get_dl_device_param is relevant for the configuration of the device:

Parameter Type Description
settable_device_params dictionary Provides the runtime-specific device parameters that can be configured.
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)

The settable_device_params dictionary contains the following parameters:

Name Type Values Default Description
enable_cpu_mem_arena string 'true', 'false' 'true' Enables or disables the CPU memory arena, which pre-allocates large memory blocks and reuses them to avoid repeated system allocations. Enabling the option speeds up inference, while disabling it reduces memory usage at the cost of performance.
enable_mem_pattern string 'true', 'false' 'true' Enables or disables memory pattern optimization, which records the memory allocation pattern during the first inference and reuses it for subsequent runs with a pre-planned contiguous block.
execution_mode string 'ort_sequential', 'ort_parallel' 'ort_sequential' Controls whether ONNX operators in the graph are executed sequentially or in parallel. For models with many branches, 'ort_parallel' may improve performance.
inter_op_threads integer 0..256 0 Specifies the number of threads used for parallel execution between ONNX operators. A value of 0 lets ONNX Runtime determine the number of threads.
intra_op_threads integer 0..256 0 Specifies the number of threads used for parallel execution within ONNX operators. A value of 0 lets ONNX Runtime determine the number of threads.
log_level string 'error', 'warning', 'info' 'error' Specifies the logging level.
set_denormal_as_zero string 'true', 'false' 'true' Enables or disables flushing denormal floating-point values to zero. This avoids slowdowns some CPUs incur when processing denormal values, which can occur in quantized or near-zero-activation models, at the cost of a negligible loss of precision for values near zero.

More information on the supported parameters can be found at the ONNX Runtime Documentation .

For example, the number of threads used for parallel execution between operators can be configured using inter_op_threads:

get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
SettableDeviceParams.inter_op_threads := 8
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* In order for the changes to take effect, the device has to be assigned again
set_dl_model_param (DLModelHandleConverted, 'device', DLDevice)

Device parameters that affect model execution have to be configured before the device is assigned to the deep learning model.

Threading

ONNX Runtime distinguishes between:

  • Intra-op parallelism, which controls parallel execution within individual ONNX operators.
  • Inter-op parallelism, which controls parallel execution between ONNX operators.
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
SettableDeviceParams.inter_op_threads := 8
SettableDeviceParams.intra_op_threads := 2
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* In order for the changes to take effect, the device has to be assigned again
set_dl_model_param (DLModelHandleConverted, 'device', DLDevice)

Optimization configuration

The following parameters are relevant for configuring the optimization step of CPU Execution Provider:

Parameter Type Description
optimize_for_inference_params dictionary Provides the parameters supported for model optimization in optimize_dl_model_for_inference

Example code:

get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)

The optimize_for_inference_params dictionary provides parameters for model optimization:

Parameter Values Type Description
graph_optimization_level 0..99 integer Graph optimization level: 0=disabled, 1=basic, 2=extended, 3=layout, 99=all (default).

1 Basic Graph Optimizations: These are semantics-preserving graph rewrites which remove redundancies. For example, parts of the graph that rely on constant initializers are computed in advance which eliminates their overhead during runtime.

2 Extended Graph Optimizations: These optimizations include complex node fusions.

3 Layout Optimizations: These optimizations change the data layout for applicable nodes to achieve higher performance improvements.

Example code:

get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)
OptimizeForInferenceParams.graph_optimization_level := 0
optimize_dl_model_for_inference (DLModelHandle, DLDevice, 'float32', [], OptimizeForInferenceParams,\
                                 DLModelHandleConverted, ConversionReport)

Device information

The device information can be retrieved using get_dl_device_param.

Parameter Type Description
name string Name of the CPU device.
cast_precisions tuple of strings Returns 'float32'.
info dictionary Provides information about the ONNX Runtime AI² interface.

Example code:

get_dl_device_param (DLDevice, 'name', CpuName)
get_dl_device_param (DLDevice, 'cast_precisions', CastPrecisions)
get_dl_device_param (DLDevice, 'info', DeviceInfo)

The info dictionary contains additional information about the AI².

Value Type Description
onnxruntime_version string Version of ONNX Runtime.
plugin_version string Version of the ONNX Runtime AI² plugin.

Unsupported models

Model Description
Instance Segmentation Models created with create_dl_model_detection where 'instance_segmentation' is set to 'true'.

HDevelop Examples

The following example is present in HDevelop after an installation of the ONNX Runtime AI² with SOM:

  • examples/hdevelop/Deep-Learning/run_deep_ocr_on_onnxruntime_cpu_ep.hdev - Demonstrates running inference on a pre-trained OCR model using the CPU Execution Provider.

Usage of third-party libraries

This execution provider does not use any additional third-party libraries other than the ONNX Runtime AI² itself.

See Usage of third-party libraries for details.