CPU Execution Provider
General
The CPU Execution Provider enables fast inference for HALCON deep learning models through the ONNX Runtime.
Especially on aarch64-linux systems, it is significantly faster than the built-in cpu runtime of HALCON.
System requirements
| OS | Architecture | Requirements |
|---|---|---|
| Windows | x64 |
No additional requirements. |
| Linux | x64 |
GLIBC_2.35, GLIBCXX_3.4.33* |
| Linux | aarch64 |
GLIBC_2.34, GLIBCXX_3.4.30* |
* On Ubuntu Linux the GLIBCXX requirements imply at least Ubuntu 24.04 LTS.
Installation
The CPU Execution Provider is included in the ONNX Runtime AI² package. No additional installation is required.
See Installation of ONNX Runtime AI² for more details.
Workflows
The CPU Execution Provider can be used to optimize a deep learning model for inference and run the optimized model on the selected device. The optimized model can also be serialized and used on another system.
Optimize and infer on the same system
If the model is optimized and used on the same system the workflow follows these three steps:
- Query the CPU Execution Provider device
- Optionally configure the device
- Create an optimized model
This example code illustrates the steps:
* 1.) Query the CPU Execution Provider device
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)
* 2.) Optionally configure the device
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
SettableDeviceParams.inter_op_threads := 8
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* 3.) Create an optimized model
get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)
* OptimizeForInferenceParams.graph_optimization_level := 99
optimize_dl_model_for_inference (DLModelHandle, DLDevice, 'float32', [], OptimizeForInferenceParams,\
DLModelHandleConverted, ConversionReport)
* If the original full model is not used anymore, clean it up to save resources
clear_dl_model (DLModelHandle)
* DLModelHandleConverted is now ready to be used.
* The ONNX Runtime CPU Execution Provider device is already set.
Optimize on a development system and deploy on target systems for inference
In this scenario the optimized model has to be serialized and deserialized. This way, the optimization operation only needs to be performed once, reducing memory consumption and runtime on target systems.
On the development system:
- Query the CPU Execution Provider device
- Create an optimized model
- Serialize the optimized model
* 1.) Query the CPU Execution Provider device
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)
* 2.) Create an optimized model
get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)
* OptimizeForInferenceParams.graph_optimization_level := 99
optimize_dl_model_for_inference (DLModelHandle, DLDevice, 'float32', [], OptimizeForInferenceParams,\
DLModelHandleConverted, ConversionReport)
* 3.) Serialize the optimized model
write_dl_model (DLModelHandleConverted, 'optimized.hdl')
On the target system:
- Read the optimized model
- Query the CPU Execution Provider device
- Optionally configure the device
- Assign the device
* 1.) Read the optimized model
read_dl_model ('optimized.hdl', DLModelHandleConverted)
* 2.) Query the CPU Execution Provider device
query_available_dl_devices (['ai2', 'execution_provider'], ['onnxruntime', 'CPU'], DLDevice)
* 3.) Optionally configure the device
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* SettableDeviceParams.inter_op_threads := 8
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* 4.) Assign the device
set_dl_model_param (DLModelHandleConverted, 'device', DLDevice)
* DLModelHandleConverted is now ready to be used.
* The ONNX Runtime CPU Execution Provider device is set.
Reference documentation
This section provides a detailed explanation of operations and parameters of the CPU Execution Provider.
Device query
Use query_available_dl_devices to query the ONNX Runtime AI² devices available on the system.
To select the CPU Execution Provider, query the device using the following parameters:
| Parameter | Value | Type | Description |
|---|---|---|---|
ai2 |
onnxruntime |
string | Identifies the ONNX Runtime AI² interface. |
execution_provider |
CPU |
string | Identifies the CPU Execution Provider. |
If DLDevice is empty, it means no device could be found. This can have multiple reasons:
- ONNX Runtime AI² is not installed (see Installation)
- The system requirements have not been met (see System requirements)
Device configuration
The following parameter of get_dl_device_param is relevant for the configuration of the device:
| Parameter | Type | Description |
|---|---|---|
settable_device_params |
dictionary | Provides the runtime-specific device parameters that can be configured. |
The settable_device_params dictionary contains the following parameters:
| Name | Type | Values | Default | Description |
|---|---|---|---|---|
enable_cpu_mem_arena |
string | 'true', 'false' |
'true' |
Enables or disables the CPU memory arena, which pre-allocates large memory blocks and reuses them to avoid repeated system allocations. Enabling the option speeds up inference, while disabling it reduces memory usage at the cost of performance. |
enable_mem_pattern |
string | 'true', 'false' |
'true' |
Enables or disables memory pattern optimization, which records the memory allocation pattern during the first inference and reuses it for subsequent runs with a pre-planned contiguous block. |
execution_mode |
string | 'ort_sequential', 'ort_parallel' |
'ort_sequential' |
Controls whether ONNX operators in the graph are executed sequentially or in parallel. For models with many branches, 'ort_parallel' may improve performance. |
inter_op_threads |
integer | 0..256 |
0 |
Specifies the number of threads used for parallel execution between ONNX operators. A value of 0 lets ONNX Runtime determine the number of threads. |
intra_op_threads |
integer | 0..256 |
0 |
Specifies the number of threads used for parallel execution within ONNX operators. A value of 0 lets ONNX Runtime determine the number of threads. |
log_level |
string | 'error', 'warning', 'info' |
'error' |
Specifies the logging level. |
set_denormal_as_zero |
string | 'true', 'false' |
'true' |
Enables or disables flushing denormal floating-point values to zero. This avoids slowdowns some CPUs incur when processing denormal values, which can occur in quantized or near-zero-activation models, at the cost of a negligible loss of precision for values near zero. |
More information on the supported parameters can be found at the ONNX Runtime Documentation .
For example, the number of threads used for parallel execution between operators can be configured using inter_op_threads:
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
SettableDeviceParams.inter_op_threads := 8
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* In order for the changes to take effect, the device has to be assigned again
set_dl_model_param (DLModelHandleConverted, 'device', DLDevice)
Device parameters that affect model execution have to be configured before the device is assigned to the deep learning model.
Threading
ONNX Runtime distinguishes between:
- Intra-op parallelism, which controls parallel execution within individual ONNX operators.
- Inter-op parallelism, which controls parallel execution between ONNX operators.
get_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
SettableDeviceParams.inter_op_threads := 8
SettableDeviceParams.intra_op_threads := 2
set_dl_device_param (DLDevice, 'settable_device_params', SettableDeviceParams)
* In order for the changes to take effect, the device has to be assigned again
set_dl_model_param (DLModelHandleConverted, 'device', DLDevice)
Optimization configuration
The following parameters are relevant for configuring the optimization step of CPU Execution Provider:
| Parameter | Type | Description |
|---|---|---|
optimize_for_inference_params |
dictionary | Provides the parameters supported for model optimization in optimize_dl_model_for_inference |
Example code:
The optimize_for_inference_params dictionary provides parameters for model optimization:
| Parameter | Values | Type | Description |
|---|---|---|---|
graph_optimization_level |
0..99 |
integer | Graph optimization level: 0=disabled, 1=basic, 2=extended, 3=layout, 99=all (default). |
1 Basic Graph Optimizations:
These are semantics-preserving graph rewrites which remove redundancies. For example, parts of the graph that rely on constant initializers are computed in advance which eliminates their overhead during runtime.
2 Extended Graph Optimizations:
These optimizations include complex node fusions.
3 Layout Optimizations:
These optimizations change the data layout for applicable nodes to achieve higher performance improvements.
Example code:
get_dl_device_param (DLDevice, 'optimize_for_inference_params', OptimizeForInferenceParams)
OptimizeForInferenceParams.graph_optimization_level := 0
optimize_dl_model_for_inference (DLModelHandle, DLDevice, 'float32', [], OptimizeForInferenceParams,\
DLModelHandleConverted, ConversionReport)
Device information
The device information can be retrieved using get_dl_device_param.
| Parameter | Type | Description |
|---|---|---|
name |
string | Name of the CPU device. |
cast_precisions |
tuple of strings | Returns 'float32'. |
info |
dictionary | Provides information about the ONNX Runtime AI² interface. |
Example code:
get_dl_device_param (DLDevice, 'name', CpuName)
get_dl_device_param (DLDevice, 'cast_precisions', CastPrecisions)
get_dl_device_param (DLDevice, 'info', DeviceInfo)
The info dictionary contains additional information about the AI².
| Value | Type | Description |
|---|---|---|
onnxruntime_version |
string | Version of ONNX Runtime. |
plugin_version |
string | Version of the ONNX Runtime AI² plugin. |
Unsupported models
| Model | Description |
|---|---|
| Instance Segmentation | Models created with create_dl_model_detection where 'instance_segmentation' is set to 'true'. |
HDevelop Examples
The following example is present in HDevelop after an installation of the ONNX Runtime AI² with SOM:
examples/hdevelop/Deep-Learning/run_deep_ocr_on_onnxruntime_cpu_ep.hdev- Demonstrates running inference on a pre-trained OCR model using the CPU Execution Provider.
Usage of third-party libraries
This execution provider does not use any additional third-party libraries other than the ONNX Runtime AI² itself.
See Usage of third-party libraries for details.