Benchmark results presented here are only for reference and will be extended to additional types of Industrial Edge Devices and reference models for future versions.
Vision performance
Test scenario: Low complexity use case
Use case characteristics:
The AI model only tells what is on the whole input image (image classification).
Low input resolution: 224 x 224 ≈ 0,05 Mpx.
Images are evaluated and post processed individually.
Examples:
Identify what type of product is on the image.
Identify if the product on the image is okay on large scale or not.
Basic performance setup:
Single camera
1 Gigabit network
CPU based device
High performance setup:
n.a.
Test scenario: Medium complexity use case
Use case characteristics:
The AI model identifies multiple objects on the input image (object detection).
Medium input resolution: 640 x 640 ≈ 0,4 Mpx.
Images taken at the same time might be processed together.
Examples:
Identify smaller defects on an object.
Identify faulty objects arriving in multiple rows on a conveyor belt.
Basic performance setup
Single camera
1 Gigabit network
CPU based device
High performance setup
Multiple cameras
10 Gigabit network
GPU based device
Test scenario: High complexity use case
Use case characteristics:
The AI model identifies multiple objects on the input image (object detection).
High input resolution: 2448 x 2024 ≈ 5 Mpx.
Images taken at the same time are processed together.
Examples:
Identify very small defects on a large object, e.g. a car body part.
Basic performance setup
n.a.
High performance setup
Multiple cameras
10 Gigabit network
GPU based device
| Test scenario / Use case | Image resolution | Color format | Model | Hardware setup | Thoughput | Latency | CPU and GPU load (%) |
|---|---|---|---|---|---|---|---|
| Low complexity | 224x224 ~ 0,05 MPx | RG8 | MobileNet with 224x224 input | 1-8 Basler ace 2 cameras, 10 Gigabit network, IPC BX-59A with NVIDIA L4 GPU | 160 FPS overall ≈ 9600 PPM | 6 ms avg / 34 ms max pipeline latency | < 20% |
| Medium complexity | 640x640 ~ 0,4 MPx | RG8 | YOLO v7 tiny with 640x640 input | 1-8 Basler ace 2 cameras, 10 Gigabit network, IPC BX-59A with NVIDIA L4 GPU | 78 FPS overall ≈ 4800 PPM | 13 ms avg / 43 ms max pipeline latency | < 20% |
| High complexity | 2560x1920 ~ 5 MPx | RG8 | upscaled object detection with 2560x1920 input | 1-8 Basler ace 2 cameras, 10 Gigabit network, IPC BX-59A with NVIDIA L4 GPU | 4 FPS overall ≈ 300 PPM | 500 ms avg / 515 ms max pipeline latency | < 20% CPU, <15% GPU |
Notes: Every use case is individual. The benchmarks were chosen to represent an average. Tuning performance for a given use case means finding a viable balance between prediction speed and accuracy. Tuning levers include but are not limited to:
Hardware used
Image and model resolution
Choice of model architecture
Pre- and postprocessing code
Selection of training data
Throughput and load figures include:
Siemens Vision Connector receiving images from a real camera
AI Inference Server running the images through the given AI pipeline
Basic pre- and postprocessing in the pipeline.
Base load of Industrial Edge device
Extra parameters for the test execution:
Image visualization turned off