AI Inference Server is an industrial Edge app, which activates the Edge devices by embedding the inference function implemented in Python. Each AI Inference Server application variant includes the following functionalities:
It enables a complete Edge inference runtime: Importing AI pipelines for execution on the device, setting up the data connections/mapping and controlling the pipeline execution, all via one user interface. The imported configuration package can be supplemented with a delta upload. In this manner, only the changes are uploaded and the time for pipeline import is significantly reduced. AI Inference Server provides a basic graphic visualization for displaying input and output data.
AI Inference Server supports importing or receiving of successor versions of a configuration package, which contains a package ID for traceability purposes. It is also possible to change the values of the parameters and perform parameter mapping.
The AI model can use data directly from various Edge connectors. This reduces the CPU utilization in the Edge device and speeds up data transmission.
AI Inference Server offers the option of parallel step execution. Parallel step execution is used to create one or more running instances from a step within the same pipeline. This reduces the overall calculation time of the entire pipeline.
AI Inference Server allows simultaneous execution of pipelines. You can execute more than one AI model with different scripts.
The AI Inference Server allows you to fine-tune the internal queue (ZMQ) so as to ensure smooth and efficient data flow from the data connectors to the "runtime family". Without this tuning, the "runtime family" may be in a situation in which the work memory is insufficient or data is lost (e.g. due to memory limitations of the queue).
AI Inference Server provides Openness API support; the integrated functions can be used via Rest-API calls. More information is provided in the user documentation for the AI Inference Server Openness API.
The standardized image transport uses the Vision Connector application, including the new image set type with metadata information (such as camera ID, timestamp, image height, image width).
AI Inference Server GPU accelerated (MLFB 6AV2170-0LA11-0AA0) brings additional benefits:
Running the AI models on real GPU hardware which significantly reduces the calculation/processing time of AI models.
On-the-fly model optimization
Supports larger model size using additional VRAM of GPU hardware.
User actions and system response
|
User action |
System response |
|---|---|---|
1 |
The data scientist creates a configuration package with AI SDK containing the AI model in ONNX format. The steps (e.g. preprocessing) to be executed on the GPU have been set in pipeline_config.yml. |
AI SDK generates a configuration package containing the ONNX model. The tuning option for TensorRT was activated by the data scientist in the pipeline_config.yml file. |
2 |
Automation Engineer imports the configuration package. |
AI Inference Server reads the configuration package and saves it on the local Edge device. After the successful import, AI Inference Server indicates the Python version in use and which step is running on the CPU and GPU. The GPU execution is only possible with the variant AI Inference Server GPU accelerated (MLFB 6AV2170-0LA11-0AA0). |
3 |
The automation engineer creates the data connections and performs the variable mapping. He checks and, if necessary, modifies the setup of the parallel execution (step name, number of the parallel execution). |
AI Inference Server saves the data connections and the specified topic information. |
4 |
The automation engineer starts the pipeline. |
AI Inference Server initializes the pipeline and checks which step is being executed on the CPU and GPU. The GPU execution is only possible with the variant AI Inference Server GPU accelerated (MLFB 6AV2170-0LA11-0AA0). |
5 |
The automation engineer checks the statistics and visualization. |
The statistics are available as usual as pipeline and step statistics. |
Mapping and executing the pipeline
The AI Inference Server app imports a configuration package file of the pipeline and checks whether the mapping of the variables and parameters is complete for each step.
The AI model (pipeline) is a code generated by the user, and it is the responsibility of the user to ensure that it is free of viruses/Trojans.
If this is not the case, the app sets the pipeline status to Warning and informs you of the step for which the mapping information must be completed.
You can enter the missing mapping information either manually or using the browse tags option. In this happens, you can see the selected items by clicking the Mapping field after the tag pool update.
The app visualizes what the pipeline looks like, including the mapping information of variables and parameters. A pipeline contains at least one step. Each step has input variables, the values of which can either come from the data bus or from other steps. Standard values of the parameters can come from the data connector. The output values are calculated.
You can initialize, start, stop and pause your mapped pipeline. You can also check the status to determine if the pipeline has already started. For more details, see the section "Executing the pipeline".