Data acquisition from the IE Vision Connector

IE Vision Connector with low data throughput (Databus-based, string)

Only an earlier version of the IE Vision Connector (p 0.9.5) supports sending images to the Databus. This payload format in the AI Inference Server is only retained for reasons of backward compatibility. Newer versions of IE Vision Connector (≥ v 1.0.0) only support data transfer with a high data throughput.

• Pipeline configuration

  • The type String should be chosen for the relevant variable.

• Like with the other connectors, this connector should be selected/created in the Data connection section of the configuration page.

• The preset port is the same as for the Databus: 1883

• The Topic-Variable-Mapping is the same as in the case of the Databus.

• The payload structure with use of the IE Vision Connector via Databus looks like this:

The Python script should take care of handling this payload format.

Example script, assuming the vision_payload parameter contains the above payload JSON object, get_image_from_vision_payload returns the image object as PIL image:

IE Vision Connector with high data throughput (ZMQ-based, multi-part)

• For performance reasons, the Vision Connector can publish the images in binary format via the ZMQ protocol

• Pipeline configuration

• The AI expert should select the type (ImageSet or Object (legacy type)) for the variable of the model. ImageSet is preferred.

• The connector configuration is shown in the following screenshot. The default port is 5555

Example of a topic-variable mapping:

The Python script receives this variable as dict, which is structured differently for the ImageSet and Object input types.

The data flow from the camera to the Python script looks like this:

For the ImageSet type, the dict variable is structured as follows:

• The "ImageSet" input type in the pipeline configuration is only supported with the newly introduced "process_input" interface.

• The data always contains the "timestamp" field as string in ISO 8601 format. For the ImageSet type, this is identical to the "timestamp" within the "data" input.

• The data contains the variable from the pipeline package if it was set to "image" in this example.

• The types of the various elements in the image are as follows:

  • version – String – Version of the message format. Latest: "1"

  • cameraid – String – ID of the camera device

  • timestamp – String - in ISO 8601 format with millisecond accuracy, e.g. "2022-01-31T20:09:28.987Z"

  • customfields – String - JSON containing additional information added to the image message

  • detail – array<Image Detail> - Array field consisting of N elements. The content includes necessary information for the image processing

  Image Detail

  • – String – Image format specified in the GenICam standard, see: https://www.emva.org/wp-content/uploads/GenICam_SFNC_v2_7.pdf, e.g. RGB8, BGR8, Mono8. The most likely values might be: "Genicam", "Compressed".

  • width – Integer – Image width Mandatory for formats that require a width (e.g. RGB8 and all Genicam formats).

  • heigth – Integer – Image height Mandatory for formats that require a height (e.g. RGB8 and all Genicam formats).

  • metadata – String – Any JSON data about the image

  • linepadding – optional integer – is provided for some Genicam formats. Although optional, it is set to 0 by the VCA.

  • image – Binary – Binary representation of the image

• The so-called multi-part ZMQ binary format looks like this in Python syntax:

For the legacy type Object, the dict variable is structured as follows:

• The Object and ImageSet input types in the pipeline configuration are only supported with the newly introduced interface process_input.

• The data contains the variable from the pipeline package if it was set to image in this example.

• The data also includes the time stamp of the image

  • timestamp: String in ISO8601 format with millisecond accuracy, e.g. "2022-01-31T20:09:28.987Z"

• The types of the various elements in the image are as follows:

  • mimeType: String – always "image/raw"

  • resolutionWidth, resolutionHeight: Integer – example: 640 or 480

  • dataType: String – always "uint8" – 1 byte per color channel

  • channelsPerPixel – Always 3, and the color order is always BGR

  • image – bytes() – contains the image, each pixel is defined by 3 bytes

• The so-called multi-part ZMQ binary format looks like this in Python syntax:

• The mapping between the multi-part binary file and elements of the dict for the above image variable is as follows in the input:

Notes on the multi-part structure

• Multiple resolutions, data types and image files are allowed in a single message. The AI Inference Server splits them and the Python script receives them individually.

• The AI model should validate the input as best practice. The AI Inference Server converts the ZMQ format into the "Python" format, but cannot and does not check the contents. For example, the fixed content of the mimeType is a contract that is provided by the VCA provider and not by the AI Inference Server.