Preprocessing prepares batch inputs for the GPU inference step. Use one of the following methods:
NumPy array
Python list
The inference step input as a NumPy array
The data scientist prepares a large NumPy array with N inputs (single array). It is transferred as a single variable to the inference step. In this case, AI Inference Server calculates the automatic batch number only.
The inference step is aware of the shape of a single input. It can retrieve it from the model and use the shape to calculate the actual batch number together with the large array.
The output handling is similar.
The inference step input as a Python list
The data scientist prepares a Python list of N inputs. It is passed to the inference step as an array/batch for variables. The inference step is aware of the shape of a single input and the batch number. It puts together the independent inputs to the array.
The output handling is similar. The response contains the batch number and AI Inference Server splits the response and provides an array for the post-processing step.
This procedure requires the following batch attributes of the pipeline_config.yml file to be set:
inputType: |
- name: input |
type: String |
name: preprocessing |
batch: |
inputBatch: No |
outputBatch: Yes |
outputType: |
- name: tensors |
... |
inputType: |
- name: input |
type: String |
name: classification |
batch: |
inputBatch: Yes |
outputBatch: Yes |
... |
inputType: |
- name: input |
type: String |
name: postprocessing |
batch: |
inputBatch: Yes |
outputBatch: No |