Concepts

AI Model Monitor application provides several metrics to monitor. To make it easier for you to differentiate between the metrics, a specific naming convention has been applied.

Metric names beginning with these prefixes come from the edge device hosting the AI Model Monitor Agent:

• host_box_*: Metrics describing a monitored Edge device. These metrics provide information about the hardware usage on the device, the status of installed applications and the device's availability.

• pipeline_*: Metrics describing data related to AI Inference Server on the monitored Edge device. These metrics provide information about the running pipeline and statistics about the nodes composing the pipeline. The metrics are extended with labels identifying the running pipeline where it is originated from.

• pipeline_\StepName\_gpuruntime: Metrics that describe the inference statistics of the GPU runtime by AI Inference Server. These metrics provide information about the GPU-specific model of a pipeline.

Metrics without these prefixes are either collected from pipeline's Monitoring Node extension or they are predefined by the Prometheus component of the AI Model Monitor application.

Note:

Metric names should consist only of Prometheus-acceptable characters, otherwise they will be skipped. In addition to valid Prometheus characters (ASCII letters, digits, underscores, and colons), AI Model Monitor Agent replaces spaces in a string with underscores, but other special characters remain unchanged by the application.

Metrics collected from Edge devices hosting AI Model Monitor Agent are extended with specific labels describing their origin.

• device_name label: A label representing the origin of the given metric. It contains the name of the device hosting AI Model Monitor Agent.

Metrics collected from AI Inference Server are extended with labels describing the active pipeline they are originated from. These are:

• pipeline_name: The name of the active pipeline.

• pipeline_version: The version of the active pipeline.

• pipeline_id: The ID of the active pipeline. In case of older packages where id cannot be acquired, the value is replaced with "N/A" plain text.

The batch metrics collected by the AI Inference Server from the GPU runtime are extended by a specific label that describes the exact size of the batch.

• batch_size: A label that indicates the exact size of the batches.

For further information on metric labeling refer to the official documentation of Prometheus.

Metrics collected from Edge devices can be placed in two different categories based on their data type:

• gauge: Values are represented by one metric describing the data.

• counter: Values are represented by two metrics in Prometheus. The first one that describes the starting date of the counter, and the other one describes the value of the counter with a strictly monotone increasing function.

Most metrics are of the gauge type. However, Prometheus can store other metrics as well. For more information on metric types refer to the Metric types of Prometheus.

AI Model Monitor represents metrics collected from various sources.

• Edge device: The metrics from this source represent the usage of hardware resources on the device and the availability of the device, as well as the number and status of installed applications.

• AI Inference Server: The metrics from this source represent pipeline-related data such as the number of node inputs/outputs, execution statistics as well as custom metrics added to the model.

• Monitoring Node: The metrics from this source are metrics collected from the models with Monitoring Node Extension. At the time of recording these metrics were provided by the Monitoring Node Extension. Labels are added to the metrics that identify the running pipeline from which they originate. To get the current lists of available metrics, please refer to the corresponding documentation.

• AI Model Monitor Agent: The metrics from this source are generated within AI Model Monitor Agent.

• Prometheus: These metrics are present in the Prometheus component of the AI Model Monitor application, and they are provided by default.

AI Model Monitor Agent provides its own error handling and monitoring metric. If any error occurs during data collection (called "scraping" in Prometheus terms), the number of occurrences will be represented as a counter type metric and the type of error will be presented as a label.

• error_type: The type of the error that occurred during the scraping phase.

• endpoint_url: The scraped endpoint URL where the error occurred.

Because Error Metric is a counter type metric, it is represented by two metrics. Prometheus represents counter type data with two metrics by default:

• error_metric_total: The number of occurrences of given error.

• error_metric_created: The starting time of the counter in Unix time format.