While the primary purpose of AI Model Monitor Agent is to collect and possibly send metric data to the AI Model Monitor application, it can optionally be configured to monitor incoming data and, based on the configured settings, display alerts on the Prometheus service UI or even send notifications to a variety of supported receiver services. To configure the alerting functionality of AI Model Monitor Agent, you can edit the following configuration files:
prometheus.rules.yml
alertmanager-config.yml
manager-e-mail-template.txt
Alerting mechanism is enabled by Prometheus which is a part of the AI Model Monitor Agent and requires two configuration files for alerting purposes. The first one is the prometheus.rules.yml which contains the rules used to trigger an action, and the other is the alertmanager-config.yml which defines the actions. Both templates contain predefined settings to provide appropriate examples.
Alerting rules
All the alerting rules should be defined in the```prometheus.rules.yml` file.
You can use all metrics and labels in a Prometheus Query Language (PromQL) expression to define a trigger.
The following example describes a rule for an alert triggered by an overloaded host CPU:
group:
- name: Alerts
rules:
- alert: CpuUsageHigh
expr: host_box_cpu_percentage{job="host_box"} > 80
for: 10m
labels:
severity: critical
annotations:
summary: CPU usage of the host is dangerously high
Further information on the configuration of alerts in Prometheus can be found in the Prometheus alerting rules documentation.
Alerting settings
All alerting settings except alerting rules are to be defined in the alertmanager-config.yml file.
The provided configuration sets up an SMTP connection, applies an e-mail template (asalertmanager-e-mail-template.txt) and sends e-mail notifications about alerts.
Besides e-mail alerts you can configure other notification channels to receive alerts as long as they do not require the upload of additional files, such as templates, credential, certificates.
Note:
The current version of AI Model Monitor only supports alert channels that can be fully configured in the alertmanager-config.yml file, and do not require additional file uploads to the service.
Further information about the configuration of alert manager can be found in the Prometheus configuration webpage.