Monitoring with the Alert tab

We have now come to the Alert tab, the last of the graph panel tabs. In this pane, you can configure the panel with an alert. While we are going to look at the Grafana alerting system in more detail in Chapter 9, Grafana Alerting, let's take a peek at the interface to get a feel for what it takes to create an alert (spoiler: not much!). Click on the Create Alert button to have a look inside an alert.

The following screenshot shows a newly created alert:

An alert is composed of four key components:

• Rule
• Conditions
• No Data & Error Handling
• Notifications

Let's take a look at them now.

Rule

An Alert rule can be broken down into two basic settings: the rule name and an evaluation period. The rule Name is used by Grafana to keep track of each rule so that it can continuously check each alert that's spread over every dashboard for a triggering event. As we found in the previous chapter, Alerts are managed in the Alert Rules tab, which can be found on the Alerting page. Once you have configured the Alerts, you'll find it much easier to manage them from this page, rather than clicking through many dashboards and panels.

Evaluate every is used to set the frequency at which Grafana should check for a threshold violation to trigger an Alert. Once the threshold has been crossed, the For field determines how long the threshold must be exceeded before actually triggering an Alert.

Conditions

Thresholds are set in the Conditions group. Simply put, the threshold condition is defined by an aggregation on a query that exceeds a certain value. If this sounds familiar, you will recall that it is similar to the definition of a threshold. Rather than simply triggering it on a single value, an aggregation is done over a time window, so an alert condition can represent a more qualitative measure than the simple quantitative measure of a threshold.

No Data & Error Handling

What happens if the query stops producing usable data or throws an error? That question is answered by the No Data & Error Handling settings. In this section, you'll be able to determine whether data loss or an error should trigger an alert or whether the system should wait it out. This is useful if you have a separate monitoring system for your data source servers that would otherwise trigger an alert, and you don't want to get two sets of alerts for essentially the same incident.

Notifications

Once the alert has been triggered, the Notifications group is where you determine what actions Grafana should take. Based on your configured notification channels, you can use Send to to send one or more channels the specified Message. Below the message are key/value pairs called Tags. Currently, those tags only feed Prometheus Alertmanager the additional information.

Try creating an Alert for yourself. You can use the Test rule button to check your rule conditions and see whether they fire. If you don't want to keep the Alert, go ahead and Delete it. We will cover Alerts in more detail in Chapter 9, Grafana Alerting.