Configure escalation chains
Escalation chains define the sequence of actions taken when an alert is triggered in Grafana IRM. They automate your incident response workflow by executing ordered steps until an alert is acknowledged, resolved, or all steps complete.
An effective escalation chain:
- Ensures alerts reach the right people at the right time
- Implements tiered response procedures based on severity
- Automates notification and escalation processes
- Prevents alerts from being missed
Create and manage escalation chains
Create a new escalation chain
- Navigate to IRM > Escalation Chains in the Grafana Cloud menu
- Click New escalation chain
- Enter a unique name and optional team assignment
- Click Add escalation step to add steps to your chain
- Configure steps and arrange them using drag-and-drop
- Click Save
Edit or delete an escalation chain
- To edit: Select a chain and click Edit, then make changes and save
- To delete: Select a chain, click Delete, and confirm
Note
Before deleting, check the Linked integrations and routes panel. Changes to the chain affect all associated integrations and routes.
Manage escalation chains with Terraform
You can use the Grafana Terraform provider to define and manage your escalation chains as code. This enables version control and easier reuse of your on-call workflows.
Notify a specific user, user group, or schedule
To notify individual responders or teams (such as a person, user group, or on-call schedule),
use the corresponding step type in the
grafana_oncall_escalation
resource.
Then define the object to notify using the associated *_to_notify
parameters:
notify_persons
+persons_to_notify
: Notifies specific usersnotify_persons_next_each_time
+persons_to_notify_next_each_time
: Round robin notificationnotify_on_call_from_schedule
+notify_on_call_from_schedule
: Notifies the currently on-call person from a schedulenotify_user_group
+group_to_notify
: Notifies a defined Slack user group
Notify a channel
To post alert groups in a Slack, MS Teams, or Telegram channel, you must define the channel in the grafana_oncall_route
resource.
Then, in the grafana_oncall_escalation
resource, use the notify_whole_channel
step to send a message to that channel when the escalation chain is triggered. Please note that this step is only available for Slack channels and that it will trigger a notification for all users in the channel, so it should be used with care.
To notify an entire Slack channel:
- Use the
slack_channel_id
in yourgrafana_oncall_route
resource. - Use the
notify_whole_channel = true
step type in yourgrafana_oncall_escalation
resource.
Note
The route defines where the alert group is delivered, and the escalation controls when and whether the whole channel is notified. You can’t post an alert group to one channel and notify a different one.
For more detailed set up instructions, refer to Manage Grafana IRM in Grafana Cloud using Terraform.
Types of escalation steps
Notification steps
- Notify users: Send notifications to specific users or groups
- Notify from on-call schedule: Alert currently on-call users
- Notify all team members: Alert everyone in a specified team
- Notify Slack channel/user group: Send notifications to Slack users
- Round robin notifications: Rotate through a list of users sequentially
Timing and control steps
- Wait: Pause for a specified duration before proceeding to the next escalation step
- Repeat escalation: Loop the escalation steps up to five times
- Time-based escalation: Continue only during specified time periods
- Threshold-based escalation: Escalate only after a threshold of alerts
Action steps
- Resolve incident automatically: Mark the alert group as “Resolved automatically” without user intervention
- Trigger outgoing webhook: Send data to an external system using a configured outgoing webhook
- Declare incident: Create a new incident with specified severity. Limited to one incident per route at a time; additional alerts are grouped into the active incident
Notification types
When configuring escalation steps, you can specify which set of notification rules to use:
- Default notifications: Standard alert notifications as configured in the user’s profile
- Important notifications: High-priority notifications that use more aggressive channels or frequencies
Each user can customize their notification rules in the IRM tab of their user.
Example escalation chains
Basic notification chain
- Notify primary on-call person (important)
- Wait 5 minutes
- Notify primary again (important)
- Wait 10 minutes
- Notify backup on-call person (important)
Business hours vs. after hours
- Time-based escalation (continue if 9 AM - 5 PM, Monday-Friday)
- If true: Notify business hours team
- If false: Proceed to step 2
- Notify 24/7 on-call engineer (important)
Critical system
- Notify primary on-call person (important)
- Notify Slack channel #critical-alerts
- Wait 5 minutes
- Notify backup on-call person (important)
- Wait 2 minutes
- Notify all team members (important)
- Wait 2 minutes
- Declare incident with severity “critical” if not acknowledged
Best practices
- Start simple: Begin with basic notification steps before adding complexity
- Test thoroughly: Verify chains with non-production alerts first
- Document your chains: Maintain explanations of each chain’s purpose
- Include wait steps: Add appropriate delays between notifications
- Use important notifications sparingly: Reserve for truly critical alerts
- Consider time zones: Create chains respecting global team distribution
Next steps
- Configure alert routes to direct alerts to your chains
- Configure alert templates to customize alert appearance
- Set up on-call schedules to automate rotation