Icinga notifications: why alerts don’t arrive—and how to find the problem
The service is red. The dashboard says CRITICAL, but the phone stays silent. Or the opposite happens: a switch fails and messages arrive every minute, without making it clear who should act. A single threshold rarely fixes either problem.
To troubleshoot Icinga notifications, follow one specific failure from the check to the recipient. Several decisions happen along the way. We will work through them in Icinga Director: from the service template to the recipient and notification rule, then deployment and recovery. That gives you a way to locate the break and verify a targeted change.
Six stages between CRITICAL and a useful message
A red service initially tells you about a check result. Whether anyone gets notified also depends on the confirmed state, rules and individual user. Only then does the notification command start. Even a successful command does not tell you whether a person has seen the message.
- 1 → Check resultWhat was measured, when, and on which node?
- 2 → SOFT / HARDHave retries confirmed the failure?
- 3 → Notification ruleDo state, type, time period and dependencies allow it?
- 4 → RecipientDo the user’s own filters and time period allow the message?
- 5 → CommandDid the script run, and what result did it return?
- 6 → DeliveryDid the transport accept it, and did the intended recipient receive it?
Find the first stage where the expected evidence is missing. If there is no matching Notification object, investigating the mail server is premature. If the command already runs, look at its exit code, the transport log and the destination address.
What Director shows—and what Icinga is running
First open the affected service in Icinga Web. Record its state, SOFT/HARD state type, last check and Check Source. You can also see downtime, acknowledgement and notifications disabled at runtime. These describe the running service.
Next, open the corresponding service or Service Apply Rule in Director. Follow inherited templates under Imports. Check Send notifications, check intervals and filters. An empty field can mean an inherited value. The resolved configuration in the preview is what matters.
Saving a form does not activate the change. Check the Activity Log for pending changes and Deployments for the version successfully activated. If validation fails, Icinga may continue with the previous configuration. That is one reason Director and the dashboard can suggest different expectations.
Director sends configuration packages through the API. Edits to generated package files would disappear at the next deployment. Icinga’s explanation of rendering and deployment covers the background.
Create a small test service in Director
A service whose state you can control makes the path easier to understand. Use an existing, reachable test host managed by Director; ours is called dexor-alert-lab. The Icinga instance needs its notification feature enabled. Review existing global notification rules first so the exercise does not reach real on-call recipients.
We use the ITL check dummy. In Director, its dummy_state variable can be changed from 0 (OK) to 2 (CRITICAL) and back. No additional check script is needed.
- Open Services → Service Templates in Director and create
tpl-alert-lab-service. - Choose Check command: dummy, enter the values below and save the template.
| Director field | Value | Purpose |
|---|---|---|
| Check interval | 60 seconds | Regular checks in healthy and confirmed states. |
| Retry interval | 20 seconds | Faster retries while a failure is unconfirmed. |
| Max check attempts | 3 | Confirm a failure before an ordinary problem notification. |
| Execute active checks | Yes | Actually run the test. |
| Send notifications | Yes | Allow notifications for this service. |
| Enable flap detection | No | For this synthetic exercise only, to keep transitions reproducible. |
If dummy_state has no input field, create one under Data fields. Use Field name: dummy_state, Caption: Test state and the numeric data type Number. Assign it to the service template under Fields. The field name is dummy_state, without vars.; the generated Icinga object will contain vars.dummy_state.
Then open the test host under Hosts and select Services → Add service. Name the service bereitschaft, import tpl-alert-lab-service and set Test state to 0 on that specific service. Make later state changes here too. This avoids accidentally changing every service that imports the same template.
Prepare the command, recipient and notification template
Start with a local log entry
For the first round, the command should write to the system log. The node executing notifications needs /usr/bin/logger and a working local syslog/journal connection.
- Open Commands and create
dexor-lab-log. - Set Command type to Notification Plugin Command. Enter
/usr/bin/loggerunder Command. - Add the following two entries in Arguments. Keep Value type set to String for both.
| Argument name | Value | Position | Skip key |
|---|---|---|---|
-t | icinga-alert-lab | 0 | No |
message | The macro line below | 1 | Yes |
$notification.type$ | $host.name$!$service.name$ | $user.name$ | $service.state$
Copy this line into the Value of the message argument. Skip key: Yes passes only the resolved text, without the word “message”. The dollar signs belong to Icinga’s runtime macros. Enter the line as a string in Director, rather than a shell command or DSL expression.
Create the user and its own filters
A Director User is a notification recipient here, rather than a login to the web interface. Under Users → User templates, first create tpl-alert-lab-user. Set Send notifications: Yes, States: OK, Warning, Critical, Unknown and Transition types: Problem, Recovery.
Create User dexor-lab-user with that template under Imports. Our local logging command needs no email address. For a real delivery channel later, supply the address or channel details that its command expects.
Leave Time period unrestricted in the new template. In a real on-call setup, check the user’s period as well as the notification’s. Allowing a channel does not override a user filter that blocks the message.
Build the notification template
Open Notifications → Templates, create tpl-alert-lab-notification and use these settings:
| Director field | Exercise value |
|---|---|
| Notification command | dexor-lab-log |
| Users | dexor-lab-user |
| States | OK, Warning, Critical, Unknown |
| Transition types | Problem, Recovery |
| Notification interval | 1800 seconds, or 30 minutes |
| First notification delay | 0 |
| Time period | Unrestricted in the new template |
| Disabled | No |
The two filtering layers are deliberate: States and Transition types are evaluated on both Notification and User. Start with a clear, simple configuration. Add time periods and escalation to match ownership afterwards.
Assign the Apply Rule and deploy the intended configuration
- Under Notifications, open Apply Rules and create
dexor-lab. - Choose Apply to: Services and import
tpl-alert-lab-notification. - Under Assign where, join two conditions with AND:
host.nameequalsdexor-alert-labandservice.nameequalsbereitschaft. Only this combination should match. - Check the configuration preview. Its assignment should have this structure:
apply Notification "dexor-lab" to Service {
import "tpl-alert-lab-notification"
assign where host.name == "dexor-alert-lab" && service.name == "bereitschaft"
}
Compare the structure with your preview; make the change in Director itself. Also check whether other rules create additional Notifications for the same service. Separate rules do not automatically form an escalation chain; they may simply send duplicates.
Before Deploy, read the pending changes in the Activity Log. The usual workflow deploys the shared configuration, rather than only the form you have open. Afterwards, check the deployment result for successful validation and activation. The new Notification should now be visible on the running service.
Wait for at least one successful OK check. This starts the exercise from a known healthy state. Icinga’s Director introduction and configuration guide provide further help with the workflow and packages.
Three attempts do not mean a three-minute wait
Suppose the first failed check finishes at 10:00:00. With three attempts and a 20-second retry interval, the sequence is approximately:
| Since first failed check | Result | What happens? |
|---|---|---|
| 0 seconds | CRITICAL, SOFT | First failed attempt; no ordinary problem notification yet. |
| 20 seconds | CRITICAL, SOFT | Second failed attempt; another check follows. |
| 40 seconds | CRITICAL, HARD | Third failed attempt; the normal notification path is evaluated. |
Those 40 seconds start at the first observed failure. If the actual outage began just after the previous successful check, add the wait until the next check. With a one-minute interval, that can be almost another 60 seconds, plus execution and scheduling time. An additional times.begin delays notification further, measured from the HARD state transition.
More retries absorb brief glitches but delay detection of a sustained failure. Shorter intervals add load to monitoring and the target system. Choose according to the service’s acceptable response time. A completed overnight import needs different timing from a public checkout. See Icinga’s explanation of state transitions.
Make every alert a clear next step.
We help investigate missing messages and alert floods, from check logic to the notification and delivery path.
Less noise starts with the action you expect
For every message, ask what the recipient should do next. High CPU usage alone rarely answers that question. A queue that is no longer being processed and is affecting orders can justify a concrete task. WARNING and CRITICAL describe technical states; operational urgency comes from the affected service.
Dependencies help suppress secondary alerts during a shared failure. Check disable_notifications and the reachability actually evaluated. A poorly modelled dependency can hide exactly the message you need. Downtime explains planned work; acknowledgement records that someone has taken ownership of an incident. Both should be visible and traceable over time.
Set Notification interval: 0 in Director to disable reminders while still allowing Problem and Recovery events. That can suit a dependable ticketing system or another escalation process. Without that handover, a single overlooked message may remain unattended.
A useful message might look like this. Replace the addresses with your own diagnostic pages and runbook:
CRITICAL · Order import · Production
Observation: No successful import for 18 minutes.
Impact: New orders are missing from the downstream system.
Owner: Application operations, following the on-call rota.
First action: Check the import log, then source connectivity.
Monitoring: https://monitoring.example.org/…
Runbook: https://wissen.example.org/…
Include incident start and last check, with time zone.
For a backup alert, include the latest run’s result and the age of the data in each copy. The team can then tell whether one run failed or the off-site copy is already too old. The Restic backup monitoring guide explains which signals to track.
“Completed sending” is not a delivery receipt
This deserves a closer look. Icinga executes notification commands asynchronously. An entry such as Completed sending therefore does not prove that the script succeeded or that a message arrived. Look for the process result and exit code as well.
We reproduced this on Icinga 2.16.6 using a local test command that deliberately returned exit code 1 for the problem event. With interval = 0, there was no automatic delivery retry while the service remained CRITICAL. When it returned to OK, the recovery command still ran.
The explanation is in the bookkeeping: the user is recorded as notified when the problem command is queued. A later command failure does not undo that decision. You can follow this in the versioned notification source and the process-result handler.
Operationally, a notification interval is not a guaranteed transport queue. If dependable retries are required, the delivery path needs to provide them, for example through a monitored local mail transport or durable queue. Verify these separately: command started, exit code, transport acceptance and arrival at the intended recipient. Even exit code 0 proves only what that particular script defines as success.
The script runs on the node executing the Notification object. A service check’s command_endpoint does not automatically move notifications to the agent. On the executing node, check the service account, file permissions, program paths, DNS and outbound connectivity. A successful manual command as root can conceal those differences.
Start with a narrow log window around the state transition. On systemd, try journalctl -u icinga2 --since '10 minutes ago'. Depending on logging configuration, details may also be in /var/log/icinga2/icinga2.log. Search for the full object and user names. If you temporarily enable debug logging, keep the investigation bounded and turn it off afterwards.
Why the recovery message may be missing
Open the Notification template in Director, then the User template. Is Recovery allowed under Transition types in both? Next check time periods, active suppression and the preceding problem notification. In the normal Problem/Recovery flow, previously selected problem recipients receive the recovery. A rule explicitly configured only for recoveries is a special case.
There is a version-specific detail around states. The current documentation recommends OK for recoveries. Our 2.16.6 test delivered recovery without OK in either state filter: the code exempts Recovery from that state check. We keep OK in the readable example, but would not consider missing recovery fixed simply by adding that value. Check the actual sequence on your version.
Turn the symptom into a focused check
| Symptom | Look at | Targeted change |
|---|---|---|
| Red service, no Notification object | Director: Apply Rule, Imports and last successful activation. | Correct Assign where, review the preview and deploy again. |
| Object exists, sending never starts | Runtime: SOFT/HARD and suppression. Director: Send notifications, User and Notification filters. | Correct the specific condition blocking the message. |
| Unexpectedly long delay | Last check, retries, execution time and times.begin. | Adjust the relevant part of the delay to the response requirement. |
| Many messages for one outage | Multiple rules, interval, state changes and a shared dependency. | Remove duplicate assignments or deliberately design dependencies and reminders. |
| Silence during an incident | Downtime, acknowledgement, flapping and reachability. | Record intended suppression; correct unintended suppression. |
| “Sent”, but nothing in the inbox | Command exit code, transport acceptance, recipient and spam/chat filters. | Repair the failed delivery stage and repeat the test. |
| No recovery message | Recovery type on both layers, previous recipients, periods and command errors. | Verify with a fresh, complete Problem/Recovery cycle. |
Walk through failure and recovery in Director
The test service is OK and its Notification is active. Now change the state on that specific service:
- Open Hosts → dexor-alert-lab → Services → bereitschaft. Change Test state from
0to2, save and deploy the reviewed configuration. - In Icinga Web, follow the failed checks until HARD. The local system log should contain
PROBLEM, the host, service and user. - Set the same service back to
0in Director and deploy again. After the next successful check, expectRECOVERY. - Then schedule downtime and repeat the state change during that maintenance window. Ordinary problem notifications should remain suppressed.
For a brief transient failure, temporarily increase Retry interval to 60 seconds in the test template. That gives you more time to set the state back to 0 and deploy after the first SOFT failure. If the check returns to OK before HARD, expect no ordinary Problem/Recovery notification. Restore the retry interval to 20 seconds afterwards.
You may need someone with access to the executing Icinga node to inspect its logs. On a systemd installation:
sudo journalctl -t icinga-alert-lab --since '10 minutes ago'
sudo journalctl -u icinga2 --since '10 minutes ago'
Director shows configuration and deployments; exit codes and actual transport belong to runtime diagnosis. For a real delivery test, copy the test template, select your installation’s existing mail or chat command, and use exactly one agreed test recipient. Check the problem message, text, links and recovery. A forced custom notification can test transport, but does not replace the normal state-transition path.
- The active configuration matches the intended Director version.
- Exactly the expected rules and recipients match.
- The failure is confirmed within the required time.
- Command and transport produce traceable results.
- Problem and recovery reach the agreed recipient.
If a step fails, return to the first stage with missing evidence. That turns a red dashboard into a focused investigation. We can help through Monitoring & Icinga and practical Icinga training.