One of the main points of Observability is monitoring: receiving an alert when something goes down allows you to react timely and keep downtimes to a minimum. However, increasing the amount of notifications and alerts doesnât equate to better awareness: in fact, thereâs a point where too many notifications start becoming counterproductive.
This concept is known as notification overload (or fatigue), and itâs far from being exclusive to Observability. Think about your daily phone usage:
- How many times do you receive a group chat message and just swipe the notification away without reading it?
- How about when you receive work emails outside your working hours?
- What about all the apps constantly trying to grab your attention with push notifications?
Information overload in general is a fascinating topic, and if youâre curious to learn more about I can recommend the paper from Arnold et al1, which goes through the topic in greater detail.
Is this a real problem? đ«
This phenomenon is ubiquitous to multiple environments, but it has a common root: if our brain receives too many notifications, it loses sensitivity to them. Not only this can make you feel stressed and uneasy, but it can also draw your attention away from the few notifications that really matter.
In IT, itâs pretty common to find situations that are unusual, but possibly not critical. Say you have some kind of database:
- you want to make sure the database is always healthy, so you decide to alert (among other things) when the database load is too high;
- however, your users routinely perform resource-intensive queries, producing a temporary load spike;
- you keep receiving those alerts on your phone, but you swipe them away: after all, you know itâs just business as usual;
- after some time, an actual problem happens; you swipe it away without checking, and boom! đŁ
Even though this is a made-up scenario just to illustrate the concept, the issue is very real and, as mentioned before, transcends multiple fields.
Recently, a study by Rozenes et al.2 has been conducted on patients in Intensive Care Units, which are monitored on lots of parameters: according to the paper, the both the amount and inaccuracy of these alerts can lead to alert fatigue, âresulting in errors and inaccurate diagnosesâ. Addressing the issue by reducing the alerts and improving their validity is shown to âsignificantly reduce alarm fatigue, thereby improving the quality of care for ICU patientsâ.
Whether notification overload happens in your daily life, when monitoring your deployments, or anywhere else, itâs definitely a problem worth addressing. But how?
What can I do about it? đ§
For your personal life, my advice is the following: the next time youâre about to ignore a notification, stop for a second. Ask yourself, âis this something I often skip over without looking?â, and if the answer is yes, disable that notification. Even though you might think âIâll just ignore it, whatâs the big dealâ, itâll build up into making you feel overwhelmed. You have plenty of options:
- The pesky group chat spam? đ Mute the group and check it out manually from time to time.
- Some social media compelling you to check out the latest content you absolutely cannot miss? đ Go into the app and disable those notifications.
Thereâs even a number of applications that will âgroup upâ notifications and deliver them to you in batches, on configurable intervals. Try that out and see how it works for you!
In Observability, you should make sure youâre not alerting on non-critical issues just because you can. You should re-evaluate whatâs important, and only keep alerts that actually require you to take action!
Footnotes
-
Arnold M, Goldschmitt M, Rigotti T. Dealing with information overload: a comprehensive review. Front Psychol. 2023 Jun 21;14:1122200. doi: 10.3389/fpsyg.2023.1122200. PMID: 37416535; PMCID: PMC10322198. â©
-
Rozenes, S., Fux, A., Kagan, I. et al. Alert-Grouping: Smart Personalization of Monitoring System Thresholds to Help Healthcare Teams Struggle with Alarm Fatigue in Intensive Care. J Med Syst 47, 113 (2023). https://doi.org/10.1007/s10916-023-02010-6 â©