SRE Blog

Better living through reliability.

Proactive vs Reactive Alerts

2020-08-03

Every alert falls along a continuum of reactive vs proactive. And every service probably has a mix of each.

Reactive alerts mean that every failure is a fire drill to get things working again. Fire fighting constantly means the team is doing little else and will have a hard time focusing on higher-value projects.

Proactive alerts notify before issues become fire drills. Getting out of fire fighting mode means devoting team time to understanding failure modes, capacity planning, etc. Proactive alerts are a tool to give the team time to address issues before things break.

For example, a "disk usage too high" alert for your database can open a ticket well in advance of the disk filling up — preventing middle of the night pages.

These types of alerts also serve double duty to help with capacity planning. Ideally, the team is well versed in capacity planning and might be done every quarter. If so, that's great and proactive alerts will never fire! But if there isn't much time devoted to capacity planning, proactive alerts will help remind you.