Good observability of your system is key - but unfortunately it is not usually detected that this is system is weak until something goes wrong. Several years ago I witnessed a snapshot data capture had been down for several weeks, but there were no errors. There were no errors as the snapshot technically was working, but the underlying data was not refreshing. After its discovery, this taught me to not be reliant on error messages alone for a system of monitoring.
I really like the term "Observability-Driven Development".
Monitoring is important, but observability is what saves your team for being firefighting and avoiding major issues.
Good observability of your system is key - but unfortunately it is not usually detected that this is system is weak until something goes wrong. Several years ago I witnessed a snapshot data capture had been down for several weeks, but there were no errors. There were no errors as the snapshot technically was working, but the underlying data was not refreshing. After its discovery, this taught me to not be reliant on error messages alone for a system of monitoring.