3 years ago

Category: Database, Security, Data, artificial-intelligence

The mantras of “Measure anything, measure everything” and “sometimes we monitor stuff that doesn’t move just in case it decides to make a run for it” became the standards for finding solutions to replace ageing Nagios installations and better ways to manage logs without paying thousands of pounds a month in licensing fees in exchange for analysing relatively small amounts of data. AWStats (nownearly 20 years old!) was all the rage for analysing your Apache web server logs to see what site visitors had been up to, and monitoring alerts generally fell into three categories: For a good few years this approach had worked on platforms hosted in physical data centres such as shared hosting clusters with thousands of websites, however as “the cloud” gained popularity it had become clear that a seismic shift was required in the way we observed our platforms and how our customers interacted with them.

This is based heavily on my experience of layering monitoring tools and solutions on top of each other, but sets it out in an easy to follow way, demonstrating that you don’t need a “big bang” monitoring solution, and that you can easily add to it as your team or organisation grows.

As the river reaches the end of its journey and we drift out onto our data lake, we start to look at visualisation — how do you ensure that you present your data to your customers (both internal and external) in a way that is understandable and constructive?

Now that we understand the flow of our monitoring, we’ll be looking at each part of the river in more depth in future posts, so join me as we set sail (Sorry again! Ed).