Why Compute and Storage Should Be Decoupled for Log Management at Scale

4 years ago thenewstack.io

Summary: This is a summary of an article originally published by The New Stack. Read the full original article here →

Most log management solutions store log data in a database and enable search by storing an index of the data. When companies are handling terabytes of data every day, the database-backed log management system becomes untenable.

A company ingesting around one terabyte of data per day would need multiple terabytes of storage and a proportional amount of RAM if they wanted to keep 30 days worth of log data searchable.

It’s dramatically cheaper to store data in cold storage than in a database, and scaling cold storage is easier than scaling a database.

When log data is accessed directly in the cold storage, users don’t have to worry about managing indexes between hot and cold store tiers, rehydrating data, or building complex policies.

DevOps Articles