Source: thenewstack.io
Why Compute and Storage Should Be Decoupled for Log Management at ScaleCategory: Business, Database, Data
Most log management solutions store log data in a database and enable search by storing an index of the data. When companies are handling terabytes of data every day, the database-backed log management system becomes untenable.
A company ingesting around one terabyte of data per day would need multiple terabytes of storage and a proportional amount of RAM if they wanted to keep 30 days worth of log data searchable.
It’s dramatically cheaper to store data in cold storage than in a database, and scaling cold storage is easier than scaling a database.
When log data is accessed directly in the cold storage, users don’t have to worry about managing indexes between hot and cold store tiers, rehydrating data, or building complex policies.
A company ingesting around one terabyte of data per day would need multiple terabytes of storage and a proportional amount of RAM if they wanted to keep 30 days worth of log data searchable.
It’s dramatically cheaper to store data in cold storage than in a database, and scaling cold storage is easier than scaling a database.
When log data is accessed directly in the cold storage, users don’t have to worry about managing indexes between hot and cold store tiers, rehydrating data, or building complex policies.
Related Articles
Community Partners
DevOps Careers