1 year ago

Anyone who’s lived through the past few years knows how Twitter’s usage can spike in an instant. For these times of heightened interest, Twitter’s Search Infrastructure engineers have added a proxy, ingestion layer, and a backfill layer to the search system’s architecture in order to reduce latency in the real-time platform.

The backfill service addresses the above issue by loading large amounts of data safely and efficiently into an Elasticsearch index in three stages.

The backfill orchestrator, the brains of the backfill service, connects information from the sink to internal Twitter service environments where Elasticsearch clusters are hosted.

The backfill workers are small distributed applications that read the index requests from storage and index the data into the cluster.