Back in 2017, we shared how we built our message search system to index billions of messages. We designed our search infrastructure to be performant, cost-effective, scalable, and easy to operate. We chose to use Elasticsearch, with Discord messages sharded over indices, the logical namespace for Elasticsearch messages, on two Elasticsearch clusters. Messages were sharded either by Discord server (which we’ll refer to as a guild from here on out) or direct message (DM). This allowed us to store ...