How to build a distributed queue in a single JSON file on object storage
February 12, 2026•Dan Harrison (Engineer)We recently replaced our internal indexing job queue, which notifies indexing
nodes to build and update search indexes after data is written to the
WAL. The queue is not part of the write path; it's purely
a notification system used to schedule asynchronous indexing work. The prior
version sharded queues across indexing nodes, so a slow node would block all
jobs assigned to it even if other nodes were idle. The new version uses a single
queue file on obje...
Read more at turbopuffer.com