IntroI recently undertook the delusional project of writing my own implementation of a Kafka Broker: MonKafka. Deep into that rabbit hole, I fell into a different one when trying to implement compression for Kafka’s record batches. Kafka supports, as of now, four schemes: GZIP, Snappy, LZ4, and ZSTD. While proceeding with my implementation, I realized I really didn’t know that much about the fascinating topic of compression. I vaguely remembered Huffman trees and some information theory from sch...