Changing production infrastructure can be scary because we don’t want to break things for customers, mistakes happen, and there can be many unknowns.In this post I’ll be discussing a recent over-the-air (OTA) software update to Rivian vehicles that went badly. It is speculative; I have no insider knowledge of Rivian’s software, systems or practices. However, I do have over 20 years experience building, operating and leading teams who build and operate scalable and reliable online systems at comp...