NanoDO: A minimal ("nano-sized") Transformer decoder-only language model implementation in JAX. Inspired by minGPT/nanoGPT and flax/examples we provide a minimal implementation of a Transformer decoder-only language model in Jax. The purpose is to be maximally hackable, forkable, and readable for researchers, to enable highly exploratory research. Magic is great for products, but it is harmful in many cases for research and so we minimize abstraction as a design goal. Currently we use: flax for ...