Contents Model Simplifications High-Level Architecture Virtual Weights and the Residual Stream as a Communication Channel Attention Heads are Independent and Additive Attention Heads as Information Movement The Path Expansion Trick Splitting Attention Head terms into Query-Key and Output-Value Circuits Interpretation as Skip-Trigrams Summarizing OV/QK Matrices Do We "Fully Understand" One-Layer Models? Three Kinds of Composition Path Expansion of Logits Path Expansion of Attention Scores QK Circ...