A small collection of insights on the structure of embeddings (latent spaces) produced by deep neural networks. Manifold Hypothesis: High-dimensional data sampled from natural (real-world) processes lies in a low-dimensional manifold. https://arxiv.org/abs/2208.11665 Hierarchical Organization: Features organize hierarchically across layers - earlier layers capture low-level (small context) features while deeper layers represent increasingly abstract (large context) concepts. https://colah.github...