Study of 1,100+ AI models reveals neural networks converge to shared low-dimensional subspaces regardless of task or training; findings could enable more efficient, reusable models with lower carbon footprint.

The Universal Weight Subspace Hypothesis

View PDF HTML (experimental) Abstract:We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we ...