Researchers Unveil Modern Hopfield Network: Exponential Storage, One-Update Retrieval, Equivalent to Transformer Attention; Improves State-of-the-Art in Multiple Domains

Hopfield Networks is All You Need

Authors:Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter View PDF Abstract:We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, re...