Associative memory
A grid of 100 neurons, each ON (+1) or OFF (−1). Store a few patterns with the Hebbian rule, corrupt one, and hit Recall: neurons take a weighted vote from all the others, in random order, until the pattern settles into the nearest stored memory — its energy sliding downhill the whole way.
Click a thumbnail to store or remove it.
Energy per update pass — it can only go down or stay flat (Lyapunov).
The 0.138N wall
Every stored memory is superimposed into one weight matrix, so they interfere. The classical network retrieves cleanly only up to P ≈ 0.138·N memories — then it collapses, sharply, like a phase transition. The modern continuous network keeps every memory separate and barely notices.
Monte-Carlo over fresh random patterns. Success = overlap with the target after retrieval. The dashed line marks the classical limit α = 0.138.
- Classical binary
- Modern continuous
Modern continuous — and it's attention
The modern network (Ramsauer et al., 2020) retrieves in a single step: score the query against every memory, take a softmax (sharpened by inverse temperature β), and return the weighted average — mostly just the winner. That update is the transformer's attention, and softmax is the Boltzmann distribution. Slide β to cool the system onto one memory.
High β → the distribution condenses onto the nearest memory (retrieval). Low β → all memories blend (a fuzzy average).