Linear Mapping have a Memory
Linear Mappings Have a Memory I've been looking at something surprisingly simple: what happens when a linear mapping is treated explicitly as a linear associative memory. One particularly interesting observation is that when a weighted sum is trained beyond capacity, new examples don't simply erase old ones. Instead, the weight vector is continually perturbed, gradually adding noise to previously…
Linear Mappings Have a Memory
The author has been examining the behavior of linear mappings when treated as linear associative memories. One striking finding is that when a weighted sum is trained beyond capacity, new examples do not simply overwrite old ones. Instead, the weight vector is continually perturbed, leading to the gradual addition of noise to previously stored associations. While more recent examples are recalled more effectively, older examples can persist statistically for an unexpectedly long period.
Interestingly, below capacity, removing a training example may have no effect on the weight vector altogether. However, once weight decay is introduced, the mapping can shift to a different, lower-norm solution while still preserving the remaining associations. These basic observations have significant implications for initialization, weight decay, the dynamics of stochastic gradient descent (SGD), and ultimately for CCSLM (Continuous Compressive Sensing Learning Model), where the local experts can be considered as factorized associative memories.
The author has detailed the technical aspects and potential implications of these findings in the article titled "Linear Mappings have a Memory."
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
