Cortico-subcortical multi-head self-attention as a substrate for cognitive performance
The neocortex is central to mammalian cognition, yet a computational framework that is both biologically constrained and capable of performing complex cognitive tasks remains missing. Here we show that cortico-thalamic circuits are well suited to implement multi-head self- and cross-attention, the mechanism underlying the cognitive abilities of transformer networks. We propose that layer 2/3…
We haven't written up this one. bioRxiv has the full story — the link below goes straight to it.
