Temporal Rank in Recurrent Neural Networks
Recurrent Neural Networks (RNNs) are a natural framework for temporal modelling, yet are often dismissed due to their sequential, non-parallelisable computation and difficulty in capturing long-range dependencies, typically attributed to vanishing and exploding gradients. We identify a distinct and under-explored limitation: an information bottleneck induced by rank-constrained hidden state…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.