Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methods
Adaptive optimization methods such as AdaGrad and Adam are widely used in modern neural-network training, but their adaptive scaling is primarily designed for vector-valued parameters and does not explicitly exploit matrix structure. Recent matrix-aware optimizers demonstrate the benefits of structured optimization, yet a general theoretical framework for deriving matrix-aware adaptivity…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
