Properties of the Hessian Matrix of Neural Networks and Its Applications in Algorithm Analysis
Date:
Part I: why Transformers need Adam \u2014 from a Hessian perspective, leading to the Adam-mini algorithm. Part II: on the special (near block-diagonal) structure of the Hessian of neural networks.