CAREER: An Architecture-Aware Optimization Theory for Deep Learning: Non-Euclidean Descent, Structured Preconditioning, and Scale Invariance

Training modern artificial intelligence systems requires large amounts of computing time, energy, and money. Many of the optimization methods used to train neural networks are still chosen largely through trial and error because existing theory does not adequately explain why some methods work better than others on different model architectures.…