Scaling Limits for Learning: Dynamics and Statics

          @misc{ scivideos_PIRSA:25040085,
            doi = {10.48660/25040085},
            url = {https://pirsa.org/25040085},
            author = {Bordelon, Blake},
            keywords = {},
            language = {en},
            title = {Scaling Limits for Learning: Dynamics and Statics},
            publisher = {Perimeter Institute for Theoretical Physics},
            year = {2025},
            month = {apr},
            note = {PIRSA:25040085 see, \url{https://scivideos.org/index.php/pirsa/25040085}}
          }
          

Abstract

In this talk, I will discuss how physics can help improve our understanding of deep learning systems and guide improvements to their scaling strategies. I will first discuss mathematical results based on mean-field techniques from statistical physics to analyze the feature learning dynamics of neural networks as well as posteriors of large Bayesian neural networks. This theory will provide insights to develop initialization and optimization schemes for neural networks that admit well defined infinite width and depth limits and behave consistently across model scales, providing practical advantages. These limits also enable a theoretical characterization of the types of learned solutions reached by deep networks, and provide a starting point to characterize generalization and neural scaling laws (see Cengiz Pehlevan's talk).