- Physical Review Journals
All JournalsPhysics Magazine

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Soft mode in the dynamics of over-realizable online learning for soft committee machines

Frederieke Richert1,*, Roman Worschech2,1,*, and Bernd Rosenow1

  • 1Institut für Theoretische Physik, Universität Leipzig, Brüderstrasse 16, 04103 Leipzig, Germany
  • 2Max Planck Institute for Mathematics in the Sciences, D-04103 Leipzig, Germany

  • *These authors contributed equally to this work.
PDF

Phys. Rev. E 105, L052302 – Published 17 May, 2022

DOI: https://doi.org/10.1103/PhysRevE.105.L052302

Abstract

Overparametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of overparametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For online learning of a two-layer soft committee machine in the over-realizable case, we present evidence that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled and if the numbers of student and teacher hidden units are commensurate.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (39)

  1. Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature (London) 521, 436 (2015).
  2. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning (MIT Press, Cambridge, 2016).
  3. A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, in Advances in Neural Information Processing Systems, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Curran Associates, Red Hook, NY, 2012), Vol. 25, p. 1097.
  4. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, Mastering the game of Go without human knowledge, Nature (London) 550, 354 (2017).
  5. G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborova, Machine learning and the physical sciences, Rev. Mod. Phys. 91, 045002 (2019).
  6. D. O. Hebb, The Organization of Behavior: A Neuropsychological Theory (John Wiley and Sons, Inc., New York, 1949).
  7. John A. Hertz, Introduction to the Theory of Neural Computation (Addison-Wesley, Reading MA, 1991).
  8. T. L. H. Watkin, A. Rau, and M. Biehl, The statistical mechanics of learning a rule, Rev. Mod. Phys. 65, 499 (1993).
  9. D. Saad, On-Line Learning in Neural Networks (Cambridge University Press, New York, 1998).
  10. A. Engel and C. van den Broeck, The Statistical Mechanics of Learning (Cambridge University Press, Cambridge, UK, 2001).
  11. Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical mechanics of deep learning, Annu. Rev. Condens. Matter Phys. 11, 501 (2020).
  12. E. Gardner, The space of interactions in neural network models, J. Phys. A: Math. Gen. 21, 257 (1988).
  13. G. Gyorgyi and N. Tishby, Statistical Theory of Learning a Rule, in Workshop on Neural Networks and Spin Glasses, edited by K. Theumann and W. K. Koeberle (World Scientific, Singapore, 1990), pp. 3–36.
  14. M. S. Seung, H. Sompolinsky, and N. Tishby, Statistical mechanics of learning from examples, Phys. Rev. A 45, 6056 (1992).
  15. H. Schwarze and J. Hertz, Generalization in fully connected committee machines, Europhys. Lett. 21, 785 (1993).
  16. M. Opper, Learning and Generalization in a Two-Layer Neural Network: The Role of the Vapnik-Chervonenkis Dimension, Phys. Rev. Lett. 72, 2113 (1994).
  17. M. Biehl and H. Schwarze, Learning by online gradient descent, J. Phys. A: Math. Gen. 28, 643 (1995).
  18. P. Riegler and M. Biehl, On-line backpropagation in two-layered neural networks, J. Phys. A: Math. Gen. 28, L507 (1995).
  19. D. Saad and S. A. Solla, Exact Solution for On-Line Learning in Multilayer Neural Networks, Phys. Rev. Lett. 74, 4337 (1995).
  20. G. Reents and R. Urbanczik, Self-Averaging and On-Line Learning, Phys. Rev. Lett. 80, 5445 (1998).
  21. S. Goldt, M. Advani, A. M. Sace, F. Krzakala, and L. Zdeborová, Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup, Advances in Neural Information Processing Systems (Curran Associates, Inc., 2019), p. 6981.
  22. M. Biehl, P. Riegler, and C. Wöhler, Transient dynamics of on-line learning in two-layered neural networks, J. Phys. A: Math. Gen. 29, 4769 (1996).
  23. M. Biehl, E. Schlösser, and M. Ahr, Phase transitions in soft-committee machines, Europhys. Lett. 44, 261 (1998).
  24. D. Saad and S. A. Solla, On-line learning in soft committee machines, Phys. Rev. E 52, 4225 (1995).
  25. Y. Yoshida and M. Okada, Data-dependence of plateau phenomenon in learning with neural network—Statistical mechanical analysis, J. Stat. Mech. (2020) 124013.
  26. S. Goldt, M. Mezard, F. Krzakala, and L. Zdeborova, Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model, Phys. Rev. X 10, 041044 (2020).
  27. S. Mei, A. Montanari, and P. Nguyen, A mean field view of the landscape of two-layer neural networks, Proc. Natl. Acad. Sci. USA 115, E7665 (2018).
  28. L. Chizat and F. Bach, On the Global Convergence of Gradient Descent for Over-Parameterized Models Using Optimal Transport, in Advances in Neural Information Processing Systems (Curran Associates, Inc., 2018), Vol. 31, pp. 3040–3050.
  29. E. Oostwal, M. Straat, and M. Biehl, Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation, Physica A 564, 125517 (2021).
  30. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, Understanding deep learning requires rethinking generalization, in Proceedings of International Conference on Learning Representations (Association for Computing Machinery, 2017).
  31. A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and Generalization in Neural Networks, in NIPS '18: Proceedings of the 32nd International Conference on Neural Information Processing Systems (Curran Associates, Inc., 2018), p. 8580.
  32. Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang, On exact computation with an infinitely wide neural net, Advances in Neural Information Processing Systems (Curran Associates, Inc., 2019), Vol. 32.
  33. M. Belkin, D. Hsu, S. Ma, and S. Mandal, Reconciling modern machine-learning practice and the classical bias-variance trade-off, Proc. Natl. Acad. Sci. USA 116, 15849 (2019).
  34. Y. Cao and Q. Gu, Generalization Error Bounds of Gradient Descent for Learning Over-Parameterized Deep ReLU Networks, in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI Press, Palo Alto, California USA, 2020), Vol. 34, p. 3349.
  35. S. Arora, S. Du, W. Hu, Z. Li, and R. Wang, Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks, in Proceedings of the 36th International Conference on Machine Learning (PMLR) (Proceedings of Machine Learning Research (PMLR), 2019), Vol. 97, p. 322.
  36. Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang, A Generalized Neural Tangent Kernel Analysis for Two-Layer Neural Networks, Advances in Neural Information Processing Systems (Curran Associates, Inc., 2020), Vol 33, pp. 13363–13373.
  37. O. Cohen, O. Malka, and Z. Ringel, Learning curves for overparametrized deep neural networks: A field theory perspective, Phys. Rev. Research 3, 023034 (2021).
  38. G. Naveh, O. B. David, H. Sompolinsky, and Z. Ringel, Predicting the outputs of finite deep neural networks trained with noisy gradients, Phys. Rev. E 104, 064301 (2021).
  39. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevE.105.L052302 for more details on the differential equations and further numerical results.

Outline

Information

Phys. Rev. E 105, L052302– Published 17 May, 2022
  • Received 8 July 2021
  • Revised 8 March 2022
  • Accepted 4 April 2022
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI. Open access publication funded by the Max Planck Society.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation