- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Modular Connectivity in Neural Networks Emerges from Poisson Noise-Motivated Regularization and Promotes Robustness and Compositional Generalization

Daoyuan Qian1,2,3, Qiyao Liang1,2,4, and Ila Fiete1,2,3,*

  • *Contact author: fiete@mit.edu
PDF

PRX Life 4, 033022 – Published 1 September, 2026

DOI: https://doi.org/10.1103/8bps-t2vq

Abstract

Circuits in the brain commonly exhibit modular architectures that factorize complex tasks, resulting in the ability to compositionally generalize and reduce catastrophic forgetting. In contrast, artificial neural networks (ANNs) appear to mix all processing, because modular solutions are difficult to find as they are vanishing subspaces in the space of possible solutions. Here we draw inspiration from fault-tolerant computation and the Poisson-like firing of real neurons to show that activity-dependent neural noise, combined with nonlinear neural responses, drives the emergence of solutions that reflect an accurate understanding of modular tasks, corresponding to acquisition of a correct world model. We find that noise-driven modularization can be recapitulated by a deterministic regularizer that multiplicatively combines weights and activations, revealing rich phenomenology not captured in linear networks or by standard regularization methods. We show that modularized ANNs exhibit superior robustness to noise and adversarial attacks. When applied as an architectural constraint, a premodularized ANN acquires the ability to generalize and extrapolate well beyond training data, compared to ANNs without such inductive biases. Together, our work demonstrates a regularizer and architectures that could encourage modularity emergence to yield functional benefits.

View figure in article

Physics Subject Headings (PhySH)

Collections

This article appears in the following collection:

PRX Life Collection on adaptation and learning

PRX Life presents a special collection on ‘Biological systems that learn’, curated by the organizers of the National Institute for Theory and Mathematics in Biology (NITMB) workshop together with early-career researchers. The papers in this collection illustrate how living systems acquire functions, showing it is useful to describe them as tunable matter with many individually adjustable degrees of freedom characterizing microscopic properties. Drawing inspiration from AI and non-biological systems that can acquire desired functions, the featured research highlights the timeliness of the topic and its potential to drive unifying frameworks for emergent collective function in living tunable matter.

Every article in this collection is subject to a rigorous peer review process, upholding the high standards applied to all papers in the journal. The PRX Life editorial team manages peer review and makes all editorial decisions.

Article Text

References (50)

  1. D. M. Lorenz, A. Jeng, and M. W. Deem, The emergence of modularity in biological systems, Phys. Life Rev. 8, 129 (2011).
  2. N. Chomsky, Aspects of the Theory of Syntax (MIT Press, Cambridge, 1965).
  3. A. Boopathy, S. Jiang, W. Yue, J. Hwang, A. Iyer, and I. Fiete, Breaking neural network scaling laws with modularity, Proceedings of the 42nd International Conference on Machine Learning (ICLR, Vancouver, 2025).
  4. T. Veniat, L. Denoyer, and M. Ranzato, Efficient continual learning with modular networks and task-driven priors, Proceedings of the 38th International Conference on Machine Learning (ICLR, 2021).
  5. F. Alet, E. Weng, T. Lozano-Pérez, and L. P. Kaelbling, Neural relational inference with fast modular meta-learning, Advances in Neural Information Processing Systems 32, Vancouver, 2019 (Curran Associates, Red Hook, 2020).
  6. G. Parascandolo, N. Kilbertus, M. Rojas-Carulla, and B. Schölkopf, Learning independent causal mechanisms, in Proceedings of the International Conference on Machine Learning, Stockholm, 2018, edited by J. Dy and A. Krause (PMLR, Cambridge, 2018), Vol. 80.
  7. K. O. Ellefsen, J.-B. Mouret, and J. Clune, Neural modularity helps organisms evolve to learn new skills without forgetting old skills, PLoS Comput. Biol. 11, e1004128 (2015).
  8. M. Hamidi, S. Khajehabdollahi, E. Giannakakis, T. Schäfer, A. Levina, and C. M. Wu, Modular growth of hierarchical networks: Efficient, general, and robust curriculum learning, Proceedings of the ALIFE 2024: Proceedings of the 2024 Artificial Life Conference (ASME, 2024), p. 55.
  9. T. McCourt, I. R. Fiete, and I. L. Chuang, Noisy dynamical systems evolve error correcting codes and modularity, PRX Life 4, 023002 (2026).
  10. S. Gu, M. G. Mattar, H. Tang, and G. Pan, Emergence and reconfiguration of modular structure for artificial neural networks during continual familiarity detection, Sci. Adv. 10, eadm8430 (2024).
  11. M. L. Montero, C. J. H. Ludwig, R. P. Costa, G. Malhotra, and J. S. Bowers, The role of disentanglement in generalisation, Proceedings of the Ninth International Conference on Learning Representations, 2021 (ICLR, Appleton, 2021).
  12. Q. Liang, D. Qian, L. Ziyin, and I. Fiete, Compositional generalization via forced rendering of disentangled latents, in Proceedings of the 42nd International Conference on Machine Learning, Vancouver, 2025, edited by A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (PMLR, Cambridge, 2025), Vol. 267.
  13. Q. Liang, Z. Liu, M. Ostrow, and I. Fiete, How diffusion models learn to factorize and compose, in Advances in Neural Information Processing Systems 37, Vancouver, 2024, edited by A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Curran, Red Hook, 2024).
  14. J. C. R. Whittington, W. Dorrell, S. Ganguli, and T. E. J. Behrens, Disentanglement with biological constraints: A theory of functional cell types, Proceedings of the 11th International Conference on Learning Representations (ICLR, 2023).
  15. A. Camuto and M. Willetts, Explicit regularisation in Gaussian noise injections, in Advances in Neural Information Processing Systems 33, Vancouver, 2020, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Curran Associates, Red Hook, 2020).
  16. B. Poole, J. Sohl-Dickstein, and S. Ganguli, Analyzing noise in autoencoders and deep networks, arXiv:1406.1831.
  17. W. Softky and C. Koch, The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs, J. Neurosci. 13, 334 (1993).
  18. M. Gur, A. Beylin, and D. M. Snodderly, Response variability of neurons in primary visual cortex (V1) of alert monkeys, J. Neurosci. 17, 2914 (1997).
  19. M. Chen and M. W. Deem, Development of modularity in the neural activity of children's brains, Phys. Biol. 12, 016009 (2015).
  20. D. Qian, https://github.com/dq219/NoiseMod (GitHub, 2025).
  21. U. von Luxburg, A tutorial on spectral clustering, Stat. Comput. 17, 395 (2007).
  22. Q. Liao, L. Ziyin, Y. Gan, B. Cheung, M. Harnett, and T. Poggio, Self-assembly of a biologically plausible learning circuit, arXiv:2412.20018.
  23. C. M. Bishop, Training with noise is equivalent to Tikhonov regularization, Neural Comput. 7, 108 (1995).
  24. B. Neyshabur, R. Tomioka, and N. Srebro, Norm-based capacity control in neural networks, in Proceedings of the 28th Conference on Learning Theory, Paris, 2015, edited by P. Grünwald, E. Hazan, and S. Kale (PMLR, Cambridge, 2015), Vol. 40.
  25. B. Neyshabur, R. Salakhutdinov, and N. Srebro, Path-SGD: Path-normalized optimization in deep neural networks, in Proceedings of the 29th International Conference on Neural Information Processing Systems, Montreal, 2015, edited by C. Cortes, D. D. Lee, M. Sugiyama, and R. Garnett (MIT Press, Cambridge, 2015), Vol. 2, pp. 2422–2430.
  26. C. Debao, Degree of approximation by superpositions of a sigmoidal function, Approx. Theory Appl. 9, 17 (1993).
  27. D. X. Zhou, Universality of deep convolutional neural networks, Appl. Comput. Harmon. Anal. 48, 787 (2020).
  28. A. Pinkus, Approximation theory of the MLP model in neural networks, Acta Numer. 8, 143 (1999).
  29. M. T. Augustine, A survey on universal approximation theorems, arXiv:2407.12895.
  30. M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Networks 6, 861 (1993).
  31. A. Krizhevsky, Learning multiple layers of features from tiny images, University of Toronto Report No. 0, 2009, available at https://www.cs.toronto.edu/∼kriz/learning-features-2009-TR.pdf (unpublished).
  32. K. He, X. Zhang, S. Ren, and J. Sun, Learning multiple layers of features from tiny images, Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016 (IEEE, Piscataway, 2016), pp. 770–778.
  33. R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Mach. Learn. 8, 229 (1992).
  34. H. Seung, Learning in spiking neural networks by reinforcement of stochastic synaptic transmission, Neuron 40, 1063 (2003).
  35. I. R. Fiete and H. S. Seung, Gradient learning in spiking neural networks by dynamic perturbation of conductances, Phys. Rev. Lett. 97, 048104 (2006).
  36. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15, 1929 (2014).
  37. A. A. Faisal, L. P. J. Selen, and D. M. Wolpert, Noise in the nervous system, Nat. Rev. Neurosci. 9, 292 (2008).
  38. H. W. Lin, M. Tegmark, and D. Rolnick, Why does deep and cheap learning work so well?, J. Stat. Phys. 168, 1223 (2017).
  39. T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, and Q. Liao, Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review, Int. J. Automat. Comput. 14, 503 (2017).
  40. M. Refinetti, A. Ingrosso, and S. Goldt, Neural networks trained with SGD learn distributions of increasing complexity, J. Stat. Mech. (2025) 024001.
  41. Y. Lu, J. Yin, Z. Chen, H. Gong, Y. Liu, L. Qian, X. Li, R. Liu, I. M. Andolina, and W. Wang, Revealing detail along the visual hierarchy: Neural clustering preserves acuity from V1 to V4, Neuron 98, 417 (2018).
  42. B. K. Hulse and V. Jayaraman, Mechanisms underlying the neural computation of head direction, Annu. Rev. Neurosci. 43, 31 (2020).
  43. C. Lyu, L. F. Abbott, and G. Maimon, Building an allocentric travelling direction signal via vector computation, Nature (London) 601, 92 (2022).
  44. T. Hafting, M. Fyhn, S. Molden, M.-B. Moser, and E. I. Moser, Microstructure of a spatial map in the entorhinal cortex, Nature (London) 436, 801 (2005).
  45. Y. Burak and I. R. Fiete, Accurate path integration in continuous attractor network models of grid cells, PLoS Comput. Biol. 5, e1000291 (2009).
  46. A. N. Kolmogorov, On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition, AMS Trans. Ser 2 28, 55 (1963).
  47. J. Schmidt-Hieber, The Kolmogorov–Arnold representation theorem revisited, Neural Networks 137, 119 (2021).
  48. O. Dugan, R. Dangovski, A. Costa, S. Kim, P. Goyal, J. Jacobson, and M. Soljačić, OccamNet: A fast neural model for symbolic regression at scale, arXiv:2007.10784.
  49. Z. Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljačić, T. Y. Hou, and M. Tegmark, KAN: Kolmogorov-arnold networks, Proceedings of the 13th International Conference on Learning Representations, Singapore, 2025 (ICLR, Appleton, 2025).
  50. M. Cranmer, A. Sanchez-Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho, Discovering symbolic models from deep learning with inductive biases, Advances in Neural Information Processing Systems 33 (Vancouver, 2020).

Outline

Information

PRX Life 4, 033022– Published 1 September, 2026
  • Received 7 December 2025
  • Accepted 29 June 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation