- Open Access
Hierarchical Block Structures and High-Resolution Model Selection in Large Networks
Phys. Rev. X 4, 011047 – Published 24 March, 2014
DOI: https://doi.org/10.1103/PhysRevX.4.011047
Abstract
Discovering and characterizing the large-scale topological features in empirical networks are crucial steps in understanding how complex systems function. However, most existing methods used to obtain the modular structure of networks suffer from serious problems, such as being oblivious to the statistical evidence supporting the discovered patterns, which results in the inability to separate actual structure from noise. In addition to this, one also observes a resolution limit on the size of communities, where smaller but well-defined clusters are not detectable when the network becomes large. This phenomenon occurs for the very popular approach of modularity optimization, which lacks built-in statistical validation, but also for more principled methods based on statistical inference and model selection, which do incorporate statistical validation in a formally correct way. Here, we construct a nested generative model that, through a complete description of the entire network hierarchy at multiple scales, is capable of avoiding this limitation and enables the detection of modular structure at levels far beyond those possible with current approaches. Even with this increased resolution, the method is based on the principle of parsimony, and is capable of separating signal from noise, and thus will not lead to the identification of spurious modules even on sparse networks. Furthermore, it fully generalizes other approaches in that it is not restricted to purely assortative mixing patterns, directed or undirected graphs, and ad hoc hierarchical structures such as binary trees. Despite its general character, the approach is tractable and can be combined with advanced techniques of community detection to yield an efficient algorithm that scales well for very large networks.
Popular Summary
Many social, technological, and biological networks are known to organize into modules or “communities,” groups of nodes with a similar connectivity pattern. At first sight, the task of identifying and characterizing these modules seems a simple one. This simplicity is, however, deceiving. Despite a large amount of effort aimed at implementing this task, several fundamental questions still remain open: How should one formally quantify the network’s modular structure? Given a formal characterization, what is the optimal method to identify modules from empirical or simulated data? How should one distinguish modules that arise from some underlying organizational principle from those that occur purely by chance? Why do some existing methods have a “resolution” limit, in other words, fail to detect modules smaller than a certain size? And, how does one overcome such limits? In this work, we show that the concepts of statistical inference and modular hierarchy may hold the key to all these questions, in particular, in terms of providing a principled means for distinguishing structure from noise and a detection method without a resolution limit.
We approach the module-detection problem in two steps. First, we view networks as the results of a nested generative model that is capable of incorporating arbitrary hierarchical modular structures present at different size scales. Indeed, this model serves as a general and formally satisfying definition of arbitrary modular structure and provides a more faithful description for many empirical networks. Based on this view, we then use principled statistical inference methods to detect modules of different sizes and show that this detection approach can reliably distinguish genuine modules from spurious ones arising from noise while at the same time not suffering from severe resolution limits present in other approaches.
Our modular-hierarchy approach does not impose any specific constraints on the types of modular structures, and the statistical inference methods are derived from basic principles rather than being ad hoc. Despite its general character, the approach can be effectively combined with some of the existing community-detection techniques for efficient implementations on very large empirical networks.
Article Text
Supplemental Material
References (107)
- M. E. J. Newman, Communities, Modules, and Large-Scale Structure in Networks, Nat. Phys. 8, 25 (2011).
- S. Fortunato, Community Detection in Graphs, Phys. Rep. 486, 75 (2010).
- M. Girvan and M. E. J. Newman, Community Structure in Social and Biological Networks, Proc. Natl. Acad. Sci. U.S.A. 99, 7821 (2002).
- E. Ravasz, A. L. Somera, D. A. Mongru, Z. N. Oltvai, and A.-L. Barabási, Hierarchical Organization of Modularity in Metabolic Networks, Science 297, 1551 (2002).
- R. J. Fletcher, Jr, A. Revell, B. E. Reichert, W. M. Kitchens, J. D. Dixon, and J. D. Austin, Network Modularity Reveals Critical Scales for Connectivity in Ecology and Evolution, Nat. Commun. 4, 2572 (2013).
- R. Albert, H. Jeong, and A.-L. Barabási, Internet: Diameter of the World Wide Web, Nature (London) 401, 130 (1999).
- S.-H. Yook, H. Jeong, and A.-L. Barabási, Modeling the Internet’s Large-Scale Topology, Proc. Natl. Acad. Sci. U.S.A. 99, 13 382 (2002).
- Y. Zhao, E. Levina, and J. Zhu, Community Extraction for Social Networks, Proc. Natl. Acad. Sci. U.S.A. 108, 7321 (2011).
- G. Palla, I. Derényi, I. Farkas, and T. Vicsek, Uncovering the Overlapping Community Structure of Complex Networks in Nature and Society, Nature (London) 435, 814 (2005).
- M. E. J. Newman and M. Girvan, Finding and Evaluating Community Structure in Networks, Phys. Rev. E 69, 026113 (2004).
- A. Clauset, M. E. J. Newman, and C. Moore, Finding Community Structure in Very Large Networks, Phys. Rev. E 70, 066111 (2004).
- V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre, Fast Unfolding of Communities in Large Networks, J. Stat. Mech. (2008) P10008.
- R. Guimerà, M. Sales-Pardo, and L. A. N. Amaral, Modularity from Fluctuations in Random Graphs and Complex Networks, Phys. Rev. E 70, 025101 (2004).
- S. Fortunato and M. Barthélemy, Resolution Limit in Community Detection, Proc. Natl. Acad. Sci. U.S.A. 104, 36 (2007).
- A. Lancichinetti and S. Fortunato, Limits of Modularity Maximization in Community Detection, Phys. Rev. E 84, 066122 (2011).
- B. H. Good, Y.-A. de Montjoye, and A. Clauset, Performance of Modularity Maximization in Practical Contexts, Phys. Rev. E 81, 046106 (2010).
- M. B. Hastings, Community Detection as an Inference Problem, Phys. Rev. E 74, 035102 (2006).
- D. Garlaschelli and M. I. Loffredo, Maximum Likelihood: Extracting Unbiased Information from Complex Networks, Phys. Rev. E 78, 015101 (2008).
- M. E. J. Newman and E. A. Leicht, Mixture Models and Exploratory Analysis in Networks, Proc. Natl. Acad. Sci. U.S.A. 104, 9564 (2007).
- J. Reichardt and D. R. White, Role Models for Complex Networks, Eur. Phys. J. B 60, 217 (2007).
- J. M. Hofman and C. H. Wiggins, Bayesian Approach to Network Modularity, Phys. Rev. Lett. 100, 258701 (2008).
- P. J. Bickel and A. Chen, A Nonparametric View of Network Models and Newman-Girvan and Other Modularities, Proc. Natl. Acad. Sci. U.S.A. 106, 21068 (2009).
- R. Guimerà and M. Sales-Pardo, Missing and Spurious Interactions and the Reconstruction of Complex Networks, Proc. Natl. Acad. Sci. U.S.A. 106, 22 073 (2009).
- B. Karrer and M. E. J. Newman, Stochastic Block Models and Community Structure in Networks, Phys. Rev. E 83, 016107 (2011).
- B. Ball, B. Karrer, and M. E. J. Newman, Efficient and Principled Method for Detecting Communities in Networks, Phys. Rev. E 84, 036103 (2011).
- J. Reichardt, R. Alamino, and D. Saad, The Interplay between Microscopic and Mesoscopic Structures in Complex Networks, PLoS One 6, e21282 (2011).
- Y. Zhu, X. Yan, and C. Moore, Oriented and Degree-Generated Block Models: Generating and Inferring Communities with Inhomogeneous Degree Distributions, J. Complex Netw. 2, 1 (2014).
- E. B. Baskerville, A. P. Dobson, T. Bedford, S. Allesina, T. Michael Anderson, and M. Pascual, Spatial Guilds in the Serengeti Food Web Revealed by a Bayesian Group Model, PLoS Comput. Biol. 7, e1002321 (2011).
- A. Decelle, F. Krzakala, C. Moore, and L. Zdeborová, Inference and Phase Transitions in the Detection of Modules in Sparse Networks, Phys. Rev. Lett. 107, 065701 (2011).
- A. Decelle, F. Krzakala, C. Moore, and L. Zdeborová, Asymptotic Analysis of the Stochastic Block Model for Modular Networks and Its Algorithmic Applications, Phys. Rev. E 84, 066106 (2011).
- E. Mossel, J. Neeman, and A. Sly, Stochastic Block Models and Reconstruction, arXiv:1202.1499.
- T. P. Peixoto, Parsimonious Module Inference in Large Networks, Phys. Rev. Lett. 110, 148701 (2013).
- P. W. Holland, K. B. Laskey, and S. Leinhardt, Stochastic Block Models: First Steps, Soc. Networks 5, 109 (1983).
- S. E. Fienberg, M. M. Meyer, and S. S. Wasserman, Statistical Analysis of Multiple Sociometric Relations, J. Am. Stat. Assoc. 80, 51 (1985).
- K. Faust and S. Wasserman, Block Models: Interpretation and Evaluation, Soc. Networks 14, 5 (1992).
- C. J. Anderson, S. Wasserman, and K. Faust, Building Stochastic Block Models, Soc. Networks 14, 137 (1992).
- M. Rosvall and C. T. Bergstrom, An Information-Theoretic Framework for Resolving Community Structure in Complex Networks, Proc. Natl. Acad. Sci. U.S.A. 104, 7327 (2007).
- J.-J. Daudin, F. Picard, and S. Robin, A Mixture Model for Rndom Graphs, Stat. Comput. 18, 173 (2008).
- M. Mariadassou, S. Robin, and C. Vacher, Uncovering Latent Structure in Valued Graphs: A Variational Approach, Ann. Appl. Stat. 4, 715 (2010).
- C. Moore, X. Yan, Y. Zhu, J.-B. Rouquier, and T. Lane, Active Learning for Node Classification in Assortative and Disassortative Networks, in Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’11) (ACM, New York, 2011), pp. 841–849.
- P. Latouche, E. Birmele, and C. Ambroise, Variational Bayesian Inference and Complexity Control for Stochastic Block Models, Stat. Model. 12, 93 (2012).
- E. Côme and P. Latouche, Model Selection and Clustering in Stochastic Bock Models with the Exact Integrated Complete Data Likelihood, arXiv:1303.2962.
- A. Clauset, C. Moore, and M. E. J. Newman, Statistical Network Analysis: Models, Issues, and New Directions, Lecture Notes in Computer Science Vol. 4503, edited by E. Airoldi, D. M. Blei, S. E. Fienberg, A. Goldenberg, E. P. Xing, and A. X. Zheng (Springer, Berlin, 2007), pp. 1–13.
- A. Clauset, C. Moore, and M. E. J. Newman, Hierarchical Structure and the Prediction of Missing Links in Networks, Nature (London) 453, 98 (2008).
- M. Rosvall and C. T. Bergstrom, Multilevel Compression of Random Walks on Networks Reveals Hierarchical Organization in Large Integrated Systems, PLoS One 6, e18209 (2011).
- M. Sales-Pardo, R. Guimerà, A. A. Moreira, and L. A. N. Amaral, Extracting the Hierarchical Organization of Complex Systems, Proc. Natl. Acad. Sci. U.S.A. 104, 15 224 (2007).
- P. Ronhovde and Z. Nussinov, Multiresolution Community Detection for Megascale Networks by Information-Based Replica Correlations, Phys. Rev. E 80, 016109 (2009).
- I. A. Kovács, R. Palotai, M. S. Szalay, and P. Csermely, Community Landscapes: An Integrative Approach to Determine Overlapping Network Module Hierarchy, Identify Key Nodes, and Predict Network Dynamics, PLoS One 5, e12528 (2010).
- Y. Park, C. Moore, and J. S. Bader, Dynamic Networks from Hierarchical Bayesian Graph Clustering, PLoS One 5, e8118 (2010).
- T. P. Peixoto, Entropy of Stochastic Block Model Ensembles, Phys. Rev. E 85, 056122 (2012).
- J. Leskovec, D. Chakrabarti, J. Kleinberg, C. Faloutsos, and Z. Ghahramani, Kronecker Graphs: An Approach to Modeling Networks, J. Mach. Learn. Res. 11, 9851042 (2010).
- G. Bianconi, Entropy of Network Ensembles, Phys. Rev. E 79, 036114 (2009).
- P. D. Grànwald, The Minimum Description Length Principle (MIT Press, Cambridge, MA, 2007).
- J. Rissanen, Information and Complexity in Statistical Modeling, (Springer, New York, 2010), 1st ed.
- C. Biernacki, G. Celeux, and G. Govaert, Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood, IEEE Trans. Pattern Anal. Mach. Intell. 22, 719 (2000).
- X. Yan, J. E. Jensen, F. Krzakala, C. Moore, C. R. Shalizi, L. Zdeborova, P. Zhang, and Y. Zhu, Model Selection for Degree-Corrected Block Models, arXiv:1207.3994.
- G. Schwarz, Estimating the Dimension of a Model, Ann. Stat. 6, 461 (1978).
- H. Akaike, A New Look at the Statistical Model Identification, IEEE Trans. Autom. Control 19, 716 (1974).
- J. Reichardt and M. Leone, (Un)detectable Cluster Structure in Sparse Networks, Phys. Rev. Lett. 101, 078701 (2008).
- D. Hu, P. Ronhovde, and Z. Nussinov, Phase Transitions in Random Potts Systems and the Community Detection Problem: Spin-Glass Type and Dynamic Perspectives, Philos. Mag. 92, 406 (2012).
- A. Condon and R. M. Karp, Algorithms for Graph Partitioning on the Planted Partition Model, Random Struct. Algorithms 18, 116 (2001).
- J. Xiang, X. G. Hu, X. Y. Zhang, J. F. Fan, X. L. Zeng, G. Y. Fu, K. Deng, and K. Hu, Multiresolution Modularity Methods and Their Limitations in Community Detection, Eur. Phys. J. B 85, 1 (2012).
- T. P. Peixoto, Efficient Monte Carlo and Greedy Heuristic for the Inference of Stochastic Block Models, Phys. Rev. E 89, 012804 (2014).
- G. Palla, L. Lovász, and T. Vicsek, Multifractal Network Generator, Proc. Natl. Acad. Sci. U.S.A. 107, 7640 (2010).
- T. P. Peixoto, Eigenvalue Spectra of Modular Networks, Phys. Rev. Lett. 111, 098701 (2013).
- R. R. Nadakuditi and M. E. J. Newman, Graph Spectra and the Detectability of Community Structure in Networks, Phys. Rev. Lett. 108, 188701 (2012).
- L. A. Adamic and N. Glance, The Political Blogosphere and the 2004 U.S. Election: Divided They Blog, in Proceedings of the 3rd International Workshop on Link Discovery (LinkKDD ’05) (ACM, New York, 2005), pp. 36–43.
- D. Holten, Hierarchical Edge Bundles: Visualization of Adjacency Relations in Hierarchical Data, IEEE Trans. Visual. Comput. Graph. 12, 741 (2006).
- D. Lusseau, K. Schneider, O. J. Boisseau, P. Haase, E. Slooten, and S. M. Dawson, The Bottlenose Dolphin Community of Doubtful Sound Features a Large Proportion of Long-Lasting Associations, Behav. Ecol. Sociobiol., 54, 396 (2003).
- J. Leskovec, J. Kleinberg, and C. Faloutsos, Graph Evolution: Densification and Shrinking Diameters, ACM Trans. Knowl. Discov. Data 1, 2 (2007).
- J. Leskovec, K. J. Lang, A. Dasgupta, and M. W. Mahoney, Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters, arXiv:0810.1355.
- J. Gehrke, P. Ginsparg, and J. Kleinberg, Overview of the 2003 KDD Cup, SIGKDD Explor. Newsletter 5, 149 (2003).
- J. Yang and J. Leskovec, Defining and Evaluating Network Communities Based on Ground Truth, in Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics (MDS ’12) (ACM, New York, 2012) p. 3:1–3:8.
- T. S. Evans, American College Football Network Files, (FigShare), http://figshare.com/articles/American_College_Football_Network_Files/93179.
- D. J. Watts and S. H. Strogatz, Collective dynamics of “Small-World” Networks, Nature (London) 393, 440 (1998).
- O. Richters and T. P. Peixoto, Trust Transitivity in Social Networks, PLoS One 6, e18384 (2011).
- K. I. Goh, M. E. Cusick, D. Valle, B. Childs, M. Vidal, and A. L. Barabási, The Human Disease Network, Proc. Natl. Acad. Sci. U.S.A. 104, 8685 (2007).
- E. Cho, S. A. Myers, and J. Leskovec, Friendship and Mobility: User Movement in Location-Based Social Networks, in Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, California, USA, 2011 (KDD ’11) (ACM, New York, NY, 2011), (Ref. [40]), pp. 1082–1090.
- M. Richardson, R. Agrawal, and P. Domingos, The Semantic Web–ISWC 2003, in Lecture Notes in Computer Science Vol. 2870, edited by D. Fensel, K. Sycara, and J. Mylopoulos (Springer, Berlin, 2003), pp. 351–368.
- J. Leskovec, D. Huttenlocher, and J. Kleinberg, Signed Networks in Social Media, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10) (ACM, New York, 2010) pp. 1361–1370.
- J. McAuley and J. Leskovec, Computer Vision ECCV 2012, in Lecture Notes in Computer Science Vol. 7575, edited by A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid (Springer, Berlin, 2012), pp. 828–841.
- J. Leskovec, J. Kleinberg, and C. Faloutsos, Graphs Over Time: Densification Laws, Shrinking diameters, and Possible Explanations, in Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, Chicago, Illinois, USA, 2005 (KDD ’05) (ACM, New York, NY, 2005), (Ref. [67]), pp. 177–187.
- E. M. Airoldi, D. M. Blei, S. E. Fienberg, and E. P. Xing, Mixed Membership stochastic Block Models, J. Mach. Learn. Res. 9, 1981 (2008).
- A. Lancichinetti, S. Fortunato, and J. Kertész, Detecting the Overlapping and Hierarchical Community Structure in Complex Networks, New J. Phys. 11, 033015 (2009).
- P. K. Gopalan and D. M. Blei, Efficient discovery of Overlapping Communities in Massive Networks, Proc. Natl. Acad. Sci. U.S.A. 110, 14 534 (2013).
- Y.-Y. Ahn, J. P. Bagrow, and S. Lehmann, Link Communities Reveal Multiscale Complexity in Networks, Nature (London) 466, 761 (2010).
- D. Liben-Nowell and J. Kleinberg, The Link-Prediction Problem for Social Networks, J. Am. Soc. Inf. Sci. Technol. 58, 1019 (2007).
- D. Grady, C. Thiemann, and D. Brockmann, Robust Classification of Salient Links in Complex Networks, Nat. Commun. 3, 864 (2012).
- G. Bianconi, P. Pin, and M. Marsili, Assessing the Relevance of Node Features for Network Structure, Proc. Natl. Acad. Sci. U.S.A. 106, 11 433 (2009).
- R. Guimerà and L. A. N. Amaral, Functional Cartography of Complex Metabolic Networks, Nature (London) 433, 895 (2005).
- A. Lancichinetti and S. Fortunato, Community Detection Algorithms: A Comparative Analysis, Phys. Rev. E 80, 056117 (2009).
- M. Rosvall and C. T. Bergstrom, Maps of Random Walks on Complex Networks Reveal Community Structure, Proc. Natl. Acad. Sci. U.S.A. 105, 1118 (2008).
- M. Rosvall, D. Axelsson, and C. T. Bergstrom, The Map Equation, Eur. Phys. J. Spec. Top. 178, 13 (2009).
- A. Lancichinetti, S. Fortunato, and F. Radicchi, Benchmark Graphs for Testing Community Detection Algorithms, Phys. Rev. E 78, 046110 (2008).
- M. Meilă, Learning Theory and Kernel Machines, in Lecture Notes in Computer Science Vol. 2777, edited by B. Schölkopf and M. K. Warmuth (Springer, Berlin, 2003), pp. 173–187.
- See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevX.4.011047 for high resolution versions of Figs. 6 and 7, and additional information on the internet data.
This specification generalizes other hierarchical constructions in a straightforward manner. For instance, the generative model of Refs. [43, 44] can be recovered as a special case by forcing a binary tree hierarchy, terminating at the individual nodes, and a strictly assortative modular structure. A similar argument holds for the variant of Ref. [51] as well.
In Ref. [32] the degree sequence entropy was taken to be , with , which implicitly assumed that the degrees are uncorrelated with the block partitions, and, hence, should be interpreted only as an upper bound to the actual description length given by Eq. (9).
Note that MDL can still be used to select the simpler model in this case: Although the complete description length will be asymptotically the same with both models for networks sampled from the traditional block model, we still have that , since the degree-corrected version still needs to include the information on the degree sequence, as in Eq. (9).
The normalized mutual information (NMI) is defined as , where and , where and are two partitions of the network.
The fact that the NMI between the true and inferred partitions remains slightly above zero in Fig. 2 for with the incomplete BMS criterion is a finite size effect, as it tends increasingly to zero as . On the other hand, according to this criterion, the inferred value of in this region increases as becomes larger.
This threshold corresponds simply to the point where it becomes impossible to fully encode the block partition in the network structure, i.e., for uniform blocks , which leads to and, hence, .
This limit cannot be significantly changed even if one introduces scale parameters to the definition of modularity [15, 62].
In the model selection context, adding a single edge between the blocks is not a necessary condition for the observation of the resolution limit, and has a negligible effect, differently from the modularity approach, where it is a deciding factor.
An efficient and fully documented C++ implementation of the algorithm described here is freely available as part of the graph-tool Python library at http://graph-tool.skewed.de.
- IPv4 Routed /24 AS Links Dataset, http://www.caida.org/data/active/ipv4-routed-topology-aslinks-dataset.xml.
Note that this is slightly different than in Ref. [94], which parametrized the fraction of internal and external degrees via a local mixing parameter , which is the same for all communities. That choice corresponds to a different parametrization of the degree-corrected block model than the one used here. However, since the blocks have different sizes, and the degrees are approximately the same in all blocks, in general, there is no choice of that would allow one to recover the fully random configuration model, since the intrinsic mixing would be different for each block in this case. Because of this, we have opted for the parametrization used here; however, this should not alter the interpretation of the benchmark and the comparison with Ref. [94] in a significant way.
