Parsimonious modeling with information filtering networks

We introduce a methodology to construct parsimonious probabilistic models. This method makes use of information filtering networks to produce a robust estimate of the global sparse inverse covariance from a simple sum of local inverse covariances computed on small subparts of the network. Being based on local and low-dimensional inversions, this method is computationally very efficient and statistically robust, even for the estimation of inverse covariance of high-dimensional, noisy, and short time series. Applied to financial data our method results are computationally more efficient than state-of-the-art methodologies such as Glasso producing, in a fraction of the computation time, models that can have equivalent or better performances but with a sparser inference structure. We also discuss performances with sparse factor models where we notice that relative performances decrease with the number of factors. The local nature of this approach allows us to perform computations in parallel and provides a tool for dynamical adaptation by partial updating when the properties of some variables change without the need of recomputing the whole model. This makes this approach particularly suitable to handle big data sets with large numbers of variables. Examples of practical application for forecasting, stress testing, and risk allocation in financial systems are also provided.

Figure 1

Two schematic representations of a TMFG network. The network is made of two cliques of four nodes (4-clique) and one separator of three nodes (3-clique). The cliques are the tetrahedra $\{1,2,3,4\}$ and $\{2,3,4,5\}$ and the separator is the triangle $\{2,3,4\}$. This is a chordal graph.

Figure 2

Decomposition of the joint probability distribution function. Bayes formulas for two sets of variables $A$ and $B$ with a separating set $A\cap B$. Variables within sets $A$ or $B$ are assumed conditionally dependent whereas variables belonging to the two separated sets $A\setminus B$ and $B\setminus A$ are assumed independent conditionally to $A\cap B$. By combining the two formulas one obtains $f(A\cup B)=f\left(A\right)f\left(B\right)/f(A\cap B)$.

Figure 3

Local-global inversion of the covariance matrix. Example for a system of $p=10$ variables associated with a decomposable TMFG graph with $M_c=7$ cliques and $M_s=6$ separators.

Figure 4

TMFG construction. The TMFG graph is generated by adding vertices (e.g., $v_k$) inside triangular faces (e.g., $\{v_1,v_2,v_3\}$) maximizing the ratio of the determinants between separator and clique $det\left({\widehat{\mathbf{R}}}_{\{v_1,v_2,v_3\}}\right)/det\left({\widehat{\mathbf{R}}}_{\{v_1,v_2,v_3,v_k\}}\right)$. This move generates a new 4-clique $\mathcal{C}=\{v_1,v_2,v_3,v_k\}$ and transforms the triangular face into a separator $\mathcal{S}=\{v_1,v_2,v_3\}$.

Figure 5

Demonstration that LoGo sparse inverse covariance represents the dependency structure better than the complete inverse covariance. The figure reports comparisons between log-likelihood for models constructed by using sparse inverse LoGo-TMFG, LoGo-MST and the complete inverse of the empirical covariance matrix (Inv. Cov.). These measures are on $p=300$ off-sample test data series of different lengths $q$ varying from 20 to 2000. The inverse matrices are computed on training data sets of the same length. Data are log returns sampled from 342 stocks prices of equities traded on the U.S. market during the period 1997–2012. The statistics is made stationary by random shuffling the time order. Symbols correspond to averages over 100 samples generated by picking at random 300 series over the 342 and assigning training and testing sets by choosing at random two nonoverlapping time windows of length $q$, the shaded bands are the 95% quantiles. The line on the top, labeled “MAX,” is the theoretical maximum, which is the log-likelihood obtained from the inverse covariance of the testing set.

Figure 6

Demonstration that LoGo sparse modeling has better performances than Glasso models with same sparsity. This figure reports the same log-likelihood as described in the caption of Fig. 5 compared with log-likelihood from state-of-the-art Glasso ${\ell }_1$ penalized sparse inverse covariance models (cross validated, G-Lasso-CV, and of the same sparsity of TMFG, G-Lasso-Sp) and Ridge ${\ell }_2$ penalized inverse model (Ridge).

Figure 7

Demonstration that LoGo models are sparser than Glasso models and are computationally more efficient. The plot on the top reports the number of nonzero off-diagonal coefficients in the precision matrix $\mathbf{J}$ for G-Lasso-CV and LoGo-TMFG. The plot on the bottom reports the computation times (seconds) for G-Lasso-CV and LoGo-TMFG. These data refer to the same simulations as for the results in Figs. 5 and 6. Note that TMFG-LoGo has a constant number of coefficients equal to $3(p-2)=894$.

Figure 8

Demonstration that LoGo sparse models with comparable performances with respect to best-performing Glasso methods produces sparser models in a fraction of the computation time. The top plot reports the fraction of log-likelihood for Glasso and for LoGo-TMFG vs. the fraction of nonzero off-diagonal coefficients of $\mathbf{J}$ for Glasso and for LoGo-TMFG. The bottom plot reports the fraction of computational time vs. the fraction of nonzero off-diagonal coefficients of $\mathbf{J}$ for Glasso and for LoGo-TMFG. The measures are on $p=300$ off-sample test data series of length $q=500$. Inverse matrices are computed on training data sets of the same length. Data are log returns sampled from 342 stocks prices of equities traded on the U.S. market during the period 1997–2012. The statistics is made stationary by random shuffling the time order. Symbols correspond to averages over 100 samples generated by picking at random 300 series over the 342 and assigning training and testing sets by choosing at random two nonoverlapping time windows of length $q$; the shaded bands are the 95% quantiles.

Figure 9

Demonstration that also for nonstationary financial data LoGo sparse inverse covariance represents the dependency structure better than the complete inverse covariance. The figure reports comparisons between log-likelihood for models constructed by using sparse inverse LoGo-TMFG, LoGo-MST, and the complete inverse of the empirical covariance matrix (Inv. Cov.). The measures are on $p=300$ off-sample test data series of different lengths $q$ varying from 20 to 2000. Inverse matrices are computed on training data sets of the same length. Data are log returns sampled from 342 stocks prices of equities traded on the U.S. market during the period 1997–2012. Symbols correspond to averages over 100 samples generated by picking at random 300 series over the 342 and assigning training and testing sets by choosing at random two consecutive nonoverlapping time windows of length $q$; the shaded bands are the 95% quantiles. Testing set is the time window preceding the training set. The line on the top, labeled “MAX,” is the theoretical maximum, which is the log-likelihood obtained from the inverse covariance of the testing set.

Figure 10

Demonstration that also for sparse factor models LoGo sparse inverse covariance can perform well when the number of factors is low but it underperforms for larger number of factors. The figure reports comparisons between log-likelihood for models constructed by using sparse inverse LoGo-TMFG, LoGo-MST and the complete inverse of the empirical covariance matrix (Inv. Cov.). The measures are on $p=300$ off-sample test data series of different lengths $q$ varying from 20 to 2000. Inverse matrices are computed on training data sets of the same length. Plots refer to sparse factor models simulations with 3 (top) and 30 (bottom) factors, respectively. Symbols correspond to averages over 100 samples generated by picking at random 300 series over the 342 and assigning training and testing sets by choosing at random two consecutive nonoverlapping time windows of length $q$, the shaded bands are the 95% quantiles. Testing set is the time window preceding the training set. The line on the top, labeled “MAX,” is the theoretical maximum, which is the log-likelihood obtained from the inverse covariance of the testing set.