Download or read book Estimation of Graphical Models Convex Formulations and Algorithms written by Jinchao Li and published by . This book was released on 2015 with total page 129 pages. Available in PDF, EPUB and Kindle. Book excerpt: A Gaussian graphical model is a graph representation of conditional independence relations among Gaussian random variables. A fundamental problem in the estimation of Gaussian graphical models is the selection of the graph topology given relatively small amounts of data. This problem is often solved via L1-regularized maximum likelihood estimation, for which many large-scale convex optimization algorithms have been developed. In this thesis, we consider several extensions of Gaussian graphical models and develop fast algorithms based on convex optimization methods. As a first extension, we consider the restricted sparse inverse covariance selection problem where the set of zero entries of the inverse covariance matrix is partially known and an L1-norm penalization is applied to the remaining entries.The proximal Newton method is an attractive algorithm for this problem since the key computations in the algorithm, which include the evaluation of gradient and Hessian of the log-likelihood function, can be implemented efficiently with sparse chordal matrix techniques. We analyze the convergence of the inexact proximal Newton method for the penalized maximum likelihood problem. The convergence analysis applies to a wider class of problems with a self-concordant term in the objective. The numerical results indicate that the method can reach a high accuracy, even with inexact computation of the proximal Newton steps. As a second extension, we consider Gaussian graphical models for time series, with focus on the estimation of multiple time series graphical models with similar graph structures or identical graph structure but different edge coefficients. We formulate a joint estimation method for estimating multiple time series graphical models simultaneously, with a group penalty on the edge coefficients for different models. We apply the Douglas-Rachford algorithm to solve the estimation problem for the joint model, and provide model selection methods for choosing parameters. Both synthetic and real data (fMRI brain activity and international stock markets) examples are provided to demonstrate the advantage of the joint estimation method. The last extension is the generalization of Gaussian graphical models for time series to latent variables. We illustrate the effect of latent variables on the conditional independence structure, and describe a Gaussian graphical model for time series with latent variables. The Douglas-Rachford method is applied to this problem. Simulations with synthetic data demonstrate how the method recovers the graph topology.

Download or read book Sparse Modeling written by Irina Rish and published by CRC Press. This book was released on 2014-12-01 with total page 255 pages. Available in PDF, EPUB and Kindle. Book excerpt: Sparse models are particularly useful in scientific applications, such as biomarker discovery in genetic or neuroimaging data, where the interpretability of a predictive model is essential. Sparsity can also dramatically improve the cost efficiency of signal processing. Sparse Modeling: Theory, Algorithms, and Applications provides an introduction to the growing field of sparse modeling, including application examples, problem formulations that yield sparse solutions, algorithms for finding such solutions, and recent theoretical results on sparse recovery. The book gets you up to speed on the latest sparsity-related developments and will motivate you to continue learning about the field. The authors first present motivating examples and a high-level survey of key recent developments in sparse modeling. The book then describes optimization problems involving commonly used sparsity-enforcing tools, presents essential theoretical results, and discusses several state-of-the-art algorithms for finding sparse solutions. The authors go on to address a variety of sparse recovery problems that extend the basic formulation to more sophisticated forms of structured sparsity and to different loss functions. They also examine a particular class of sparse graphical models and cover dictionary learning and sparse matrix factorizations.

Download or read book Learning Structured Matrices in High Dimensions written by Karthik Mohan and published by . This book was released on 2015 with total page 152 pages. Available in PDF, EPUB and Kindle. Book excerpt: The topic of learning matrix structures in the emph{high-dimensional statistical setting} has received a lot of attention in machine learning, statistics and signal processing literature. High dimensional setting refers to problems that have more parameters to estimate than samples or measurements. Examples of problems that fall in this area include matrix factorization, matrix rank minimization, and graphical model estimation. These problems arise in many applications including collaborative filtering, system identification, and learning gene-regulatory networks. The focus of this thesis is on the algorithmic and theoretical aspects of learning matrix structures in the high dimensional setting with emphasis on the two problems of matrix rank minimization and graphical model estimation. We first consider the problem of reconstructing a low-rank matrix given a few linear measurements. This problem is known to be NP-hard. We propose a family of iterative reweighted algorithms that reconstruct the low-rank matrix efficiently and accurately. We also give recovery guarantees for these algorithms under suitable assumptions on the measurement maps. Finally, we also apply our algorithms to the problems of system identification and matrix completion. Our extensive numerical experiments illustrate that our algorithms perform better than state of the art algorithms on both synthetic and real data when the rank of the true underlying matrix is unknown. We next consider the problem of learning structured Gaussian graphical models in the high dimensional setting. This problem has a lot of applications ranging from biological modeling to social networks. While existing literature focuses on learning Gaussian graphical models using a simple sparsity regularizer (e.g. the graphical lasso formulation), applications often present prior knowledge that require more structured regularizers. Examples of networks that occur in applications include scale-free networks (where a few nodes have a high degree), community networks (where the connections occur in small dense communities), etc. We propose the structured graphical lasso formulation, a framework that generalizes graphical lasso to learning graphical models with arbitrary user-specified structures. We continue by considering a special case of the structured graphical lasso that pertains to learning graphical model with a few hub nodes. We propose an ADMM algorithm to solve this formulation. Empirical studies indicate the superiority of hub graphical lasso over graphical lasso and other traditional methods in learning graphs with few hubs. We also give model selection consistency results for recovering graphs with few hubs. When the number of hubs to be learned is O(1), hub graphical lasso requires far fewer samples than graphical lasso to learn the right graphical model. We move on to study the problem of estimating multiple Gaussian graphical models where information is shared between the graphical models. We propose different formulations that capture different ways in which information is shared between the networks. Our empirical results indicate that our approach does better than existing approaches in estimating gene-regulatory networks with shared information. We switch gears by focusing on methods to reduce computational time for our models. We note that the proposed ADMM algorithm for hub graphical lasso and other formulations, though effective, involves a singular value decomposition in each iteration whose computational cost can be prohibitive for large problem sizes. To alleviate the computational bottle neck, we first provide algorithm independent approaches (based on screening rules) to speed up our proposed convex formulations. These screening rules are used to break down the problem into independent sub-problems (where possible), yielding significant speed-ups in computation. Typical algorithms for learning sparse graphical models require a Cholesky computation to check for positive definiteness of the iterates. We propose a block-coordinate descent algorithm with step-size selection rules that avoid expensive Cholesky checks and only rely on matrix vector multiplies in each iteration. Our empirical results show significant speed ups over the SVD based ADMM algorithm.

Download or read book Convex Matrix Sparsity for Demixing with an Application to Graphical Model Structure Estimation written by Marina Vinyes and published by . This book was released on 2018 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: The goal of machine learning is to learn a model from some data that will make accurate predictions on data that it has not seen before. In order to obtain a model that will generalize on new data, and avoid overfitting, we need to restrain the model. These restrictions are usually some a priori knowledge of the structure of the model. First considered approaches included a regularization, first ridge regression and later Lasso regularization for inducing sparsity in the solution. Sparsity, also known as parsimony, has emerged as a fundamental concept in machine learning. Parsimonious models are appealing since they provide more interpretability and better generalization (avoid overfitting) through the reduced number of parameters. Beyond general sparsity and in many cases, models are constrained structurally so they have a simple representation in terms of some fundamental elements, consisting for example of a collection of specific vectors, matrices or tensors. These fundamental elements are called atoms. In this context, atomic norms provide a general framework for estimating these sorts of models. The goal of this thesis is to use the framework of convex sparsity provided by atomic norms to study a form of matrix sparsity. First, we develop an efficient algorithm based on Frank-Wolfe methods that is particularly adapted to solve problems with an atomic norm regularization. Then, we focus on the structure estimation of Gaussian graphical models, where the structure of the graph is encoded in the precision matrix and study the case with unobserved variables. We propose a convex formulation with an algorithmic approach and provide a theoretical result that states necessary conditions for recovering the desired structure. Finally, we consider the problem of signal demixing into two or more components via the minimization of a sum of norms or gauges, encoding each a structural prior on the corresponding components to recover. In particular, we provide general exact recovery guarantees in the noiseless setting based on incoherence measures.

Download or read book Convex Optimization in Signal Processing and Communications written by Daniel P. Palomar and published by Cambridge University Press. This book was released on 2010 with total page 513 pages. Available in PDF, EPUB and Kindle. Book excerpt: Leading experts provide the theoretical underpinnings of the subject plus tutorials on a wide range of applications, from automatic code generation to robust broadband beamforming. Emphasis on cutting-edge research and formulating problems in convex form make this an ideal textbook for advanced graduate courses and a useful self-study guide.

Download or read book Learning in Graphical Models written by M.I. Jordan and published by Springer Science & Business Media. This book was released on 2012-12-06 with total page 658 pages. Available in PDF, EPUB and Kindle. Book excerpt: In the past decade, a number of different research communities within the computational sciences have studied learning in networks, starting from a number of different points of view. There has been substantial progress in these different communities and surprising convergence has developed between the formalisms. The awareness of this convergence and the growing interest of researchers in understanding the essential unity of the subject underlies the current volume. Two research communities which have used graphical or network formalisms to particular advantage are the belief network community and the neural network community. Belief networks arose within computer science and statistics and were developed with an emphasis on prior knowledge and exact probabilistic calculations. Neural networks arose within electrical engineering, physics and neuroscience and have emphasised pattern recognition and systems modelling problems. This volume draws together researchers from these two communities and presents both kinds of networks as instances of a general unified graphical formalism. The book focuses on probabilistic methods for learning and inference in graphical models, algorithm analysis and design, theory and applications. Exact methods, sampling methods and variational methods are discussed in detail. Audience: A wide cross-section of computationally oriented researchers, including computer scientists, statisticians, electrical engineers, physicists and neuroscientists.

Download or read book Optimization for Machine Learning written by Suvrit Sra and published by MIT Press. This book was released on 2012 with total page 509 pages. Available in PDF, EPUB and Kindle. Book excerpt: An up-to-date account of the interplay between optimization and machine learning, accessible to students and researchers in both communities. The interplay between optimization and machine learning is one of the most important developments in modern computational science. Optimization formulations and methods are proving to be vital in designing algorithms to extract essential knowledge from huge volumes of data. Machine learning, however, is not simply a consumer of optimization technology but a rapidly evolving field that is itself generating new optimization ideas. This book captures the state of the art of the interaction between optimization and machine learning in a way that is accessible to researchers in both fields. Optimization approaches have enjoyed prominence in machine learning because of their wide applicability and attractive theoretical properties. The increasing complexity, size, and variety of today's machine learning models call for the reassessment of existing assumptions. This book starts the process of reassessment. It describes the resurgence in novel contexts of established frameworks such as first-order methods, stochastic approximations, convex relaxations, interior-point methods, and proximal methods. It also devotes attention to newer themes such as regularized optimization, robust optimization, gradient and subgradient methods, splitting techniques, and second-order methods. Many of these techniques draw inspiration from other fields, including operations research, theoretical computer science, and subfields of optimization. The book will enrich the ongoing cross-fertilization between the machine learning community and these other fields, and within the broader optimization community.

Download or read book Optimization Methods for Regularized High Dimensional Graphical Model Selection written by Onkar Anant Dalal and published by . This book was released on 2013 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Graphical models yield compact representations of the dependencies present in a multivariate random vector via graphs/networks. Nodes in the graph encode random variables in a high dimensional random vector, and the edges represent different types of associations, such as conditional or marginal dependences. Sparse graphical models have been useful for encoding complex multivariate dependencies in ultra high dimensional sample starved settings, where limited sample sizes often only allow for the estimation of sparse graphs. Given the wide applicability of such models, the field has seen several key contributions from a wide spectrum of communities, including the statistics, machine learning, mathematics, computer science, computational mathematics and optimization communities. Despite tremendous efforts, the vast majority of work on graphical model selection for continuous data have been centered around the multivariate Gaussian distribution. This restriction often poses serious shortcomings in various applications. In this thesis we propose a comprehensive methodology for graphical model selection that goes beyond the Gaussian paradigm. In particular, we propose a nested sequence of families of distributions rooted in probability and statistical theory that enrich the Gaussian, so as to yield a a more flexible family. We demonstrate that our proposed class of distributions, the log-concave elliptical family, has deep and interesting structure. Moreover, this family of multivariate distributions are constructed so as to take advantage of convex optimization tools that yield fast algorithms in order to estimate high dimensional partial correlation graphs. We develop rigorous theory to give a firm foundation to our proposed approach, both from optimization and statistical perspectives. Statistical issues such as identifiability, calculation of the Fisher information, consistency and asymptotic normality are considered. Consistent estimation of the additional shape parameters of the log-concave elliptical family in a way that is computationally tractable is carefully developed. From the optimization perspective, first and second order proximal methods are used for maximizing l1 regularized log-concave elliptical likelihoods, and linear and quadratic rates of convergence for these approaches are established. To our knowledge, our endeavour is the only such approach in the literature with established theory that is applicable in moderate or high dimensions. The methodology is illustrated on both real and simulated data to demonstrate its efficacy.

Download or read book Convex Relaxation Methods for Graphical Models written by Jason Kyle Johnson and published by . This book was released on 2008 with total page 257 pages. Available in PDF, EPUB and Kindle. Book excerpt: Graphical models provide compact representations of complex probability distributions of many random variables through a collection of potential functions defined on small subsets of these variables. This representation is defined with respect to a graph in which nodes represent random variables and edges represent the interactions among those random variables. Graphical models provide a powerful and flexible approach to many problems in science and engineering, but also present serious challenges owing to the intractability of optimal inference and estimation over general graphs. In this thesis, we consider convex optimization methods to address two central problems that commonly arise for graphical models. First, we consider the problem of determining the most probable configuration-also known as the maximum a posteriori (MAP) estimate-of all variables in a graphical model, conditioned on (possibly noisy) measurements of some variables. This general problem is intractable, so we consider a Lagrangian relaxation (LR) approach to obtain a tractable dual problem. This involves using the Lagrangian decomposition technique to break up an intractable graph into tractable subgraphs, such as small "blocks" of nodes, embedded trees or thin subgraphs. We develop a distributed, iterative algorithm that minimizes the Lagrangian dual function by block coordinate descent. This results in an iterative marginal-matching procedure that enforces consistency among the subgraphs using an adaptation of the well-known iterative scaling algorithm. This approach is developed both for discrete variable and Gaussian graphical models. In discrete models, we also introduce a deterministic annealing procedure, which introduces a temperature parameter to define a smoothed dual function and then gradually reduces the temperature to recover the (non-differentiable) Lagrangian dual. When strong duality holds, we recover the optimal MAP estimate. We show that this occurs for a broad class of "convex decomposable" Gaussian graphical models, which generalizes the "pairwise normalizable" condition known to be important for iterative estimation in Gaussian models.

Download or read book Estimation of Structured Graphical Models written by Jiaxi Ying and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Multiscale Gaussian Graphical Models and Algorithms for Large scale Inference written by Myung Jin Choi (Ph. D.) and published by . This book was released on 2007 with total page 123 pages. Available in PDF, EPUB and Kindle. Book excerpt: Graphical models provide a powerful framework for stochastic processes by representing dependencies among random variables compactly with graphs. In particular, multiscale tree-structured graphs have attracted much attention for their computational efficiency as well as their ability to capture long-range correlations. However, tree models have limited modeling power that may lead to blocky artifacts. Previous works on extending trees to pyramidal structures resorted to computationally expensive methods to get solutions due to the resulting model complexity. In this thesis, we propose a pyramidal graphical model with rich modeling power for Gaussian processes, and develop efficient inference algorithms to solve large-scale estimation problems. The pyramidal graph has statistical links between pairs of neighboring nodes within each scale as well as between adjacent scales. Although the graph has many cycles, its hierarchical structure enables us to develop a class of fast algorithms in the spirit of multipole methods. The algorithms operate by guiding far-apart nodes to communicate through coarser scales and considering only local interactions at finer scales. The consistent stochastic structure of the pyramidal graph provides great flexibilities in designing and analyzing inference algorithms. Based on emerging techniques for inference on Gaussian graphical models, we propose several different inference algorithms to compute not only the optimal estimates but also approximate error variances as well. In addition, we consider the problem of rapidly updating the estimates based on some new local information, and develop a re-estimation algorithm on the pyramidal graph. Simulation results show that this algorithm can be applied to reconstruct discontinuities blurred during the estimation process or to update the estimates to incorporate a new set of measurements introduced in a local region.

Download or read book Markov Random Field Modeling in Image Analysis written by Stan Z. Li and published by Springer Science & Business Media. This book was released on 2009-04-03 with total page 372 pages. Available in PDF, EPUB and Kindle. Book excerpt: Markov random field (MRF) theory provides a basis for modeling contextual constraints in visual processing and interpretation. It enables us to develop optimal vision algorithms systematically when used with optimization principles. This book presents a comprehensive study on the use of MRFs for solving computer vision problems. Various vision models are presented in a unified framework, including image restoration and reconstruction, edge and region segmentation, texture, stereo and motion, object matching and recognition, and pose estimation. This third edition includes the most recent advances and has new and expanded sections on topics such as: Bayesian Network; Discriminative Random Fields; Strong Random Fields; Spatial-Temporal Models; Learning MRF for Classification. This book is an excellent reference for researchers working in computer vision, image processing, statistical pattern recognition and applications of MRFs. It is also suitable as a text for advanced courses in these areas.

Download or read book Computer Vision ECCV 2012 written by Andrew Fitzgibbon and published by Springer. This book was released on 2012-09-26 with total page 913 pages. Available in PDF, EPUB and Kindle. Book excerpt: The seven-volume set comprising LNCS volumes 7572-7578 constitutes the refereed proceedings of the 12th European Conference on Computer Vision, ECCV 2012, held in Florence, Italy, in October 2012. The 408 revised papers presented were carefully reviewed and selected from 1437 submissions. The papers are organized in topical sections on geometry, 2D and 3D shapes, 3D reconstruction, visual recognition and classification, visual features and image matching, visual monitoring: action and activities, models, optimisation, learning, visual tracking and image registration, photometry: lighting and colour, and image segmentation.

Download or read book Algorithms and Applications for Gaussian Graphical Models Under the Multivariate Totally Positive Constraint of Order 2 written by Uma Roy (M. Eng.) and published by . This book was released on 2019 with total page 72 pages. Available in PDF, EPUB and Kindle. Book excerpt: We consider the problem of estimating an undirected Gaussian graphical model when the underlying distribution is multivariate totally positive of order 2 (MTP2), a strong form of positive dependence. A large body of methods have been proposed for learning undirected graphical models without the MTP2 constraint. A major limitation of these methods is that their consistency guarantees in the high-dimensional setting usually require a particular choice of a tuning parameter, which is unknown a priori in real world applications. We show that an undirected graphical model under MTP2 can be learned consistently without any tuning parameters. We evaluate this new estimator on synthetic and real-world financial data sets, showing that it out-performs other methods in the literature with tuning parameters. We further explore applications of estimators in the MTP2 setting to covariance estimation for finance. In particular, the very well-explored optimal Markowitz portfolio allocation problem requires a precise estimate of the covariance matrix of returns. By exploiting the fact that the returns of assets are typically positively dependent, we propose a new estimator based on MTP2 estimation and show that this estimator outperforms (in terms of out-of-sample risk) baseline methods including shrinkage techniques and explicitly providing market factors on stock-market data spanning over thirty years.

Download or read book Distributed and Accelerated Inference Algorithms for Probabilistic Graphical Models written by Arthur Uy Asuncion and published by . This book was released on 2011 with total page 216 pages. Available in PDF, EPUB and Kindle. Book excerpt: Learning graphical models from data is of fundamental importance in machine learning and statistics; however, this task is often computationally challenging due to the complexity of the models and the large scale of the data sets involved. This dissertation presents a variety of distributed and accelerated inference algorithms for probabilistic graphical models. The first part of this dissertation focuses on a class of directed latent variable models known as topic models. We introduce synchronous and asynchronous distributed algorithms for topic models which yield significant time and memory savings without sacrificing accuracy. We also investigate various approximate inference techniques for topic models, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation and find that these methods learn models of similar accuracy as long as hyperparameters are optimized, giving us the freedom to utilize the most computationally efficient algorithm. The second part of this dissertation focuses on accelerated parameter estimation techniques for undirected models such as Boltzmann machines and exponential random graph models. We investigate an efficient blocked contrastive divergence approach that is based on the composite likelihood framework. We also present a particle filtering approach for approximate maximum likelihood estimation that is able to outperform previously proposed estimation algorithms.

Download or read book Modeling and Estimation in Gaussian Graphical Models written by Venkat Chandrasekaran and published by . This book was released on 2007 with total page 172 pages. Available in PDF, EPUB and Kindle. Book excerpt: (cont.) These algorithms involve a sequence of inference problems on tractable subgraphs over subsets of variables. This framework includes parallel iterations such as Embedded Trees, serial iterations such as block Gauss-Seidel, and hybrid versions of these iterations. We also discuss a method that uses local memory at each node to overcome temporary communication failures that may arise in distributed sensor network applications. We analyze these algorithms based on the recently developed walk-sum interpretation of Gaussian inference. We describe the walks "computed" by the algorithms using walk-sum diagrams, and show that for non-stationary iterations based on a very large and flexible set of sequences of subgraphs, convergence is achieved in walk-summable models. Consequently, we are free to choose spanning trees and subsets of variables adaptively at each iteration. This leads to efficient methods for optimizing the next iteration step to achieve maximum reduction in error. Simulation results demonstrate that these non-stationary algorithms provide a significant speedup in convergence over traditional one-tree and two-tree iterations.

Download or read book On Graphical Models for Communications and Machine Learning written by Justin H. G. Dauwels and published by . This book was released on 2006 with total page 489 pages. Available in PDF, EPUB and Kindle. Book excerpt: