Download or read book Multiscale Gaussian Graphical Models and Algorithms for Large scale Inference written by Myung Jin Choi (Ph. D.) and published by . This book was released on 2007 with total page 123 pages. Available in PDF, EPUB and Kindle. Book excerpt: Graphical models provide a powerful framework for stochastic processes by representing dependencies among random variables compactly with graphs. In particular, multiscale tree-structured graphs have attracted much attention for their computational efficiency as well as their ability to capture long-range correlations. However, tree models have limited modeling power that may lead to blocky artifacts. Previous works on extending trees to pyramidal structures resorted to computationally expensive methods to get solutions due to the resulting model complexity. In this thesis, we propose a pyramidal graphical model with rich modeling power for Gaussian processes, and develop efficient inference algorithms to solve large-scale estimation problems. The pyramidal graph has statistical links between pairs of neighboring nodes within each scale as well as between adjacent scales. Although the graph has many cycles, its hierarchical structure enables us to develop a class of fast algorithms in the spirit of multipole methods. The algorithms operate by guiding far-apart nodes to communicate through coarser scales and considering only local interactions at finer scales. The consistent stochastic structure of the pyramidal graph provides great flexibilities in designing and analyzing inference algorithms. Based on emerging techniques for inference on Gaussian graphical models, we propose several different inference algorithms to compute not only the optimal estimates but also approximate error variances as well. In addition, we consider the problem of rapidly updating the estimates based on some new local information, and develop a re-estimation algorithm on the pyramidal graph. Simulation results show that this algorithm can be applied to reconstruct discontinuities blurred during the estimation process or to update the estimates to incorporate a new set of measurements introduced in a local region.

Download or read book Scaling MCMC Inference and Belief Propagation to Large Dense Graphical Models written by Sameer Singh and published by . This book was released on 2014 with total page 185 pages. Available in PDF, EPUB and Kindle. Book excerpt: With the physical constraints of semiconductor-based electronics becoming increasingly limiting in the past decade, single-core CPUs have given way to multi-core and distributed computing platforms. At the same time, access to large data collections is progressively becoming commonplace due to the lowering cost of storage and bandwidth. Traditional machine learning paradigms that have been designed to operate sequentially on single processor architectures seem destined to become obsolete in this world of multi-core, multi-node systems and massive data sets. Inference for graphical models is one such example for which most existing algorithms are sequential in nature and are difficult to scale using parallel computations. Further, modeling large datasets leads to an escalation in the number of variables, factors, domains, and the density of the models, all of which have a substantial impact on the computational and storage complexity of inference. To achieve scalability, existing techniques impose strict independence assumptions on the model, resulting in tractable inference at the expense of expressiveness, and therefore of accuracy and utility, of the model. Motivated by the need to scale inference to large, dense graphical models, in this thesis we explore approximations to Markov chain Monte Carlo (MCMC) and belief propagation (BP) that induce dynamic sparsity in the model to utilize parallelism. In particular, since computations over some factors, variables, and values are more important than over others at different stages of inference, proposed approximations that prioritize and parallelize such computations facilitate efficient inference. First, we show that a synchronously distributed MCMC algorithm that uses dynamic partitioning of the model achieves scalable inference. We then identify bottlenecks in the synchronous architecture, and demonstrate that a collection of MCMC techniques that use asynchronous updates are able to address these drawbacks. For large domains and high-order factors, we find that dynamically inducing sparsity in variable domains, results in scalable belief propagation that enables joint inference. We also show that formulating distributed BP and joint inference as generalized BP on cluster graphs, and by using cluster message approximations, provides significantly lower communication cost and running time. With these tools for inference in hand, we are able to tackle entity tagging, relation extraction, entity resolution, cross-document coreference, joint inference, and other information extraction tasks over large text corpora.

Download or read book Learning in Graphical Models written by M.I. Jordan and published by Springer Science & Business Media. This book was released on 2012-12-06 with total page 658 pages. Available in PDF, EPUB and Kindle. Book excerpt: In the past decade, a number of different research communities within the computational sciences have studied learning in networks, starting from a number of different points of view. There has been substantial progress in these different communities and surprising convergence has developed between the formalisms. The awareness of this convergence and the growing interest of researchers in understanding the essential unity of the subject underlies the current volume. Two research communities which have used graphical or network formalisms to particular advantage are the belief network community and the neural network community. Belief networks arose within computer science and statistics and were developed with an emphasis on prior knowledge and exact probabilistic calculations. Neural networks arose within electrical engineering, physics and neuroscience and have emphasised pattern recognition and systems modelling problems. This volume draws together researchers from these two communities and presents both kinds of networks as instances of a general unified graphical formalism. The book focuses on probabilistic methods for learning and inference in graphical models, algorithm analysis and design, theory and applications. Exact methods, sampling methods and variational methods are discussed in detail. Audience: A wide cross-section of computationally oriented researchers, including computer scientists, statisticians, electrical engineers, physicists and neuroscientists.

Download or read book Machine learning using approximate inference written by Christian Andersson Naesseth and published by Linköping University Electronic Press. This book was released on 2018-11-27 with total page 39 pages. Available in PDF, EPUB and Kindle. Book excerpt: Automatic decision making and pattern recognition under uncertainty are difficult tasks that are ubiquitous in our everyday life. The systems we design, and technology we develop, requires us to coherently represent and work with uncertainty in data. Probabilistic models and probabilistic inference gives us a powerful framework for solving this problem. Using this framework, while enticing, results in difficult-to-compute integrals and probabilities when conditioning on the observed data. This means we have a need for approximate inference, methods that solves the problem approximately using a systematic approach. In this thesis we develop new methods for efficient approximate inference in probabilistic models. There are generally two approaches to approximate inference, variational methods and Monte Carlo methods. In Monte Carlo methods we use a large number of random samples to approximate the integral of interest. With variational methods, on the other hand, we turn the integration problem into that of an optimization problem. We develop algorithms of both types and bridge the gap between them. First, we present a self-contained tutorial to the popular sequential Monte Carlo (SMC) class of methods. Next, we propose new algorithms and applications based on SMC for approximate inference in probabilistic graphical models. We derive nested sequential Monte Carlo, a new algorithm particularly well suited for inference in a large class of high-dimensional probabilistic models. Then, inspired by similar ideas we derive interacting particle Markov chain Monte Carlo to make use of parallelization to speed up approximate inference for universal probabilistic programming languages. After that, we show how we can make use of the rejection sampling process when generating gamma distributed random variables to speed up variational inference. Finally, we bridge the gap between SMC and variational methods by developing variational sequential Monte Carlo, a new flexible family of variational approximations.

Download or read book Graphical Models Exponential Families and Variational Inference written by Martin J. Wainwright and published by Now Publishers Inc. This book was released on 2008 with total page 324 pages. Available in PDF, EPUB and Kindle. Book excerpt: The core of this paper is a general set of variational principles for the problems of computing marginal probabilities and modes, applicable to multivariate statistical models in the exponential family.

Download or read book Graphical Models for Machine Learning and Digital Communication written by Brendan J. Frey and published by MIT Press. This book was released on 1998 with total page 230 pages. Available in PDF, EPUB and Kindle. Book excerpt: Content Description. #Includes bibliographical references and index.

Download or read book Estimation of Graphical Models Convex Formulations and Algorithms written by Jinchao Li and published by . This book was released on 2015 with total page 129 pages. Available in PDF, EPUB and Kindle. Book excerpt: A Gaussian graphical model is a graph representation of conditional independence relations among Gaussian random variables. A fundamental problem in the estimation of Gaussian graphical models is the selection of the graph topology given relatively small amounts of data. This problem is often solved via L1-regularized maximum likelihood estimation, for which many large-scale convex optimization algorithms have been developed. In this thesis, we consider several extensions of Gaussian graphical models and develop fast algorithms based on convex optimization methods. As a first extension, we consider the restricted sparse inverse covariance selection problem where the set of zero entries of the inverse covariance matrix is partially known and an L1-norm penalization is applied to the remaining entries.The proximal Newton method is an attractive algorithm for this problem since the key computations in the algorithm, which include the evaluation of gradient and Hessian of the log-likelihood function, can be implemented efficiently with sparse chordal matrix techniques. We analyze the convergence of the inexact proximal Newton method for the penalized maximum likelihood problem. The convergence analysis applies to a wider class of problems with a self-concordant term in the objective. The numerical results indicate that the method can reach a high accuracy, even with inexact computation of the proximal Newton steps. As a second extension, we consider Gaussian graphical models for time series, with focus on the estimation of multiple time series graphical models with similar graph structures or identical graph structure but different edge coefficients. We formulate a joint estimation method for estimating multiple time series graphical models simultaneously, with a group penalty on the edge coefficients for different models. We apply the Douglas-Rachford algorithm to solve the estimation problem for the joint model, and provide model selection methods for choosing parameters. Both synthetic and real data (fMRI brain activity and international stock markets) examples are provided to demonstrate the advantage of the joint estimation method. The last extension is the generalization of Gaussian graphical models for time series to latent variables. We illustrate the effect of latent variables on the conditional independence structure, and describe a Gaussian graphical model for time series with latent variables. The Douglas-Rachford method is applied to this problem. Simulations with synthetic data demonstrate how the method recovers the graph topology.

Download or read book Hybrid Random Fields written by Antonino Freno and published by Springer Science & Business Media. This book was released on 2011-04-11 with total page 217 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents an exciting new synthesis of directed and undirected, discrete and continuous graphical models. Combining elements of Bayesian networks and Markov random fields, the newly introduced hybrid random fields are an interesting approach to get the best of both these worlds, with an added promise of modularity and scalability. The authors have written an enjoyable book---rigorous in the treatment of the mathematical background, but also enlivened by interesting and original historical and philosophical perspectives. -- Manfred Jaeger, Aalborg Universitet The book not only marks an effective direction of investigation with significant experimental advances, but it is also---and perhaps primarily---a guide for the reader through an original trip in the space of probabilistic modeling. While digesting the book, one is enriched with a very open view of the field, with full of stimulating connections. [...] Everyone specifically interested in Bayesian networks and Markov random fields should not miss it. -- Marco Gori, Università degli Studi di Siena Graphical models are sometimes regarded---incorrectly---as an impractical approach to machine learning, assuming that they only work well for low-dimensional applications and discrete-valued domains. While guiding the reader through the major achievements of this research area in a technically detailed yet accessible way, the book is concerned with the presentation and thorough (mathematical and experimental) investigation of a novel paradigm for probabilistic graphical modeling, the hybrid random field. This model subsumes and extends both Bayesian networks and Markov random fields. Moreover, it comes with well-defined learning algorithms, both for discrete and continuous-valued domains, which fit the needs of real-world applications involving large-scale, high-dimensional data.

Download or read book Distributed and Accelerated Inference Algorithms for Probabilistic Graphical Models written by Arthur Uy Asuncion and published by . This book was released on 2011 with total page 216 pages. Available in PDF, EPUB and Kindle. Book excerpt: Learning graphical models from data is of fundamental importance in machine learning and statistics; however, this task is often computationally challenging due to the complexity of the models and the large scale of the data sets involved. This dissertation presents a variety of distributed and accelerated inference algorithms for probabilistic graphical models. The first part of this dissertation focuses on a class of directed latent variable models known as topic models. We introduce synchronous and asynchronous distributed algorithms for topic models which yield significant time and memory savings without sacrificing accuracy. We also investigate various approximate inference techniques for topic models, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation and find that these methods learn models of similar accuracy as long as hyperparameters are optimized, giving us the freedom to utilize the most computationally efficient algorithm. The second part of this dissertation focuses on accelerated parameter estimation techniques for undirected models such as Boltzmann machines and exponential random graph models. We investigate an efficient blocked contrastive divergence approach that is based on the composite likelihood framework. We also present a particle filtering approach for approximate maximum likelihood estimation that is able to outperform previously proposed estimation algorithms.

Download or read book Provable Algorithms for Learning and Variational Inference in Undirected Graphical Models written by Frederic Koehler and published by . This book was released on 2021 with total page 263 pages. Available in PDF, EPUB and Kindle. Book excerpt: Graphical models are a general-purpose tool for modeling complex distributions in a way which facilitates probabilistic reasoning, with numerous applications across machine learning and the sciences. This thesis deals with algorithmic and statistical problems of learning a high-dimensional graphical model from samples, and related problems of performing inference on a known model, both areas of research which have been the subject of continued interest over the years. Our main contributions are the first computationally efficient algorithms for provably (1) learning a (possibly ill-conditioned) walk-summable Gaussian Graphical Model from samples, (2) learning a Restricted Boltzmann Machine (or other latent variable Ising model) from data, and (3) performing naive mean-field variational inference on an Ising model in the optimal density regime. These different problems illustrate a set of key principles, such as the diverse algorithmic applications of "pinning" variables in graphical models. We also show in some cases that these results are nearly optimal due to matching computational/cryptographic hardness results

Download or read book Large scale Directed Graphical Models Learning written by Gunwoong Park and published by . This book was released on 2016 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Directed graphical models are a powerful statistical method to compactly describe directional or causal relationships among the set of variables in large-scale data. However, a number of statistical and computational challenges arise that make learning directed graphical models often impossible for large-scale data. These issues include: (1) model identifiability; (2) computational guarantee; (3) sample size guarantee; and (4) combining interventional experiments with observational data. In this thesis, we focus on learning directed graphical models by addressing the above four issues. In Chapter 3, we discuss learning Poisson DAG models for modeling large-scale multivariate count data problems where each node is a Poisson random variable conditioning on its parents. We address the question of (1) model identifiability and learning algorithms with (2) computational complexity and (3) sample complexity. We prove that Poisson DAG models are fully identifiable from observational data using the notion of overdispersion, and present a polynomial-time algorithm that learns the Poisson DAG model under suitable regularity conditions. Chapter 4 focuses on learning a broader class of DAG models in large-scale settings. We address the issue of (1) model identifiability and learning algorithms with (2) computational complexity and (3) sample complexity. We introduce a new class of identifiable DAG models which include many interesting classes of distributions such as Poisson, Binomial, Geometric, Exponential, Gamma, and many more, and prove that this class of DAG models is fully identifiable using the idea of overdispersion. Furthermore, we develop statistically consistent and computationally tractable learning algorithms for the new class of identifiable DAG models in high-dimensional settings. Our algorithms exploits the sparsity of the graphs and overdispersion property. Chapter 5 concerns learning general DAG models using a combination of observational and interventional (or experimental) data. Prior work has focused on algorithms using Markov equivalence class (MEC) for the DAG and then using do-calculus rules based on interventions to learn the additional directions. However it has been shown that existing passive and active learning strategies that rely on accurate recovery of the MEC do not scale well to large-scale graphs because recovering MEC for DAG models are not successful large-scale graphs. Hence, we prove (1) model identifiability using the notion of the moralized graphs, and develop passive and active learning algorithms (4) combining interventional experiments with observational data. Lastly in Chapter 6, we concern learning directed cyclic graphical (DCG) models. We focus on (1) model identifiability for directed graphical models with feedback. We provide two new identifiability assumptions with respect to sparsity of a graph and the number of d-separation rules, and compare these new identifiability assumptions to the widely-held faithfulness and minimality assumptions. Furthermore we develop search algorithms for small-scale DCG models based on our new identifiability assumptions.

Download or read book On Graphical Models for Communications and Machine Learning written by Justin H. G. Dauwels and published by . This book was released on 2006 with total page 489 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Probabilistic Graphical Models written by Ying Liu (Ph. D.) and published by . This book was released on 2014 with total page 173 pages. Available in PDF, EPUB and Kindle. Book excerpt: In undirected graphical models, each node represents a random variable while the set of edges specifies the conditional independencies of the underlying distribution. When the random variables are jointly Gaussian, the models are called Gaussian graphical models (GGMs) or Gauss Markov random fields. In this thesis, we address several important problems in the study of GGMs. The first problem is to perform inference or sampling when the graph structure and model parameters are given. For inference in graphs with cycles, loopy belief propagation (LBP) is a purely distributed algorithm, but it gives inaccurate variance estimates in general and often diverges or has slow convergence. Previously, the hybrid feedback message passing (FMP) algorithm was developed to enhance the convergence and accuracy, where a special protocol is used among the nodes in a pseudo-FVS (an FVS, or feedback vertex set, is a set of nodes whose removal breaks all cycles) while standard LBP is run on the subgraph excluding the pseudo-FVS. In this thesis, we develop recursive FMP, a purely distributed extension of FMP where all nodes use the same integrated message-passing protocol. In addition, we introduce the subgraph perturbation sampling algorithm, which makes use of any pre-existing tractable inference algorithm for a subgraph by perturbing this algorithm so as to yield asymptotically exact samples for the intended distribution. We study the stationary version where a single fixed subgraph is used in all iterations, as well as the non-stationary version where tractable subgraphs are adaptively selected. The second problem is to perform model learning, i.e. to recover the underlying structure and model parameters from observations when the model is unknown. Families of graphical models that have both large modeling capacity and efficient inference algorithms are extremely useful. With the development of new inference algorithms for many new applications, it is important to study the families of models that are most suitable for these inference algorithms while having strong expressive power in the new applications. In particular, we study the family of GGMs with small FVSs and propose structure learning algorithms for two cases: 1) All nodes are observed, which is useful in modeling social or flight networks where the FVS nodes often correspond to a small number of high-degree nodes, or hubs, while the rest of the networks is modeled by a tree. 2) The FVS nodes are latent variables, where structure learning is equivalent to decomposing an inverse covariance matrix (exactly or approximately) into the sum of a tree-structured matrix and a low-rank matrix. We perform experiments using synthetic data as well as real data of flight delays to demonstrate the modeling capacity with FVSs of various sizes.

Download or read book Query specific Learning and Inference for Probabilistic Graphical Models written by Anton Chechetka and published by . This book was released on 2011 with total page 205 pages. Available in PDF, EPUB and Kindle. Book excerpt: Abstract: "In numerous real world applications, from sensor networks to computer vision to natural text processing, one needs to reason about the system in question in the face of uncertainty. A key problem in all those settings is to compute the probability distribution over the variables of interest (the query) given the observed values of other random variables (the evidence). Probabilistic graphical models (PGMs) have become the approach of choice for representing and reasoning with high-dimensional probability distributions. However, for most models capable of accurately representing real-life distributions, inference is fundamentally intractable. As a result, optimally balancing the expressive power and inference complexity of the models, as well as designing better approximate inference algorithms, remain important open problems with potential to significantly improve the quality of answers to probabilistic queries. This thesis contributes algorithms for learning and approximate inference in probabilistic graphical models that improve on the state of the art by emphasizing the computational aspects of inference over the representational properties of the models. Our contributions fall into two categories: learning accurate models where exact inference is tractable and speeding up approximate inference by focusing computation on the query variables and only spending as much effort on the remaining parts of the model as needed to answer the query accurately. First, for a case when the set of evidence variables is not known in advance and a single model is needed that can be used to answer any query well, we propose a polynomial time algorithm for learning the structure of tractable graphical models with quality guarantees, including PAC learnability and graceful degradation guarantees. Ours is the first efficient algorithm to provide this type of guarantees. A key theoretical insight of our approach is a tractable upper bound on the mutual information of arbitrarily large sets of random variables that yields exponential speedups over the exact computation. Second, for a setting where the set of evidence variables is known in advance, we propose an approach for learning tractable models that tailors the structure of the model for the particular value of evidence that become known at test time. By avoiding a commitment to a single tractable structure during learning, we are able to expand the representation power of the model without sacrificing efficient exact inference and parameter learning. We provide a general framework that allows one to leverage existing structure learning algorithms for discovering high-quality evidence-specific structures. Empirically, we demonstrate state of the art accuracy on real-life datasets and an order of magnitude speedup. Finally, for applications where the intractable model structure is a given and approximate inference is needed, we propose a principled way to speed up convergence of belief propagation by focusing the computation on the query variables and away from the variables that are of no direct interest to the user. We demonstrate significant speedups over the state of the art on large-scale relational models. Unlike existing approaches, ours does not involve model simplification, and thus has an advantage of converging to the fixed point of the full model. More generally, we argue that the common approach of concentrating on the structure of representation provided by PGMs, and only structuring the computation as representation allows, is suboptimal because of the fundamental computational problems. It is the computation that eventually yields answers to the queries, so directly focusing on structure of computation is a natural direction for improving the quality of the answers. The results of this thesis are a step towards adapting the structure of computation as a foundation of graphical models."

Download or read book Learning in Graphical Models written by M. I. Jordan and published by . This book was released on 2014-01-15 with total page 648 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Modeling and Estimation in Gaussian Graphical Models written by Venkat Chandrasekaran and published by . This book was released on 2007 with total page 172 pages. Available in PDF, EPUB and Kindle. Book excerpt: (cont.) These algorithms involve a sequence of inference problems on tractable subgraphs over subsets of variables. This framework includes parallel iterations such as Embedded Trees, serial iterations such as block Gauss-Seidel, and hybrid versions of these iterations. We also discuss a method that uses local memory at each node to overcome temporary communication failures that may arise in distributed sensor network applications. We analyze these algorithms based on the recently developed walk-sum interpretation of Gaussian inference. We describe the walks "computed" by the algorithms using walk-sum diagrams, and show that for non-stationary iterations based on a very large and flexible set of sequences of subgraphs, convergence is achieved in walk-summable models. Consequently, we are free to choose spanning trees and subsets of variables adaptively at each iteration. This leads to efficient methods for optimizing the next iteration step to achieve maximum reduction in error. Simulation results demonstrate that these non-stationary algorithms provide a significant speedup in convergence over traditional one-tree and two-tree iterations.

Download or read book Advances in Neural Information Processing Systems 16 written by Sebastian Thrun and published by MIT Press. This book was released on 2004 with total page 1694 pages. Available in PDF, EPUB and Kindle. Book excerpt: Papers presented at the 2003 Neural Information Processing Conference by leading physicists, neuroscientists, mathematicians, statisticians, and computer scientists. The annual Neural Information Processing (NIPS) conference is the flagship meeting on neural computation. It draws a diverse group of attendees -- physicists, neuroscientists, mathematicians, statisticians, and computer scientists. The presentations are interdisciplinary, with contributions in algorithms, learning theory, cognitive science, neuroscience, brain imaging, vision, speech and signal processing, reinforcement learning and control, emerging technologies, and applications. Only thirty percent of the papers submitted are accepted for presentation at NIPS, so the quality is exceptionally high. This volume contains all the papers presented at the 2003 conference.