Download or read book Distributed and Accelerated Inference Algorithms for Probabilistic Graphical Models written by Arthur Uy Asuncion and published by . This book was released on 2011 with total page 216 pages. Available in PDF, EPUB and Kindle. Book excerpt: Learning graphical models from data is of fundamental importance in machine learning and statistics; however, this task is often computationally challenging due to the complexity of the models and the large scale of the data sets involved. This dissertation presents a variety of distributed and accelerated inference algorithms for probabilistic graphical models. The first part of this dissertation focuses on a class of directed latent variable models known as topic models. We introduce synchronous and asynchronous distributed algorithms for topic models which yield significant time and memory savings without sacrificing accuracy. We also investigate various approximate inference techniques for topic models, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation and find that these methods learn models of similar accuracy as long as hyperparameters are optimized, giving us the freedom to utilize the most computationally efficient algorithm. The second part of this dissertation focuses on accelerated parameter estimation techniques for undirected models such as Boltzmann machines and exponential random graph models. We investigate an efficient blocked contrastive divergence approach that is based on the composite likelihood framework. We also present a particle filtering approach for approximate maximum likelihood estimation that is able to outperform previously proposed estimation algorithms.
Download or read book Hybrid Random Fields written by Antonino Freno and published by Springer Science & Business Media. This book was released on 2011-04-11 with total page 217 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents an exciting new synthesis of directed and undirected, discrete and continuous graphical models. Combining elements of Bayesian networks and Markov random fields, the newly introduced hybrid random fields are an interesting approach to get the best of both these worlds, with an added promise of modularity and scalability. The authors have written an enjoyable book---rigorous in the treatment of the mathematical background, but also enlivened by interesting and original historical and philosophical perspectives. -- Manfred Jaeger, Aalborg Universitet The book not only marks an effective direction of investigation with significant experimental advances, but it is also---and perhaps primarily---a guide for the reader through an original trip in the space of probabilistic modeling. While digesting the book, one is enriched with a very open view of the field, with full of stimulating connections. [...] Everyone specifically interested in Bayesian networks and Markov random fields should not miss it. -- Marco Gori, Università degli Studi di Siena Graphical models are sometimes regarded---incorrectly---as an impractical approach to machine learning, assuming that they only work well for low-dimensional applications and discrete-valued domains. While guiding the reader through the major achievements of this research area in a technically detailed yet accessible way, the book is concerned with the presentation and thorough (mathematical and experimental) investigation of a novel paradigm for probabilistic graphical modeling, the hybrid random field. This model subsumes and extends both Bayesian networks and Markov random fields. Moreover, it comes with well-defined learning algorithms, both for discrete and continuous-valued domains, which fit the needs of real-world applications involving large-scale, high-dimensional data.
Download or read book Machine learning using approximate inference written by Christian Andersson Naesseth and published by Linköping University Electronic Press. This book was released on 2018-11-27 with total page 39 pages. Available in PDF, EPUB and Kindle. Book excerpt: Automatic decision making and pattern recognition under uncertainty are difficult tasks that are ubiquitous in our everyday life. The systems we design, and technology we develop, requires us to coherently represent and work with uncertainty in data. Probabilistic models and probabilistic inference gives us a powerful framework for solving this problem. Using this framework, while enticing, results in difficult-to-compute integrals and probabilities when conditioning on the observed data. This means we have a need for approximate inference, methods that solves the problem approximately using a systematic approach. In this thesis we develop new methods for efficient approximate inference in probabilistic models. There are generally two approaches to approximate inference, variational methods and Monte Carlo methods. In Monte Carlo methods we use a large number of random samples to approximate the integral of interest. With variational methods, on the other hand, we turn the integration problem into that of an optimization problem. We develop algorithms of both types and bridge the gap between them. First, we present a self-contained tutorial to the popular sequential Monte Carlo (SMC) class of methods. Next, we propose new algorithms and applications based on SMC for approximate inference in probabilistic graphical models. We derive nested sequential Monte Carlo, a new algorithm particularly well suited for inference in a large class of high-dimensional probabilistic models. Then, inspired by similar ideas we derive interacting particle Markov chain Monte Carlo to make use of parallelization to speed up approximate inference for universal probabilistic programming languages. After that, we show how we can make use of the rejection sampling process when generating gamma distributed random variables to speed up variational inference. Finally, we bridge the gap between SMC and variational methods by developing variational sequential Monte Carlo, a new flexible family of variational approximations.
Download or read book Learning in Graphical Models written by M.I. Jordan and published by Springer Science & Business Media. This book was released on 2012-12-06 with total page 658 pages. Available in PDF, EPUB and Kindle. Book excerpt: In the past decade, a number of different research communities within the computational sciences have studied learning in networks, starting from a number of different points of view. There has been substantial progress in these different communities and surprising convergence has developed between the formalisms. The awareness of this convergence and the growing interest of researchers in understanding the essential unity of the subject underlies the current volume. Two research communities which have used graphical or network formalisms to particular advantage are the belief network community and the neural network community. Belief networks arose within computer science and statistics and were developed with an emphasis on prior knowledge and exact probabilistic calculations. Neural networks arose within electrical engineering, physics and neuroscience and have emphasised pattern recognition and systems modelling problems. This volume draws together researchers from these two communities and presents both kinds of networks as instances of a general unified graphical formalism. The book focuses on probabilistic methods for learning and inference in graphical models, algorithm analysis and design, theory and applications. Exact methods, sampling methods and variational methods are discussed in detail. Audience: A wide cross-section of computationally oriented researchers, including computer scientists, statisticians, electrical engineers, physicists and neuroscientists.
Download or read book Estimation of Distribution Algorithms written by Pedro Larrañaga and published by Springer Science & Business Media. This book was released on 2012-12-06 with total page 398 pages. Available in PDF, EPUB and Kindle. Book excerpt: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation is devoted to a new paradigm for evolutionary computation, named estimation of distribution algorithms (EDAs). This new class of algorithms generalizes genetic algorithms by replacing the crossover and mutation operators with learning and sampling from the probability distribution of the best individuals of the population at each iteration of the algorithm. Working in such a way, the relationships between the variables involved in the problem domain are explicitly and effectively captured and exploited. This text constitutes the first compilation and review of the techniques and applications of this new tool for performing evolutionary computation. Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation is clearly divided into three parts. Part I is dedicated to the foundations of EDAs. In this part, after introducing some probabilistic graphical models - Bayesian and Gaussian networks - a review of existing EDA approaches is presented, as well as some new methods based on more flexible probabilistic graphical models. A mathematical modeling of discrete EDAs is also presented. Part II covers several applications of EDAs in some classical optimization problems: the travelling salesman problem, the job scheduling problem, and the knapsack problem. EDAs are also applied to the optimization of some well-known combinatorial and continuous functions. Part III presents the application of EDAs to solve some problems that arise in the machine learning field: feature subset selection, feature weighting in K-NN classifiers, rule induction, partial abductive inference in Bayesian networks, partitional clustering, and the search for optimal weights in artificial neural networks. Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation is a useful and interesting tool for researchers working in the field of evolutionary computation and for engineers who face real-world optimization problems. This book may also be used by graduate students and researchers in computer science. `... I urge those who are interested in EDAs to study this well-crafted book today.' David E. Goldberg, University of Illinois Champaign-Urbana.
Download or read book Compact Implementation of Distributed Inference Algorithms for Network Monitoring written by Ashima Atul and published by . This book was released on 2008 with total page 180 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book An Introduction to Lifted Probabilistic Inference written by Guy Van den Broeck and published by MIT Press. This book was released on 2021-08-17 with total page 455 pages. Available in PDF, EPUB and Kindle. Book excerpt: Recent advances in the area of lifted inference, which exploits the structure inherent in relational probabilistic models. Statistical relational AI (StaRAI) studies the integration of reasoning under uncertainty with reasoning about individuals and relations. The representations used are often called relational probabilistic models. Lifted inference is about how to exploit the structure inherent in relational probabilistic models, either in the way they are expressed or by extracting structure from observations. This book covers recent significant advances in the area of lifted inference, providing a unifying introduction to this very active field. After providing necessary background on probabilistic graphical models, relational probabilistic models, and learning inside these models, the book turns to lifted inference, first covering exact inference and then approximate inference. In addition, the book considers the theory of liftability and acting in relational domains, which allows the connection of learning and reasoning in relational domains.
Download or read book Improving and Accelerating Particle based Probabilistic Inference written by Michael Hongyu Zhu and published by . This book was released on 2021 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Probabilistic inference is a powerful approach for reasoning under uncertainty that goes beyond point estimation of model parameters to full estimation of the posterior distribution. However, approximating intractable posterior distributions and estimating expectations involving high-dimensional integrals pose algorithmic and computational challenges, especially for large-scale datasets. Two main approaches are sampling-based approaches, such as Markov Chain Monte Carlo (MCMC) and Particle Filters, and optimization-based approaches, like Variational Inference. This thesis presents research on improving and accelerating particle-based probabilistic inference in the areas of MCMC, Particle Filters, Particle-Based Variational Inference, and discrete graphical models. First, we present Sample Adaptive MCMC, a particle-based adaptive MCMC algorithm. We demonstrate how Sample Adaptive MCMC does not require any tuning of the proposal distribution, potentially automating the sampling procedure, and employs global proposals, potentially leading to large speedups over existing MCMC methods. Second, we present a pathwise derivative estimator for Particle Filters including the resampling step. The problem preventing a fully differentiable Particle Filter is the non-differentiability of the discrete particle resampling step. The key idea of our proposed method is to reformulate the Particle Filter algorithm in such a way that eliminates the discrete particle resampling step and makes the reformulated Particle Filter completely continuous and fully differentiable. Third, we propose stochastic variance reduction and quasi-Newton methods for Particle-Based Variational Inference. The insight of our work is that for accurate posterior inference, highly accurate solutions to the Particle-Based Variational Inference optimization problem are needed, so we leverage ideas from large-scale optimization. Lastly, we introduce a meta-algorithm for probabilistic inference in discrete graphical models based on random projections. The key idea is to run approximate inference algorithms for an exponentially large number of samples obtained by random projections. The number of samples used controls the trade-off between the accuracy of the approximate inference algorithm and the variance of the estimator.
Download or read book Computational Social Science written by R. Michael Alvarez and published by Cambridge University Press. This book was released on 2016-03-10 with total page 339 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book provides an overview of cutting-edge approaches to computational social science.
Download or read book Bottom up Approaches to Approximate Inference and Learning in Discrete Graphical Models written by Andrew Edward Gelfand and published by . This book was released on 2014 with total page 217 pages. Available in PDF, EPUB and Kindle. Book excerpt: Probabilistic graphical models offer a convenient and compact way to describe complex and uncertain relationships in data. A graphical model defines a joint probability distribution over many random variables that factors over an underlying graph structure. Unfortunately, inference is generally intractable in graphical models which accurately describe the complex dependencies occurring in real data. In this thesis, we focus on theory and algorithms for learning and approximate inference in graphical models. We propose and investigate a bottom-up approach to inference and learning, where we start with an initial, computationally cheap approximation and then improve upon the initial approximation through additional computation. We study the computation-accuracy trade-off inherent to the bottom-up approach in three different settings. First, we consider the task of finding the most probable (MAP) configuration of a model. We focus on a class of graphical models corresponding to the weighted matching problem - a classic combinatorial optimization problem - and on MAP inference algorithms based on linear programming (LP) relaxations. In this setting, the optimum of the LP relaxation provides an upper bound on the MAP solution to the weighted matching problem that may be quite loose. We thus propose a bottom-up, cutting-plane algorithm which iteratively adds constraints that tighten the upper bound on the matching solution. We then derive a max-product belief propagation algorithm that provably solves the matching problem for certain choices of tightening constraints. Second, we consider the task of computing the marginal probabilities of a model. Loopy Belief Propagation (BP) is an algorithm for obtaining marginal probability estimates by iteratively passing messages between neighboring nodes in a cyclic graphical model. Generalized Belief Propagation (GBP) is a class of approximate inference algorithms that builds upon Loopy BP by passing messages between clusters of nodes. GBP offers the promise to yield marginal estimates that are far more accurate than Loopy BP, but is also very sensitive to the choice of clusters used. We thus propose a criteria - tree-robustness - for choosing the collection of clusters used by GBP that is, in some sense, no worse than Loopy BP when the factors defining our model induce a tree. We propose a method to find a collection of clusters that are tree-robust and empirically demonstrate the effectiveness of the proposed criteria. Third, we consider the task of learning the parameters of a model from data. Maximum likelihood estimation in graphical models is difficult to the intractability of computing the log-partition function and marginals. In surrogate likelihood training, one approximates these quantities using an approximate inference algorithm. We focus on approximate inference methods that utilize a control parameter to trade computation for accuracy and examine when investing more computation leads to more accurate parameter estimates and models that yield more accurate predictions. Surprisingly, we show that it is not always beneficial to increase computation during learning, particularly in data sets containing relatively few observations and also when the model being fit is heavily misspecified. We also expose an interesting bias-variance trade-off between low computation inference methods and high computation inference methods.
Download or read book Sparse Graphical Modeling for High Dimensional Data written by Faming Liang and published by CRC Press. This book was released on 2023-08-02 with total page 151 pages. Available in PDF, EPUB and Kindle. Book excerpt: A general framework for learning sparse graphical models with conditional independence tests Complete treatments for different types of data, Gaussian, Poisson, multinomial, and mixed data Unified treatments for data integration, network comparison, and covariate adjustment Unified treatments for missing data and heterogeneous data Efficient methods for joint estimation of multiple graphical models Effective methods of high-dimensional variable selection Effective methods of high-dimensional inference
Download or read book Query specific Learning and Inference for Probabilistic Graphical Models written by Anton Chechetka and published by . This book was released on 2011 with total page 205 pages. Available in PDF, EPUB and Kindle. Book excerpt: Abstract: "In numerous real world applications, from sensor networks to computer vision to natural text processing, one needs to reason about the system in question in the face of uncertainty. A key problem in all those settings is to compute the probability distribution over the variables of interest (the query) given the observed values of other random variables (the evidence). Probabilistic graphical models (PGMs) have become the approach of choice for representing and reasoning with high-dimensional probability distributions. However, for most models capable of accurately representing real-life distributions, inference is fundamentally intractable. As a result, optimally balancing the expressive power and inference complexity of the models, as well as designing better approximate inference algorithms, remain important open problems with potential to significantly improve the quality of answers to probabilistic queries. This thesis contributes algorithms for learning and approximate inference in probabilistic graphical models that improve on the state of the art by emphasizing the computational aspects of inference over the representational properties of the models. Our contributions fall into two categories: learning accurate models where exact inference is tractable and speeding up approximate inference by focusing computation on the query variables and only spending as much effort on the remaining parts of the model as needed to answer the query accurately. First, for a case when the set of evidence variables is not known in advance and a single model is needed that can be used to answer any query well, we propose a polynomial time algorithm for learning the structure of tractable graphical models with quality guarantees, including PAC learnability and graceful degradation guarantees. Ours is the first efficient algorithm to provide this type of guarantees. A key theoretical insight of our approach is a tractable upper bound on the mutual information of arbitrarily large sets of random variables that yields exponential speedups over the exact computation. Second, for a setting where the set of evidence variables is known in advance, we propose an approach for learning tractable models that tailors the structure of the model for the particular value of evidence that become known at test time. By avoiding a commitment to a single tractable structure during learning, we are able to expand the representation power of the model without sacrificing efficient exact inference and parameter learning. We provide a general framework that allows one to leverage existing structure learning algorithms for discovering high-quality evidence-specific structures. Empirically, we demonstrate state of the art accuracy on real-life datasets and an order of magnitude speedup. Finally, for applications where the intractable model structure is a given and approximate inference is needed, we propose a principled way to speed up convergence of belief propagation by focusing the computation on the query variables and away from the variables that are of no direct interest to the user. We demonstrate significant speedups over the state of the art on large-scale relational models. Unlike existing approaches, ours does not involve model simplification, and thus has an advantage of converging to the fixed point of the full model. More generally, we argue that the common approach of concentrating on the structure of representation provided by PGMs, and only structuring the computation as representation allows, is suboptimal because of the fundamental computational problems. It is the computation that eventually yields answers to the queries, so directly focusing on structure of computation is a natural direction for improving the quality of the answers. The results of this thesis are a step towards adapting the structure of computation as a foundation of graphical models."
Download or read book Graph Representation Learning written by William L. William L. Hamilton and published by Springer Nature. This book was released on 2022-06-01 with total page 141 pages. Available in PDF, EPUB and Kindle. Book excerpt: Graph-structured data is ubiquitous throughout the natural and social sciences, from telecommunication networks to quantum chemistry. Building relational inductive biases into deep learning architectures is crucial for creating systems that can learn, reason, and generalize from this kind of data. Recent years have seen a surge in research on graph representation learning, including techniques for deep graph embeddings, generalizations of convolutional neural networks to graph-structured data, and neural message-passing approaches inspired by belief propagation. These advances in graph representation learning have led to new state-of-the-art results in numerous domains, including chemical synthesis, 3D vision, recommender systems, question answering, and social network analysis. This book provides a synthesis and overview of graph representation learning. It begins with a discussion of the goals of graph representation learning as well as key methodological foundations in graph theory and network analysis. Following this, the book introduces and reviews methods for learning node embeddings, including random-walk-based methods and applications to knowledge graphs. It then provides a technical synthesis and introduction to the highly successful graph neural network (GNN) formalism, which has become a dominant and fast-growing paradigm for deep learning with graph data. The book concludes with a synthesis of recent advancements in deep generative models for graphs—a nascent but quickly growing subset of graph representation learning.
Download or read book Distributed Inference written by Chris Calabrese (M. Eng.) and published by . This book was released on 2013 with total page 97 pages. Available in PDF, EPUB and Kindle. Book excerpt: The study of inference techniques and their use for solving complicated models has taken off in recent years, but as the models we attempt to solve become more complex, there is a worry that our inference techniques will be unable to produce results. Many problems are difficult to solve using current approaches because it takes too long for our implementations to converge on useful values. While coming up with more efficient inference algorithms may be the answer, we believe that an alternative approach to solving this complicated problem involves leveraging the computation power of multiple processors or machines with existing inference algorithms. This thesis describes the design and implementation of such a system by combining a variational inference implementation (Variational Message Passing) with a high-level distributed framework (Graphlab) and demonstrates that inference is performed faster on a few large graphical models when using this system.
Download or read book Exact Algorithms for Probabilistic and Deterministic Graphical Models written by Rina Dechter and published by . This book was released on 2013 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Annotation Graphical models (e.g., Bayesian and constraint networks, influence diagrams, and Markov decision processes) have become a central paradigm for knowledge representation and reasoning in both artificial intelligence and computer science in general. These models are used to perform many reasoning tasks, such as scheduling, planning and learning, diagnosis and prediction, design, hardware and software verification, and bioinformatics. These problems can be stated as the formal tasks of constraint satisfaction and satisfiability, combinatorial optimization, and probabilistic inference. It is well known that the tasks are computationally hard, but research during the past three decades has yielded a variety of principles and techniques that significantly advanced the state of the art. In this book we provide comprehensive coverage of the primary exact algorithms for reasoning with such models. The main feature exploited by the algorithms is the model's graph. We present inference-based, message-passing schemes (e.g., variable-elimination) and search-based, conditioning schemes (e.g., cycle-cutset conditioning and AND/OR search). Each class possesses distinguished characteristics and in particular has different time vs. space behavior. We emphasize the dependence of both schemes on few graph parameters such as the treewidth, cycle-cutset, and (the pseudo-tree) height. We believe the principles outlined here would serve well in moving forward to approximation and anytime-based schemes. The target audience of this book is researchers and students in the artificial intelligence and machine learning area, and beyond.
Download or read book Advancements in Bayesian Methods and Implementations written by and published by Academic Press. This book was released on 2022-10-06 with total page 322 pages. Available in PDF, EPUB and Kindle. Book excerpt: Advancements in Bayesian Methods and Implementation, Volume 47 in the Handbook of Statistics series, highlights new advances in the field, with this new volume presenting interesting chapters on a variety of timely topics, including Fisher Information, Cramer-Rao and Bayesian Paradigm, Compound beta binomial distribution functions, MCMC for GLMMS, Signal Processing and Bayesian, Mathematical theory of Bayesian statistics where all models are wrong, Machine Learning and Bayesian, Non-parametric Bayes, Bayesian testing, and Data Analysis with humans, Variational inference or Functional horseshoe, Generalized Bayes. Provides the authority and expertise of leading contributors from an international board of authors Presents the latest release in the Handbook of Statistics series Updated release includes the latest information on Advancements in Bayesian Methods and Implementation
Download or read book Predicting Structured Data written by Neural Information Processing Systems Foundation and published by MIT Press (MA). This book was released on 2007 with total page 368 pages. Available in PDF, EPUB and Kindle. Book excerpt: State-of-the-art algorithms and theory in a novel domain of machine learning, prediction when the output has structure.
