Download or read book Controlled Markov Chains Graphs and Hamiltonicity written by Jerzy A. Filar and published by Now Publishers Inc. This book was released on 2007 with total page 95 pages. Available in PDF, EPUB and Kindle. Book excerpt: "Controlled Markov Chains, Graphs & Hamiltonicity" summarizes a line of research that maps certain classical problems of discrete mathematics--such as the Hamiltonian cycle and the Traveling Salesman problems--into convex domains where continuum analysis can be carried out. (Mathematics)

Download or read book Hamiltonian Cycle Problem and Markov Chains written by Vivek S. Borkar and published by Springer Science & Business Media. This book was released on 2012-04-23 with total page 205 pages. Available in PDF, EPUB and Kindle. Book excerpt: This research monograph summarizes a line of research that maps certain classical problems of discrete mathematics and operations research - such as the Hamiltonian Cycle and the Travelling Salesman Problems - into convex domains where continuum analysis can be carried out. Arguably, the inherent difficulty of these, now classical, problems stems precisely from the discrete nature of domains in which these problems are posed. The convexification of domains underpinning these results is achieved by assigning probabilistic interpretation to key elements of the original deterministic problems. In particular, the approaches summarized here build on a technique that embeds Hamiltonian Cycle and Travelling Salesman Problems in a structured singularly perturbed Markov decision process. The unifying idea is to interpret subgraphs traced out by deterministic policies (including Hamiltonian cycles, if any) as extreme points of a convex polyhedron in a space filled with randomized policies. The above innovative approach has now evolved to the point where there are many, both theoretical and algorithmic, results that exploit the nexus between graph theoretic structures and both probabilistic and algebraic entities of related Markov chains. The latter include moments of first return times, limiting frequencies of visits to nodes, or the spectra of certain matrices traditionally associated with the analysis of Markov chains. However, these results and algorithms are dispersed over many research papers appearing in journals catering to disparate audiences. As a result, the published manuscripts are often written in a very terse manner and use disparate notation, thereby making it difficult for new researchers to make use of the many reported advances. Hence the main purpose of this book is to present a concise and yet easily accessible synthesis of the majority of the theoretical and algorithmic results obtained so far. In addition, the book discusses numerous open questions and problems that arise from this body of work and which are yet to be fully solved. The approach casts the Hamiltonian Cycle Problem in a mathematical framework that permits analytical concepts and techniques, not used hitherto in this context, to be brought to bear to further clarify both the underlying difficulty of NP-completeness of this problem and the relative exceptionality of truly difficult instances. Finally, the material is arranged in such a manner that the introductory chapters require very little mathematical background and discuss instances of graphs with interesting structures that motivated a lot of the research in this topic. More difficult results are introduced later and are illustrated with numerous examples.

Download or read book Analytic Perturbation Theory and Its Applications written by Konstantin E. Avrachenkov and published by SIAM. This book was released on 2013-12-11 with total page 384 pages. Available in PDF, EPUB and Kindle. Book excerpt: Mathematical models are often used to describe complex phenomena such as climate change dynamics, stock market fluctuations, and the Internet. These models typically depend on estimated values of key parameters that determine system behavior. Hence it is important to know what happens when these values are changed. The study of single-parameter deviations provides a natural starting point for this analysis in many special settings in the sciences, engineering, and economics. The difference between the actual and nominal values of the perturbation parameter is small but unknown, and it is important to understand the asymptotic behavior of the system as the perturbation tends to zero. This is particularly true in applications with an apparent discontinuity in the limiting behavior?the so-called singularly perturbed problems. Analytic Perturbation Theory and Its Applications includes a comprehensive treatment of analytic perturbations of matrices, linear operators, and polynomial systems, particularly the singular perturbation of inverses and generalized inverses. It also offers original applications in Markov chains, Markov decision processes, optimization, and applications to Google PageRank? and the Hamiltonian cycle problem as well as input retrieval in linear control systems and a problem section in every chapter to aid in course preparation.

Download or read book Markov Processes and Controlled Markov Chains written by Zhenting Hou and published by Springer Science & Business Media. This book was released on 2013-12-01 with total page 501 pages. Available in PDF, EPUB and Kindle. Book excerpt: The general theory of stochastic processes and the more specialized theory of Markov processes evolved enormously in the second half of the last century. In parallel, the theory of controlled Markov chains (or Markov decision processes) was being pioneered by control engineers and operations researchers. Researchers in Markov processes and controlled Markov chains have been, for a long time, aware of the synergies between these two subject areas. However, this may be the first volume dedicated to highlighting these synergies and, almost certainly, it is the first volume that emphasizes the contributions of the vibrant and growing Chinese school of probability. The chapters that appear in this book reflect both the maturity and the vitality of modern day Markov processes and controlled Markov chains. They also will provide an opportunity to trace the connections that have emerged between the work done by members of the Chinese school of probability and the work done by the European, US, Central and South American and Asian scholars.

Download or read book Selected Topics on Continuous Time Controlled Markov Chains and Markov Games written by Tomás Prieto-Rumeau and published by World Scientific. This book was released on 2012-03-16 with total page 292 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book concerns continuous-time controlled Markov chains, also known as continuous-time Markov decision processes. They form a class of stochastic control problems in which a single decision-maker wishes to optimize a given objective function. This book is also concerned with Markov games, where two decision-makers (or players) try to optimize their own objective function. Both decision-making processes appear in a large number of applications in economics, operations research, engineering, and computer science, among other areas. An extensive, self-contained, up-to-date analysis of basic optimality criteria (such as discounted and average reward), and advanced optimality criteria (e.g., bias, overtaking, sensitive discount, and Blackwell optimality) is presented. A particular emphasis is made on the application of the results herein: algorithmic and computational issues are discussed, and applications to population models and epidemic processes are shown. This book is addressed to students and researchers in the fields of stochastic control and stochastic games. Moreover, it could be of interest also to undergraduate and beginning graduate students because the reader is not supposed to have a high mathematical background: a working knowledge of calculus, linear algebra, probability, and continuous-time Markov chains should suffice to understand the contents of the book. Contents:IntroductionControlled Markov ChainsBasic Optimality CriteriaPolicy Iteration and Approximation TheoremsOvertaking, Bias, and Variance OptimalitySensitive Discount OptimalityBlackwell OptimalityConstrained Controlled Markov ChainsApplicationsZero-Sum Markov GamesBias and Overtaking Equilibria for Markov Games Readership: Graduate students and researchers in the fields of stochastic control and stochastic analysis. Keywords:Markov Decision Processes;Continuous-Time Controlled Markov Chains;Stochastic Dynamic Programming;Stochastic GamesKey Features:This book presents a reader-friendly, extensive, self-contained, and up-to-date analysis of advanced optimality criteria for continuous-time controlled Markov chains and Markov games. Most of the material herein is quite recent (it has been published in high-impact journals during the last five years) and it appears in book form for the first timeThis book introduces approximation theorems which, in particular, allow the reader to obtain numerical approximations of the solution to several control problems of practical interest. To the best of our knowledge, this is the first time that such computational issues are studied for denumerable state continuous-time controlled Markov chains. Hence, the book has an adequate balance between, on the one hand, theoretical results and, on the other hand, applications and computational issuesThe books that analyze continuous-time controlled Markov chains usually restrict themselves to the case of bounded transition and reward rates, which can be reduced to discrete-time models by using the uniformization technique. In our case, however, the transition and the reward rates might be unbounded, and so the uniformization technique cannot be used. By the way, let us mention that in models of practical interest the transition and the reward rates are, typically, unboundedReviews:“The book contains a large number of recent research results on CMCs and Markov games and puts them in perspective. It is written in a very conscious manner, contains detailed proofs of all main results, as well as extensive bibliographic remarks. The book is a very valuable piece of work for researchers on continuous-time CMCs and Markov games.”Zentralblatt MATH

Download or read book Hamiltonian Cycle Problem and Markov Chains written by Springer and published by . This book was released on 2012-04-24 with total page 216 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Markov Chains and Stochastic Stability written by Sean Meyn and published by Cambridge University Press. This book was released on 2009-04-02 with total page 595 pages. Available in PDF, EPUB and Kindle. Book excerpt: Meyn and Tweedie is back! The bible on Markov chains in general state spaces has been brought up to date to reflect developments in the field since 1996 - many of them sparked by publication of the first edition. The pursuit of more efficient simulation algorithms for complex Markovian models, or algorithms for computation of optimal policies for controlled Markov models, has opened new directions for research on Markov chains. As a result, new applications have emerged across a wide range of topics including optimisation, statistics, and economics. New commentary and an epilogue by Sean Meyn summarise recent developments and references have been fully updated. This second edition reflects the same discipline and style that marked out the original and helped it to become a classic: proofs are rigorous and concise, the range of applications is broad and knowledgeable, and key ideas are accessible to practitioners with limited mathematical background.

Download or read book Discrete Time Markov Chains written by G. George Yin and published by Springer Science & Business Media. This book was released on 2005-10-04 with total page 354 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book focuses on two-time-scale Markov chains in discrete time. Our motivation stems from existing and emerging applications in optimization and control of complex systems in manufacturing, wireless communication, and ?nancial engineering. Much of our e?ort in this book is devoted to designing system models arising from various applications, analyzing them via analytic and probabilistic techniques, and developing feasible compu- tionalschemes. Ourmainconcernistoreducetheinherentsystemcompl- ity. Although each of the applications has its own distinct characteristics, all of them are closely related through the modeling of uncertainty due to jump or switching random processes. Oneofthesalientfeaturesofthisbookistheuseofmulti-timescalesin Markovprocessesandtheirapplications. Intuitively,notallpartsorcom- nents of a large-scale system evolve at the same rate. Some of them change rapidly and others vary slowly. The di?erent rates of variations allow us to reduce complexity via decomposition and aggregation. It would be ideal if we could divide a large system into its smallest irreducible subsystems completely separable from one another and treat each subsystem indep- dently. However, this is often infeasible in reality due to various physical constraints and other considerations. Thus, we have to deal with situations in which the systems are only nearly decomposable in the sense that there are weak links among the irreducible subsystems, which dictate the oc- sional regime changes of the system. An e?ective way to treat such near decomposability is time-scale separation. That is, we set up the systems as if there were two time scales, fast vs. slow. xii Preface Followingthetime-scaleseparation,weusesingularperturbationmeth- ology to treat the underlying systems.

Download or read book Mathematical Aspects of Mixing Times in Markov Chains written by Ravi R. Montenegro and published by Now Publishers Inc. This book was released on 2006 with total page 133 pages. Available in PDF, EPUB and Kindle. Book excerpt: Mathematical Aspects of Mixing Times in Markov Chains is a comprehensive, well-written review of the subject that will be of interest to researchers and students in computer and mathematical sciences.

Download or read book Controlled Markov Processes written by Evgeniĭ Borisovich Dynkin and published by Springer. This book was released on 1979 with total page 320 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book is devoted to the systematic exposition of the contemporary theory of controlled Markov processes with discrete time parameter or in another termi nology multistage Markovian decision processes. We discuss the applications of this theory to various concrete problems. Particular attention is paid to mathe matical models of economic planning, taking account of stochastic factors. The authors strove to construct the exposition in such a way that a reader interested in the applications can get through the book with a minimal mathe matical apparatus. On the other hand, a mathematician will find, in the appropriate chapters, a rigorous theory of general control models, based on advanced measure theory, analytic set theory, measurable selection theorems, and so forth. We have abstained from the manner of presentation of many mathematical monographs, in which one presents immediately the most general situation and only then discusses simpler special cases and examples. Wishing to separate out difficulties, we introduce new concepts and ideas in the simplest setting, where they already begin to work. Thus, before considering control problems on an infinite time interval, we investigate in detail the case of the finite interval. Here we first study in detail models with finite state and action spaces-a case not requiring a departure from the realm of elementary mathematics, and at the same time illustrating the most important principles of the theory.

Download or read book Cont Markov Chains written by Borkar and published by CRC Press. This book was released on 1991-04-30 with total page 196 pages. Available in PDF, EPUB and Kindle. Book excerpt: Provides a novel treatment of many problems in controlled Markov chains based on occupation measures and convex analysis. Includes a rederivation of many classical results, a general treatment of the ergodic control problems and an extensive study of the asymptotic behavior of the self-tuning adaptive controller and its variant, the Kumar-Becker-Lin scheme. Also includes a novel treatment of some multiobjective control problems, inaccessible to traditional methods. Annotation copyrighted by Book News, Inc., Portland, OR

Download or read book Finite Markov Chains and Algorithmic Applications written by Olle Häggström and published by Cambridge University Press. This book was released on 2002-05-30 with total page 132 pages. Available in PDF, EPUB and Kindle. Book excerpt: Based on a lecture course given at Chalmers University of Technology, this 2002 book is ideal for advanced undergraduate or beginning graduate students. The author first develops the necessary background in probability theory and Markov chains before applying it to study a range of randomized algorithms with important applications in optimization and other problems in computing. Amongst the algorithms covered are the Markov chain Monte Carlo method, simulated annealing, and the recent Propp-Wilson algorithm. This book will appeal not only to mathematicians, but also to students of statistics and computer science. The subject matter is introduced in a clear and concise fashion and the numerous exercises included will help students to deepen their understanding.

Download or read book Adaptive Markov Control Processes written by Onesimo Hernandez-Lerma and published by Springer Science & Business Media. This book was released on 2012-12-06 with total page 160 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book is concerned with a class of discrete-time stochastic control processes known as controlled Markov processes (CMP's), also known as Markov decision processes or Markov dynamic programs. Starting in the mid-1950swith Richard Bellman, many contributions to CMP's have been made, and applications to engineering, statistics and operations research, among other areas, have also been developed. The purpose of this book is to present some recent developments on the theory of adaptive CMP's, i. e. , CMP's that depend on unknown parameters. Thus at each decision time, the controller or decision-maker must estimate the true parameter values, and then adapt the control actions to the estimated values. We do not intend to describe all aspects of stochastic adaptive control; rather, the selection of material reflects our own research interests. The prerequisite for this book is a knowledgeof real analysis and prob ability theory at the level of, say, Ash (1972) or Royden (1968), but no previous knowledge of control or decision processes is required. The pre sentation, on the other hand, is meant to beself-contained,in the sensethat whenever a result from analysisor probability is used, it is usually stated in full and references are supplied for further discussion, if necessary. Several appendices are provided for this purpose. The material is divided into six chapters. Chapter 1 contains the basic definitions about the stochastic control problems we are interested in; a brief description of some applications is also provided.

Download or read book Topics in Controlled Markov Chains written by Vivek S. Borkar and published by . This book was released on 1991 with total page 191 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Discrete Time Markov Control Processes written by Onesimo Hernandez-Lerma and published by Springer. This book was released on 1995-12-01 with total page 216 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents the first part of a planned two-volume series devoted to a systematic exposition of some recent developments in the theory of discrete-time Markov control processes (MCPs). Interest is mainly confined to MCPs with Borel state and control (or action) spaces, and possibly unbounded costs and noncompact control constraint sets. MCPs are a class of stochastic control problems, also known as Markov decision processes, controlled Markov processes, or stochastic dynamic pro grams; sometimes, particularly when the state space is a countable set, they are also called Markov decision (or controlled Markov) chains. Regardless of the name used, MCPs appear in many fields, for example, engineering, economics, operations research, statistics, renewable and nonrenewable re source management, (control of) epidemics, etc. However, most of the lit erature (say, at least 90%) is concentrated on MCPs for which (a) the state space is a countable set, and/or (b) the costs-per-stage are bounded, and/or (c) the control constraint sets are compact. But curiously enough, the most widely used control model in engineering and economics--namely the LQ (Linear system/Quadratic cost) model-satisfies none of these conditions. Moreover, when dealing with "partially observable" systems) a standard approach is to transform them into equivalent "completely observable" sys tems in a larger state space (in fact, a space of probability measures), which is uncountable even if the original state process is finite-valued.

Download or read book Mathematical Reviews written by and published by . This book was released on 2005 with total page 1156 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Topics in Controlled Markov Chains written by Vivek S. Borkar and published by Longman Sc & Tech. This book was released on 1991 with total page 179 pages. Available in PDF, EPUB and Kindle. Book excerpt: