Download or read book New Efficient Decision making Strategies in Selected Multi armed Bandits Problems written by Chao Tao and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: The stochastic multi-armed bandits (MAB) problems have attracted a lot of attention after Robbins's seminal work. In the simplest form, there are multiple alternative arms. Each arm is associated with an unknown distribution supported on a bounded range. Each time the learner pulls an arm, she will obtain a stochastic reward generated from the distribution associated with that arm. A popularly studied goal of the learner is to find a strategy to obtain as much reward as possible. This model captures well the trade-off between exploration and exploitation as a good learner has to exploit her past experience to select the arm that appears the best and on the other hand, she has to explore seemingly sub-optimal arms to gather more information about them. The above-mentioned ordinary MAB model and its variants have been a hot topic in recent years not only because of their nice mathematical formulations, but also due to that they have been applied to numerous applications such as website optimization, crowdsourcing, assortment optimization, personalized recommendation and so on.In this dissertation, we study and design new efficient decision-making strategies in different variants of the ordinary MAB model including both single-player and multi-player games. In particular, motivated by real-world applications, we investigate best arm identification in linear bandits, thresholding bandits with the goal minimizing the aggregate regret, multinomial logit bandits (MNL-bandit) under risk criteria, and best arm identification in the multi-player MAB. Except for the last problem whose contribution is mainly theoretical, for all the other problems, we provide both theoretical guarantees and empirical simulations to demonstrate the feasibility and efficiency of the proposed algorithms. Many of the proposed ideas and tools are general and can be easily applied to solve other problems.As a by-product, we also develop BanditPyLib, a Python simulation library allowing fast and robust comparison between different bandit algorithms, which may be of independent interest.
Download or read book Regret Analysis of Stochastic and Nonstochastic Multi armed Bandit Problems written by Sébastien Bubeck and published by Now Pub. This book was released on 2012 with total page 138 pages. Available in PDF, EPUB and Kindle. Book excerpt: In this monograph, the focus is on two extreme cases in which the analysis of regret is particularly simple and elegant: independent and identically distributed payoffs and adversarial payoffs. Besides the basic setting of finitely many actions, it analyzes some of the most important variants and extensions, such as the contextual bandit model.
Download or read book Introduction to Multi Armed Bandits written by Aleksandrs Slivkins and published by . This book was released on 2019-10-31 with total page 306 pages. Available in PDF, EPUB and Kindle. Book excerpt: Multi-armed bandits is a rich, multi-disciplinary area that has been studied since 1933, with a surge of activity in the past 10-15 years. This is the first book to provide a textbook like treatment of the subject.
Download or read book Multi armed Bandit Experimental Design written by David Simchi-Levi and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Multi-armed bandit has been well-known for its efficiency in online decision-making in terms of minimizing the loss of the participants' welfare during experiments (i.e., the regret). In clinical trials and many other scenarios, the statistical power of inferring the treatment effects (i.e., the gaps between the mean outcomes of different arms) is also crucial. Nevertheless, minimizing the regret entails harming the statistical power of estimating the treatment effect, since the observations from some arms can be limited. In this paper, we investigate the trade-off between efficiency and statistical power by casting the multi-armed bandit experimental design into a minimax multi-objective optimization problem. We introduce the concept of Pareto optimality to mathematically characterize the situation in which neither the statistical power nor the efficiency can be improved without degrading the other. We derive a useful sufficient and necessary condition for the Pareto optimal solutions to the minimax multi-objective optimization problem. Additionally, we design an effective Pareto optimal multi-armed bandit experiment that can be tailored to different levels of the trade-off between the two objectives. Finally, we extend the design and analysis to the setting where the outcome of each arm consists of an adversarial baseline reward and a stochastic treatment effect, demonstrating the robustness of our design.
Download or read book Bandit Algorithms written by Tor Lattimore and published by Cambridge University Press. This book was released on 2020-07-16 with total page 537 pages. Available in PDF, EPUB and Kindle. Book excerpt: A comprehensive and rigorous introduction for graduate students and researchers, with applications in sequential decision-making problems.
Download or read book Multi armed Bandits in Large scale Complex Systems written by Xiao Xu and published by . This book was released on 2020 with total page 175 pages. Available in PDF, EPUB and Kindle. Book excerpt: This dissertation focuses on the multi-armed bandit problem (MAB) where the objective is a sequential arm selection policy that maximizes the total reward over time. In canonical formulations of MAB, the following assumptions are adopted: the size of the action space is much smaller than the length of the time horizon, computation resources such as memory are unlimited in the learning process, and the generative models of arm rewards are time-invariant. This dissertation aims to relax these assumptions, which are unrealistic in emerging applications involving large-scale complex systems, and develop corresponding techniques to address the resulting new issues. The first part of the dissertation aims to address the issue of a massive number of actions. A stochastic bandit problem with side information on arm similarity and dissimilarity is studied. The main results include a unit interval graph (UIG) representation of the action space that succinctly models the side information and a two-step learning structure that fully exploits the topological structure of the UIG to achieve an optimal scaling of the learning cost with the size of the action space. Specifically, in the UIG representation, each node represents an arm and the presence (absence) of an edge between two nodes indicates similarity (dissimilarity) between their mean rewards. Based on whether the UIG is fully revealed by the side information, two settings with complete and partial side information are considered. For each setting, a two-step learning policy consisting of an offline reduction of the action space and online aggregation of reward observations from similar arms is developed. The computation efficiency and the order optimality of the proposed strategies in terms of the size of the action space and the time length are established. Numerical experiments on both synthetic and real-world datasets are conducted to verify the performance of the proposed policies in practice. In the second part of the dissertation, the issue of limited memory during the learning process is studied in the adversarial bandit setting. Specifically, a learning policy can only store the statistics of a subset of arms summarizing their reward history. A general hierarchical learning structure that trades off the regret order with memory complexity is developed based on multi-level partitions of the arm set into groups and the time horizon into epochs. The proposed learning policy requires only a sublinear order of memory space in terms of the number of arms. Its sublinear regret orders with respect to the time horizon are established for both weak regret and shifting regret in expectation and/or with high probability, when appropriate learning strategies are adopted as subroutines at all levels. By properly choosing the number of levels in the adopted hierarchy, the policy adapts to different sizes of the available memory space. A memory-dependent regret bound is established to characterize the tradeoff between memory complexity and the regret performance of the policy. Numerical examples are provided to verify the performance of the policy. The third part of the dissertation focuses on the issue of time-varying rewards within the contextual bandit framework, which finds applications in various online recommendation systems. The main results include two reward models characterizing the fact that the preferences of users toward different items change asynchronously and distinctly, and a learning algorithm that adapts to the dynamic environment. In particular, the two models assume disjoint and hybrid rewards. In the disjoint setting, the mean reward of playing an arm is determined by an arm-specific preference vector, which is piecewise-stationary with asynchronous change times across arms. In the hybrid setting, the mean reward of an arm also depends on a joint coefficient vector shared by all arms representing the time-invariant component of user interests, in addition to the arm-specific one that is time-varying. Two algorithms based on change detection and restarts are developed in the two settings respectively, of which the performance is verified through simulations on both synthetic and real-world data. Theoretical regret analysis of the algorithm with certain modifications is provided under the disjoint reward model, which shows that a near-optimal regret order in the time length is achieved.
Download or read book Strategic Approach in Multi Criteria Decision Making written by Nolberto Munier and published by Springer. This book was released on 2019-01-29 with total page 276 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book examines multiple criteria decision making (MCDM) and presents the Sequential Interactive Modelling for Urban Systems (SIMUS) as a method to be used for strategic decision making. It emphasizes the necessity to take into account aspects related to real world scenarios and incorporating possible real life aspects for modelling. The book also highlights the use of sensitivity analysis and presents a method for using criteria marginal values instead of weights, which permits the drawing of curves that depicts the variations of the objective function due to variations of these marginal values. In this way it also gives quantitative values of the objective function allowing stakeholders to perform a comprehensive risk analysis for a solution when it is affected by exogenous variables. Strategic Approach in Multi-Criteria Decision Making: A Practical Guide for Complex Scenarios is divided into three parts. Part 1 is devoted to exploring the history and development of the discipline and the way it is currently used. It highlights drawbacks and problems that scholars have identified in different MCDM methods and techniques. Part 2 addresses best practices to assure quality MCDM process. Part 3 introduces the concept of Linear Programming and the proposed SIMUS method as techniques to deal with MCDM. It also includes case studies in order to help document and illustrate difficult concepts, especially related to demands from a scenario and also in their modelling. The decision making process can be a complex task, especially with multi-criteria problems. With large amounts of information, it can be an extremely difficult to make a rational decision, due to the number of intervening variables, their interrelationships, potential solutions that might exist, diverse objectives envisioned for a project, etc. The SIMUS method has been designed to offer a strategy to help organize, classify, and evaluate this information effectively.
Download or read book Advances in Knowledge Discovery and Data Mining written by João Gama and published by Springer Nature. This book was released on 2022-05-09 with total page 677 pages. Available in PDF, EPUB and Kindle. Book excerpt: The 3-volume set LNAI 13280, LNAI 13281 and LNAI 13282 constitutes the proceedings of the 26th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2022, which was held during May 2022 in Chengdu, China. The 121 papers included in the proceedings were carefully reviewed and selected from a total of 558 submissions. They were organized in topical sections as follows: Part I: Data Science and Big Data Technologies, Part II: Foundations; and Part III: Applications.
Download or read book Reinforcement Learning second edition written by Richard S. Sutton and published by MIT Press. This book was released on 2018-11-13 with total page 549 pages. Available in PDF, EPUB and Kindle. Book excerpt: The significantly expanded and updated new edition of a widely used text on reinforcement learning, one of the most active research areas in artificial intelligence. Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives while interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the field's key ideas and algorithms. This second edition has been significantly expanded and updated, presenting new topics and updating coverage of other topics. Like the first edition, this second edition focuses on core online learning algorithms, with the more mathematical material set off in shaded boxes. Part I covers as much of reinforcement learning as possible without going beyond the tabular case for which exact solutions can be found. Many algorithms presented in this part are new to the second edition, including UCB, Expected Sarsa, and Double Learning. Part II extends these ideas to function approximation, with new sections on such topics as artificial neural networks and the Fourier basis, and offers expanded treatment of off-policy learning and policy-gradient methods. Part III has new chapters on reinforcement learning's relationships to psychology and neuroscience, as well as an updated case-studies chapter including AlphaGo and AlphaGo Zero, Atari game playing, and IBM Watson's wagering strategy. The final chapter discusses the future societal impacts of reinforcement learning.
Download or read book Multi armed Bandits for Preference Learning written by Sumeet Katariya and published by . This book was released on 2018 with total page 192 pages. Available in PDF, EPUB and Kindle. Book excerpt: The multi-armed bandit (MAB) problem is one of the simplest instances of sequential or adaptive decision making, in which a learner needs to select options from a given set of alternatives repeatedly in an online manner. More specifically, the agent selects one option at a time, and observes a numerical (and typically noisy) reward signal providing information on the quality of that option, which informs its future selections. This thesis studies adaptive decision making under different circumstances. The first half of the thesis studies learning using pairwise comparisons. The algorithms depend on the objective of the experimenter. We study the objectives of finding the best item, and approximately ranking the given set of items. In the second half of the thesis, we study the problem of learning from user-clicks. A variety of models have been proposed to simulate user behavior on a search-engine results page, and we study learning in cold-start scenarios under two models: the dependent-click model and the position-based model. Finally, if partial prior information about the quality of items is available, we study learning in such warm-start circumstances. In these cases, our algorithm provides the experimenter means to control the exploration of the bandit algorithm. In all cases, we propose algorithms and prove theoretical guarantees about their performance. We also experimentally measure gains with respect to non-adaptive and state-of-the-art adaptive algorithms.
Download or read book Sequential Decision Making in Dynamic Systems written by Yixuan Zhai and published by . This book was released on 2016 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: We study sequential decision-making problems in the presence of uncertainty in dynamic pricing, intrusion detection, and routing in communication networks. A decision maker is usually able to learn from the feedback (observations) in sequential decision-making problems. We consider designing optimal strategies and analyze their performance. In the first part, we consider a dynamic pricing problem under unknown demand models. We start with a monopoly dynamic pricing problem. In this problem, a seller offers prices to a stream of customers and observes either success or failure in each sale attempt. The underlying demand model is unknown to the seller and can take one of M possible forms. We show that this problem can be formulated as a multi-armed bandit with dependent arms. We propose a dynamic pricing policy based on the likelihood ratio test. It is shown that the proposed policy achieves complete learning, i.e. it offers a bounded regret where regret is defined as the revenue loss with respect to the case with a known demand model. This is in sharp contrast with the logarithmic growing regret in multi-armed bandit with independent arms. Later, we consider an oligopoly dynamic pricing problem with a finite uncertainty of demand models. Besides just considering the learning efficiency, we assume that sellers are individually rational and consider strategies within the set of certain kind of equilibria. We formulate the oligopoly problem as a repeated Bertrand game with incomplete information. Two scenarios are investigated, sellers with equal marginal costs or asymmetric marginal cost. For the scenarios with equal marginal costs, we developed a dynamic pricing strategy called Competitive and Cooperative Demand Learning (CCDL). Under CCDL, all sellers would collude and obtain the same average total profit as a monopoly. The strategy is shown to be a subgame perfect Nash equilibrium and Pareto efficient. We further show that the proposed competitive pricing strategy achieves a bounded regret, where regret is defined as the total expected loss in profit with respect to the ideal scenario of a known demand model. For the scenarios with asymmetric marginal costs, a dynamic pricing strategy called Demand Learning under Collusion (DLC) is developed. If sellers are patient enough, a tactic collusion of a subset of sellers may be formed depending on the marginal costs and underlying demand model. Using the limit of means criterion, DLC is shown to be a subgame-perfect and Pareto-efficient equilibrium. The dynamic pricing strategy offers a bounded regret over an infinite horizon. Using discounting criterion, DLC is shown to be subgame-perfect [epsilon]-equilibrium, [epsilon]-efficient and with an arbitrarily small regret. The dual problem as an infinitely repeated Cournot competition is formulated and the economic efficiency measured by the social welfare is discussed between Bertrand and Cournot formulations. In the second part, we consider an intrusion detection problem and formulate it as a dynamic search of a target located in one of K cells with any fixed number of searches. At each time, one cell is searched, and the search result is subject to false alarms. The objective is a policy that governs the sequential selection of the cells to minimize the error probability of detecting the whereabouts of the target within a fixed time horizon. We show that the optimal search policy is myopic in nature with a simple structure. In the third part, we consider the shortest path routing problem in a communication network with random link costs drawn from unknown distributions. A realization of the total end-to-end cost is obtained when a path is selected for communication. The objective is an online learning algorithm that minimizes the total expected communication cost in the long run. The problem is formulated as a multi-armed bandit problem with dependent arms, and an algorithm based on basis-based learning integrated with a Best Linear Unbiased Estimator (BLUE) is developed.
Download or read book Foundations and Applications of Sensor Management written by Alfred Olivier Hero and published by Springer Science & Business Media. This book was released on 2007-10-23 with total page 317 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book covers control theory signal processing and relevant applications in a unified manner. It introduces the area, takes stock of advances, and describes open problems and challenges in order to advance the field. The editors and contributors to this book are pioneers in the area of active sensing and sensor management, and represent the diverse communities that are targeted.
Download or read book From Bandits to Monte Carlo Tree Search written by Rmi Munos and published by Now Pub. This book was released on 2014 with total page 146 pages. Available in PDF, EPUB and Kindle. Book excerpt: Covers the optimism in the face of uncertainty principle applied to large scale optimization problems under finite numerical budget. The initial motivation for this research originated from the empirical success of the Monte-Carlo Tree Search method popularized in Computer Go and further extended to other games, optimization, and planning problems.
Download or read book Distributed Control of Robotic Networks written by Francesco Bullo and published by Princeton University Press. This book was released on 2009-07-06 with total page 320 pages. Available in PDF, EPUB and Kindle. Book excerpt: This self-contained introduction to the distributed control of robotic networks offers a distinctive blend of computer science and control theory. The book presents a broad set of tools for understanding coordination algorithms, determining their correctness, and assessing their complexity; and it analyzes various cooperative strategies for tasks such as consensus, rendezvous, connectivity maintenance, deployment, and boundary estimation. The unifying theme is a formal model for robotic networks that explicitly incorporates their communication, sensing, control, and processing capabilities--a model that in turn leads to a common formal language to describe and analyze coordination algorithms. Written for first- and second-year graduate students in control and robotics, the book will also be useful to researchers in control theory, robotics, distributed algorithms, and automata theory. The book provides explanations of the basic concepts and main results, as well as numerous examples and exercises. Self-contained exposition of graph-theoretic concepts, distributed algorithms, and complexity measures for processor networks with fixed interconnection topology and for robotic networks with position-dependent interconnection topology Detailed treatment of averaging and consensus algorithms interpreted as linear iterations on synchronous networks Introduction of geometric notions such as partitions, proximity graphs, and multicenter functions Detailed treatment of motion coordination algorithms for deployment, rendezvous, connectivity maintenance, and boundary estimation
Download or read book EVOLVE A Bridge between Probability Set Oriented Numerics and Evolutionary Computation VI written by Alexandru-Adrian Tantar and published by Springer. This book was released on 2017-11-09 with total page 233 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book comprises selected research papers from the 2015 edition of the EVOLVE conference, which was held on June 18–June 24, 2015 in Iași, Romania. It presents the latest research on Probability, Set Oriented Numerics, and Evolutionary Computation. The aim of the EVOLVE conference was to provide a bridge between probability, set oriented numerics and evolutionary computation and to bring together experts from these disciplines. The broad focus of the EVOLVE conference made it possible to discuss the connection between these related fields of study computational science. The selected papers published in the proceedings book were peer reviewed by an international committee of reviewers (at least three reviews per paper) and were revised and enhanced by the authors after the conference. The contributions are categorized into five major parts, which are: Multicriteria and Set-Oriented Optimization; Evolution in ICT Security; Computational Game Theory; Theory on Evolutionary Computation; Applications of Evolutionary Algorithms. The 2015 edition shows a major progress in the aim to bring disciplines together and the research on a number of topics that have been discussed in previous editions of the conference matured over time and methods have found their ways in applications. In this sense the book can be considered an important milestone in bridging and thereby advancing state-of-the-art computational methods.
Download or read book Bandit problems written by Donald A. Berry and published by Springer Science & Business Media. This book was released on 2013-04-17 with total page 283 pages. Available in PDF, EPUB and Kindle. Book excerpt: Our purpose in writing this monograph is to give a comprehensive treatment of the subject. We define bandit problems and give the necessary foundations in Chapter 2. Many of the important results that have appeared in the literature are presented in later chapters; these are interspersed with new results. We give proofs unless they are very easy or the result is not used in the sequel. We have simplified a number of arguments so many of the proofs given tend to be conceptual rather than calculational. All results given have been incorporated into our style and notation. The exposition is aimed at a variety of types of readers. Bandit problems and the associated mathematical and technical issues are developed from first principles. Since we have tried to be comprehens ive the mathematical level is sometimes advanced; for example, we use measure-theoretic notions freely in Chapter 2. But the mathema tically uninitiated reader can easily sidestep such discussion when it occurs in Chapter 2 and elsewhere. We have tried to appeal to graduate students and professionals in engineering, biometry, econ omics, management science, and operations research, as well as those in mathematics and statistics. The monograph could serve as a reference for professionals or as a telA in a semester or year-long graduate level course.
Download or read book Quantum Continuous Variables written by Alessio Serafini and published by CRC Press. This book was released on 2017-07-20 with total page 258 pages. Available in PDF, EPUB and Kindle. Book excerpt: Quantum Continuous Variables introduces the theory of continuous variable quantum systems, from its foundations based on the framework of Gaussian states to modern developments, including its applications to quantum information and forthcoming quantum technologies. This new book addresses the theory of Gaussian states, operations, and dynamics in great depth and breadth, through a novel approach that embraces both the Hilbert space and phase descriptions. The volume includes coverage of entanglement theory and quantum information protocols, and their connection with relevant experimental set-ups. General techniques for non-Gaussian manipulations also emerge as the treatment unfolds, and are demonstrated with specific case studies. This book will be of interest to graduate students looking to familiarise themselves with the field, in addition to experienced researchers eager to enhance their understanding of its theoretical methods. It will also appeal to experimentalists searching for a rigorous but accessible treatment of the theory in the area.
