Download or read book Variable Selection and Parameter Estimation Using a Continuous and Differentiable Approximation to the L0 Penalty Function written by Douglas Nielsen VanDerwerken and published by . This book was released on 2011 with total page 48 pages. Available in PDF, EPUB and Kindle. Book excerpt: L0 penalized likelihood procedures like Mallows' Cp, AIC, and BIC directly penalize for the number of variables included in a regression model. This is a straightforward approach to the problem of overfitting, and these methods are now part of every statistician?s repertoire. However, these procedures have been shown to sometimes result in unstable parameter estimates as a result on the L0 penalty?s discontinuity at zero. One proposed alternative, seamless-L0 (SELO), utilizes a continuous penalty function that mimics L0 and allows for stable estimates. Like other similar methods (e.g. LASSO and SCAD), SELO produces sparse solutions because the penalty function is non-differentiable at the origin. Because these penalized likelihoods are singular (non-differentiable) at zero, there is no closed-form solution for the extremum of the objective function. We propose a continuous and everywhere-differentiable penalty function that can have arbitrarily steep slope in a neighborhood near zero, thus mimicking the L0 penalty, but allowing for a nearly closed-form solution for the beta-hat vector. Because our function is not singular at zero, beta-hat will have no zero-valued components, although some will have been shrunk arbitrarily close thereto. We employ a BIC-selected tuning parameter used in the shrinkage step to perform zero-thresholding as well. We call the resulting vector of coefficients the ShrinkSet estimator. It is comparable to SELO in terms of model performance (selecting the truly nonzero coefficients, overall MSE, etc.), but we believe it to be more intuitive and simpler to compute. We provide strong evidence that the estimator enjoys favorable asymptotic properties, including the oracle property.
Download or read book Linear Mixed Model Selection Via Minimum Approximated Information Criterion written by Olivia Abena Atutey and published by . This book was released on 2020 with total page 110 pages. Available in PDF, EPUB and Kindle. Book excerpt: The analyses of correlated, repeated measures, or multilevel data with a Gaussian response are often based on models known as the linear mixed models (LMMs). LMMs are modeled using both fixed effects and random effects. The random intercepts (RI) and random intercepts and slopes (RIS) models are two exceptional cases from the linear mixed models that are taken into consideration. Our primary focus in this dissertation is to propose an approach for simultaneous selection and estimation of fixed effects only in LMMs. This dissertation, inspired by recent research of methods and criteria for model selection, aims to extend a variable selection procedure referred to as minimum approximated information criterion (MIC) of Su et al. (2018). Our contribution presents further use of the MIC for variable selection and sparse estimation in LMMs. Thus, we design a penalized log-likelihood procedure referred to as the minimum approximated information criterion for LMMs (lmmMAIC), which is used to find a parsimonious model that better generalizes data with a group structure. Our proposed lmmMAIC method enforces variable selection and sparse estimation simultaneously by adding a penalty term to the negative log-likelihood of the linear mixed model. The method differs from existing regularized methods mainly due to the penalty parameter and the penalty function.With regards to the penalty function, the lmmMAIC mimics the traditional Bayesian information criterion (BIC)-based best subset selection (BSS) method but requires a continuous or smooth approximation to the L0 norm penalty of BSS. In this context, lmmMAIC performs sparse estimation by optimizing an approximated information criterion, which substantially requires approximating that L0 norm penalty of BSS with a continuous unit dent function. A unit dent function, motivated by bump functions called mollifiers (Friedrichs, 1944), is an even continuous function with a [0, 1] range. Among several unit dent functions, incorporating a hyperbolic tangent function is most preferred. The approximation changes the discrete nature of the L0 norm penalty of BSS to a continuous or smooth one making our method less computationally expensive. Besides, the hyperbolic tangent function has a simple form and it is much easier to compute its derivatives. This shrinkage-based method fits a linear mixed model containing all p predictors instead of comparing and selecting a correct sub-model across 2p candidate models. On this account, the lmmMAIC is feasible for high-dimensional data. The replacement, however, does not enforce sparsity since the hyperbolic tangent function is not singular at its origin. To better handle this issue, a reparameterization trick of the regression coefficients is needed to achieve sparsity.For a finite number of parameters, numerical investigations demonstrated by Shi and Tsai (2002) prove that traditional information criterion (IC)-based procedure like BIC can consistently identify a model. Following these suggestions of consistent variable selection and computational efficiency, we maintain the BIC fixed penalty parameter. Thus, our newly proposed procedure is free of using the frequently applied practices such as generalized cross validation (GCV) in selecting an optimal penalty parameter for our penalized likelihood framework. The lmmMAIC enjoys less computational time compared to other regularization methods.We formulate the lmmMAIC procedure as a smooth optimization problem and seek to solve for the fixed effects parameters by minimizing the penalized log-likelihood function. The implementation of the lmmMAIC involves an initial step of using the simulated annealing algorithm to obtain estimates. We proceed using these estimates as starting values by applying the modified Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm until convergence. After this step, we plug estimates obtained from the modified BFGS into the reparameterized hyperbolic tangent function to obtain our fixed effects estimates. Alternatively, the optimization of the penalized log-likelihood can be solved using generalized simulation annealing.Our research explores the performance and asymptotic properties of the lmmMAIC method by conducting extensive simulation studies using different model settings. The numerical results of our simulations for our proposed variable selection and estimation method are compared to other standard LMMs shrinkage-based methods such as Lasso, ridge, and elastic net. The results provide evidence that lmmMAIC is more consistent and efficient than the existing shrinkage-based methods under study. Furthermore, two applications with real-life examples are illustrated to evaluate the effectiveness of the lmmMAIC method.
Download or read book Variable Selection and Function Estimation Using Penalized Methods written by Ganggang Xu and published by . This book was released on 2012 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Penalized methods are becoming more and more popular in statistical research. This dissertation research covers two major aspects of applications of penalized methods: variable selection and nonparametric function estimation. The following two paragraphs give brief introductions to each of the two topics. Infinite variance autoregressive models are important for modeling heavy-tailed time series. We use a penalty method to conduct model selection for autoregressive models with innovations in the domain of attraction of a stable law indexed by alpha is an element of (0, 2). We show that by combining the least absolute deviation loss function and the adaptive lasso penalty, we can consistently identify the true model. At the same time, the resulting coefficient estimator converges at a rate of n(1/alpha) . The proposed approach gives a unified variable selection procedure for both the finite and infinite variance autoregressive models. While automatic smoothing parameter selection for nonparametric function estimation has been extensively researched for independent data, it is much less so for clustered and longitudinal data. Although leave-subject-out cross-validation (CV) has been widely used, its theoretical property is unknown and its minimization is computationally expensive, especially when there are multiple smoothing parameters. By focusing on penalized modeling methods, we show that leave-subject-out CV is optimal in that its minimization is asymptotically equivalent to the minimization of the true loss function. We develop an efficient Newton-type algorithm to compute the smoothing parameters that minimize the CV criterion. Furthermore, we derive one simplification of the leave-subject-out CV, which leads to a more efficient algorithm for selecting the smoothing parameters. We show that the simplified version of CV criteria is asymptotically equivalent to the unsimplified one and thus enjoys the same optimality property. This CV criterion also provides a completely data driven approach to select working covariance structure using generalized estimating equations in longitudinal data analysis. Our results are applicable to additive, linear varying-coefficient, nonlinear models with data from exponential families.
Download or read book Penalty Shrinkage and Pretest Strategies written by S. Ejaz Ahmed and published by Springer Science & Business Media. This book was released on 2013-12-11 with total page 122 pages. Available in PDF, EPUB and Kindle. Book excerpt: The objective of this book is to compare the statistical properties of penalty and non-penalty estimation strategies for some popular models. Specifically, it considers the full model, submodel, penalty, pretest and shrinkage estimation techniques for three regression models before presenting the asymptotic properties of the non-penalty estimators and their asymptotic distributional efficiency comparisons. Further, the risk properties of the non-penalty estimators and penalty estimators are explored through a Monte Carlo simulation study. Showcasing examples based on real datasets, the book will be useful for students and applied researchers in a host of applied fields. The book’s level of presentation and style make it accessible to a broad audience. It offers clear, succinct expositions of each estimation strategy. More importantly, it clearly describes how to use each estimation strategy for the problem at hand. The book is largely self-contained, as are the individual chapters, so that anyone interested in a particular topic or area of application may read only that specific chapter. The book is specially designed for graduate students who want to understand the foundations and concepts underlying penalty and non-penalty estimation and its applications. It is well-suited as a textbook for senior undergraduate and graduate courses surveying penalty and non-penalty estimation strategies, and can also be used as a reference book for a host of related subjects, including courses on meta-analysis. Professional statisticians will find this book to be a valuable reference work, since nearly all chapters are self-contained.
Download or read book Encyclopedia of Biopharmaceutical Statistics Four Volume Set written by Shein-Chung Chow and published by CRC Press. This book was released on 2018-09-03 with total page 2434 pages. Available in PDF, EPUB and Kindle. Book excerpt: Since the publication of the first edition in 2000, there has been an explosive growth of literature in biopharmaceutical research and development of new medicines. This encyclopedia (1) provides a comprehensive and unified presentation of designs and analyses used at different stages of the drug development process, (2) gives a well-balanced summary of current regulatory requirements, and (3) describes recently developed statistical methods in the pharmaceutical sciences. Features of the Fourth Edition: 1. 78 new and revised entries have been added for a total of 308 chapters and a fourth volume has been added to encompass the increased number of chapters. 2. Revised and updated entries reflect changes and recent developments in regulatory requirements for the drug review/approval process and statistical designs and methodologies. 3. Additional topics include multiple-stage adaptive trial design in clinical research, translational medicine, design and analysis of biosimilar drug development, big data analytics, and real world evidence for clinical research and development. 4. A table of contents organized by stages of biopharmaceutical development provides easy access to relevant topics. About the Editor: Shein-Chung Chow, Ph.D. is currently an Associate Director, Office of Biostatistics, U.S. Food and Drug Administration (FDA). Dr. Chow is an Adjunct Professor at Duke University School of Medicine, as well as Adjunct Professor at Duke-NUS, Singapore and North Carolina State University. Dr. Chow is the Editor-in-Chief of the Journal of Biopharmaceutical Statistics and the Chapman & Hall/CRC Biostatistics Book Series and the author of 28 books and over 300 methodology papers. He was elected Fellow of the American Statistical Association in 1995.
Download or read book Variable Selection and Parameter Estimation for Normal Linear Models written by Peter James Kempthorne and published by . This book was released on 1983 with total page 332 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book Semiparametric Regression written by David Ruppert and published by Cambridge University Press. This book was released on 2003-07-14 with total page 408 pages. Available in PDF, EPUB and Kindle. Book excerpt: Even experts on semiparametric regression should find something new here.
Download or read book Parameter Estimation in Reliability and Life Span Models written by A Clifford Cohen and published by CRC Press. This book was released on 2020-07-26 with total page 312 pages. Available in PDF, EPUB and Kindle. Book excerpt: Offers an applications-oriented treatment of parameter estimation from both complete and censored samples; contains notations, simplified formats for estimates, graphical techniques, and numerous tables and charts allowing users to calculate estimates and analyze sample data quickly and easily. Anno
Download or read book IMPROVING THE ACCURACY OF VARIABLE SELECTION USING THE WHOLE SOLUTION PATH written by Yang Liu and published by . This book was released on 2015 with total page 100 pages. Available in PDF, EPUB and Kindle. Book excerpt: The performances of penalized least squares approaches profoundly depend on the selection of the tuning parameter; however, statisticians did not reach consensus on the criterion for choosing the tuning parameter. Moreover, the penalized least squares estimation that based on a single value of the tuning parameter suffers from several drawbacks. The tuning parameter selected by the traditional selection criteria such as AIC, BIC, CV tends to pick excessive variables, which results in an over-fitting model. On the contrary, many other criteria, such as the extended BIC that favors an over-sparse model, may run the risk of dropping some relevant variables in the model. In the dissertation, a novel approach for the feature selection based on the whole solution paths is proposed, which significantly improves the selection accuracy. The key idea is to partition the variables into the relevant set and the irrelevant set at each tuning parameter, and then select the variables which have been classified as relevant for at least one tuning parameter. The approach is named as Selection by Partitioning the Solution Paths (SPSP). Compared with other existing feature selection approaches, the proposed SPSP algorithm allows feature selection by using a wide class of penalty functions, including Lasso, ridge and other strictly convex penalties. Based on the proposed SPSP procedure, a new type of scores are presented to rank the importance of the variables in the model. The scores, noted as Area-out-of-zero-region Importance Scores (AIS), are defined by the areas between the solution paths and the boundary of the partitions over the whole solution paths. By applying the proposed scores in the stepwise selection, the false positive error of the selection is remarkably reduced. The asymptotic properties for the proposed SPSP estimator have been well established. It is showed that the SPSP estimator is selection consistent when the original estimator is either estimation consistent or selection consistent. Specially, the SPSP approach on the Lasso has been proved to be consistent over the whole solution paths under the irrepresentable condition. Additionally, a number of simulation studies have been conducted to illustrate the performance of the proposed approachs. The comparison between the SPSP algorithm and the existing selection criteria on the Lasso, the adaptive Lasso, the SCAD and the MCP were provided. The results showed the proposed method outperformed the existing variable selection methods in general. Finally, two real data examples of identifying the informative variables in the Boston housing data and the glioblastoma gene expression data are given. Compared with the models selected by other existing approaches, the models selected by the SPSP procedure are much simpler with relatively smaller model errors.
Download or read book Estimation Techniques for Distributed Parameter Systems written by H.T. Banks and published by Springer Science & Business Media. This book was released on 2012-12-06 with total page 328 pages. Available in PDF, EPUB and Kindle. Book excerpt: The research detailed in this monograph was originally motivated by our interest in control problems involving partial and delay differential equations. Our attempts to apply control theory techniques to such prob lems in several areas of science convinced us that in the need for better and more detailed models of distributed/ continuum processes in biology and mechanics lay a rich, interesting, and challenging class of fundamen tal questions. These questions, which involve science and mathematics, are typical of those arising in inverse or parameter estimation problems. Our efforts on inverse problems for distributed parameter systems, which are infinite dimensional in the most common realizations, began about seven years ago at a time when rapid advances in computing capabilities and availability held promise for significant progress in the development of a practically useful as well as theoretically sound methodology for such problems. Much of the research reported in our presentation was not begun when we outlined the plans for this monograph some years ago. By publishing this monograph now, when only a part of the originally intended topics are covered (see Chapter VII in this respect), we hope to stimulate the research and interest of others in an area of scientific en deavor which has exceeded even our optimistic expectations with respect to excitement, opportunity, and stimulation. The computer revolution alluded to above and the development of new codes allow one to solve rather routinely certain estimation problems that would have been out of the question ten years ago.
Download or read book Optimization with Sparsity Inducing Penalties written by Francis Bach and published by . This book was released on 2011-12-23 with total page 124 pages. Available in PDF, EPUB and Kindle. Book excerpt: Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast as convex optimization problems by regularizing the empirical risk with appropriate nonsmooth norms. Optimization with Sparsity-Inducing Penalties presents optimization tools and techniques dedicated to such sparsity-inducing penalties from a general perspective. It covers proximal methods, block-coordinate descent, reweighted ?2-penalized techniques, working-set and homotopy methods, as well as non-convex formulations and extensions, and provides an extensive set of experiments to compare various algorithms from a computational point of view. The presentation of Optimization with Sparsity-Inducing Penalties is essentially based on existing literature, but the process of constructing a general framework leads naturally to new results, connections and points of view. It is an ideal reference on the topic for anyone working in machine learning and related areas.
Download or read book Statistical Foundations of Data Science written by Jianqing Fan and published by CRC Press. This book was released on 2020-09-21 with total page 942 pages. Available in PDF, EPUB and Kindle. Book excerpt: Statistical Foundations of Data Science gives a thorough introduction to commonly used statistical models, contemporary statistical machine learning techniques and algorithms, along with their mathematical insights and statistical theories. It aims to serve as a graduate-level textbook and a research monograph on high-dimensional statistics, sparsity and covariance learning, machine learning, and statistical inference. It includes ample exercises that involve both theoretical studies as well as empirical applications. The book begins with an introduction to the stylized features of big data and their impacts on statistical analysis. It then introduces multiple linear regression and expands the techniques of model building via nonparametric regression and kernel tricks. It provides a comprehensive account on sparsity explorations and model selections for multiple regression, generalized linear models, quantile regression, robust regression, hazards regression, among others. High-dimensional inference is also thoroughly addressed and so is feature screening. The book also provides a comprehensive account on high-dimensional covariance estimation, learning latent factors and hidden structures, as well as their applications to statistical estimation, inference, prediction and machine learning problems. It also introduces thoroughly statistical machine learning theory and methods for classification, clustering, and prediction. These include CART, random forests, boosting, support vector machines, clustering algorithms, sparse PCA, and deep learning.
Download or read book Applied Mechanics Reviews written by and published by . This book was released on 1976 with total page 990 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book Discrete Techniques of Parameter Estimation written by Jerry M. Mendel and published by . This book was released on 1973 with total page 416 pages. Available in PDF, EPUB and Kindle. Book excerpt: Equation error formulation of parameter estimation problems; Least-squares parameter estimation; Minimum-variance parameter estimation; Stochastic-gradient parameter estimation; Estimation of time-varying parameters.
Download or read book Lectures on Adaptive Parameter Estimation written by C. Richard Johnson and published by Prentice Hall. This book was released on 1988 with total page 216 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book SIAM Journal on Control and Optimization written by Society for Industrial and Applied Mathematics and published by . This book was released on 1976 with total page 1210 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book Journal of the American Statistical Association written by and published by . This book was released on 2007 with total page 764 pages. Available in PDF, EPUB and Kindle. Book excerpt: A scientific and educational journal not only for professional statisticians but also for economists, business executives, research directors, government officials, university professors, and others who are seriously interested in the application of statistical methods to practical problems, in the development of more useful methods, and in the improvement of basic statistical data.
