52 results found
Gandy A, Hahn G, Ding D, Implementing Monte Carlo tests with P-value buckets, Scandinavian Journal of Statistics: theory and applications, ISSN: 0303-6898
Software packages usually report the results of statistical tests usingp-values. Users often interpret these by comparing them to standard thresholds,e.g. 0.1%, 1% and 5%, which is sometimes reinforced by a star rating (***, **,*). We consider an arbitrary statistical test whose p-value p is not availableexplicitly, but can be approximated by Monte Carlo samples, e.g. by bootstrapor permutation tests. The standard implementation of such tests usually draws afixed number of samples to approximate p. However, the probability that theexact and the approximated p-value lie on different sides of a threshold (theresampling risk) can be high, particularly for p-values close to a threshold.We present a method to overcome this. We consider a finite set ofuser-specified intervals which cover [0,1] and which can be overlapping. Wecall these p-value buckets. We present algorithms that, with arbitrarily highprobability, return a p-value bucket containing p. We prove that for both abounded resampling risk and a finite runtime, overlapping buckets need to beemployed, and that our methods both bound the resampling risk and guarantee afinite runtime for such overlapping buckets. To interpret decisions withoverlapping buckets, we propose an extension of the star rating system. Wedemonstrate that our methods are suitable for use in standard software,including for low p-value thresholds occurring in multiple testing settings,and that they can be computationally more efficient than standardimplementations.
Ding D, Gandy A, Hahn G, 2019, A simple method for implementing Monte Carlo tests, Computational Statistics, Pages: 1-20, ISSN: 0943-4062
We consider a statistical test whose p value can only be approximated using Monte Carlo simulations. We are interested in deciding whether the p value for an observed data set lies above or below a given threshold such as 5%. We want to ensure that the resampling risk, the probability of the (Monte Carlo) decision being different from the true decision, is uniformly bounded. This article introduces a simple open-ended method with this property, the confidence sequence method (CSM). We compare our approach to another algorithm, SIMCTEST, which also guarantees an (asymptotic) uniform bound on the resampling risk, as well as to other Monte Carlo procedures without a uniform bound. CSM is free of tuning parameters and conservative. It has the same theoretical guarantee as SIMCTEST and, in many settings, similar stopping boundaries. As it is much simpler than other methods, CSM is a useful method for practical applications.
Hilbers A, Brayshaw D, Gandy A, 2019, Importance subsampling: Improving power system planning under climate-based uncertainty, Applied Energy, Vol: 251, Pages: 1-12, ISSN: 0306-2619
Recent studies indicate that the effects of inter-annual climate-based variability in power system planning are significant and that long samples of demand & weather data (spanning multiple decades) should be considered. At the same time, modelling renewable generation such as solar and wind requires high temporal resolution to capture fluctuations in output levels. In many realistic power system models, using long samples at high temporal resolution is computationally unfeasible. This paper introduces a novel subsampling approach, referred to as importance subsampling, allowing the use of multiple decades of demand & weather data in power system planning models at reduced computational cost. The methodology can be applied in a wide class of optimisation-based power system simulations. A test case is performed on a model of the United Kingdom created using the open-source modelling framework Calliope and 36 years of hourly demand and wind data. Standard data reduction approaches such as using individual years or clustering into representative days lead to significant errors in estimates of optimal system design. Furthermore, the resultant power systems lead to supply capacity shortages, raising questions of generation capacity adequacy. In contrast, importance subsampling leads to accurate estimates of optimal system design at greatly reduced computational cost, with resultant power systems able to meet demand across all 36 years of demand & weather scenarios.
Scott J, Gandy A, State-dependent Kernel selection for conditional sampling of graphs, Journal of Computational and Graphical Statistics, ISSN: 1061-8600
This article introduces new efficient algorithms for two problems: sampling conditional onvertex degrees in unweighted graphs, and conditional on vertex strengths in weighted graphs.The resulting conditional distributions provide the basis for exact tests on social networks andtwo-way contingency tables. The algorithms are able to sample conditional on the presenceor absence of an arbitrary set of edges. Existing samplers based on MCMC or sequentialimportance sampling are generally not scalable; their efficiency can degrade in large graphswith complex patterns of known edges. MCMC methods usually require explicit computationof a Markov basis to navigate the state space; this is computationally intensive even for smallgraphs. Our samplers do not require a Markov basis, and are efficient both in sparse and densesettings. The key idea is to carefully select a Markov kernel on the basis of the current state ofthe chain. We demonstrate the utility of our methods on a real network and contingency table.Supplementary materials for this article are available online.
Veraart LAM, Gandy A, 2019, Adjustable network reconstruction with applications to CDS exposures, Journal of Multivariate Analysis, Vol: 172, Pages: 193-209, ISSN: 0047-259X
This paper is concerned with reconstructing weighted directed networks from the total in- and out-weight of each node. This problem arises for example in the analysis of systemic risk of partially observed financial networks. Typically a wide range of networks is consistent with this partial information. We develop an empirical Bayesian methodology that can be adjusted such that the resulting networks are consistent with the observations and satisfy certain desired global topological properties such as a given mean density, extending the approach by Gandy and Veraart (2017). Furthermore we propose a new fitness-based model within this framework. We provide a case study based on a data set consisting of 89 fully observed financial networks of credit default swap exposures. We reconstruct those networks based on only partial information using the newly proposed as well as existing methods. To assess the quality of the reconstruction, we use a wide range of criteria, including measures on how well the degree distribution can be captured and higher order measures of systemic risk. We find that the empirical Bayesian approach performs best.
Noven R, Veraart A, Gandy A, 2018, A latent trawl process model for extreme values, Journal of Energy Markets, Vol: 11, Pages: 1-24, ISSN: 1756-3607
This paper presents a new model for characterising temporaldependence in exceedancesabove a threshold. The model is based on the class of trawl processes, which are stationary,infinitely divisible stochastic processes. The model for extreme values is constructed byembedding a trawl process in a hierarchical framework, which ensures that the marginaldistribution is generalised Pareto, as expected from classical extreme value theory. Wealso consider a modified version of this model that works witha wider class of generalisedPareto distributions, and has the advantage of separating marginal and temporal depen-dence properties. The model is illustrated by applicationsto environmental time series,and it is shown that the model offers considerable flexibilityin capturing the dependencestructure of extreme value data
Gandy A, Veraart LAM, 2017, A Bayesian methodology for systemic risk assessment in financial networks, Management Science, Vol: 63, Pages: 4428-4446, ISSN: 1526-5501
We develop a Bayesian methodology for systemic risk assessment in financial networks such as theinterbank market. Nodes represent participants in the network and weighted directed edges representliabilities. Often, for every participant, only the total liabilities and total assets within this network areobservable. However, systemic risk assessment needs the individual liabilities. We propose a modelfor the individual liabilities, which, following a Bayesian approach, we then condition on the observedtotal liabilities and assets and, potentially, on certain observed individual liabilities. We construct aGibbs sampler to generate samples from this conditional distribution. These samples can be used instress testing, giving probabilities for the outcomes of interest. As one application we derive defaultprobabilities of individual banks and discuss their sensitivity with respect to prior information includedto model the network. An R-package implementing the methodology is provided.
Gandy A, Kvaløy JT, 2017, spcadjust: an R package for adjusting for estimation error in control charts, The R Journal, Vol: 9, Pages: 458-476, ISSN: 2073-4859
In practical applications of control charts the in-control state and the corresponding chartparameters are usually estimated based on some past in-control data. The estimation error thenneeds to be accounted for. In this paper we present an R package,spcadjust, which implements abootstrap based method for adjusting monitoring schemes to take into account the estimation error.By bootstrapping the past data this method guarantees, with a certain probability, a conditionalperformance of the chart. Inspcadjustthe method is implement for various types of Shewhart,CUSUM and EWMA charts, various performance criteria, and both parametric and non-parametricbootstrap schemes. In addition to the basic charts, charts based on linear and logistic regressionmodels for risk adjusted monitoring are included, and it is easy for the user to add further charts. Useof the package is demonstrated by examples.
Lau FDH, Gandy A, 2016, Enhancing football league tables, Significance, Vol: 13, Pages: 8-9, ISSN: 1740-9705
League tables are commonly used to represent the current state of a competition, in football and other sports. But they do not tell the full story. F. Din-Houn Lau and Axel Gandy suggest a few improvements.
Gandy A, Lau F, 2016, The chopthin algorithm for resampling, IEEE Transactions on Signal Processing, Vol: 64, Pages: 4273-4281, ISSN: 1941-0476
Resampling is a standard step in particle filters andmore generally sequential Monte Carlo methods. Wepresent an algorithm, called chopthin, for resamplingweighted particles. In contrast to standard resamplingmethods the algorithm does not produce a set ofequally weighted particles; instead it merely enforcesan upper bound on the ratio between the weights.Simulation studies show that the chopthin algorithmconsistently outperforms standard resampling methods.The algorithms chops up particles with largeweight and thins out particles with low weight, henceits name. It implicitly guarantees a lower bound onthe effective sample size. The algorithm can be implementedefficiently, making it practically useful. Weshow that the expected computational effort is linearin the number of particles. Implementations for C++,R (on CRAN), Python and Matlab are available.
Gandy A, Hahn G, 2016, QuickMMCTest -- quick multiple Monte Carlo testing, Statistics and Computing, Vol: 27, Pages: 823-832, ISSN: 1573-1375
Multiple hypothesis testing is widely used to evaluate scientific studiesinvolving statistical tests. However, for many of these tests, p-values are notavailable and are thus often approximated using Monte Carlo tests such aspermutation tests or bootstrap tests. This article presents a simple algorithmbased on Thompson Sampling to test multiple hypotheses. It works with arbitrarymultiple testing procedures, in particular with step-up and step-downprocedures. Its main feature is to sequentially allocate Monte Carlo effort,generating more Monte Carlo samples for tests whose decisions are so far lesscertain. A simulation study demonstrates that for a low computational effort,the new approach yields a higher power and a higher degree of reproducibilityof its results than previously suggested methods.
Gandy A, Hahn G, 2016, A framework for Monte Carlo based multiple testing, Scandinavian Journal of Statistics, Vol: 43, Pages: 1046-1063, ISSN: 1467-9469
We are concerned with multiple testing in the setting where p-values areunknown and can only be approximated using Monte Carlo simulation. Thisscenario occurs widely in practice. We are interested in obtaining the samerejections and non-rejections as the ones obtained if the p-values for allhypotheses had been available. The present article introduces a framework forthis scenario by providing a generic algorithm for a general multiple testingprocedure. We establish conditions which guarantee that the rejections andnon-rejections obtained through Monte Carlo simulations are identical to theones obtained with the p-values. Our framework is applicable to a general classof step-up and step-down procedures which includes many established multipletesting corrections such as the ones of Bonferroni, Holm, Sidak, Hochberg orBenjamini-Hochberg. Moreover, we show how to use our framework to improvealgorithms available in the literature in such a way as to yield theoreticalguarantees on their results. These modifications can easily be implemented inpractice and lead to a particular way of reporting multiple testing results asthree sets together with an error bound on their correctness, demonstratedexemplarily using a real biological dataset.
Gandy A, Hahn G, 2014, MMCTest-A Safe Algorithm for Implementing Multiple Monte Carlo Tests, SCANDINAVIAN JOURNAL OF STATISTICS, Vol: 41, Pages: 1083-1101, ISSN: 0303-6898
Phinikettos I, Gandy A, 2014, An omnibus CUSUM chart for monitoring time to event data, LIFETIME DATA ANALYSIS, Vol: 20, Pages: 481-494, ISSN: 1380-7870
Lau FD-H, Gandy A, 2014, RMCMC: A system for updating Bayesian models, Computational Statistics & Data Analysis, Vol: 80, Pages: 99-110, ISSN: 0167-9473
A system to update estimates from a sequence of probability distributions is presented. The aim of the system is to quickly produce estimates with a user-specified bound on the Monte Carlo error. The estimates are based upon weighted samples stored in a database. The stored samples are maintained such that the accuracy of the estimates and quality of the samples are satisfactory. This maintenance involves varying the number of samples in the database and updating their weights. New samples are generated, when required, by a Markov chain Monte Carlo algorithm. The system is demonstrated using a football league model that is used to predict the end of season table. The correctness of the estimates and their accuracy are shown in a simulation using a linear Gaussian model.
Noven RC, Veraart AED, Gandy A, 2014, A Levy-driven rainfall model with applications to futures pricing
Lau FD-H, Gandy A, 2013, Optimality of Non-Restarting CUSUM Charts, SEQUENTIAL ANALYSIS-DESIGN METHODS AND APPLICATIONS, Vol: 32, Pages: 458-468, ISSN: 0747-4946
Gandy A, Kvaløy JT, 2013, Guaranteed Conditional Performance of Control Charts via Bootstrap Methods, Scandinavian Journal of Statistics, Vol: n/a, ISSN: 0303-6898
To use control charts in practice, the in-control state usually has to beestimated. This estimation has a detrimental effect on the performance ofcontrol charts, which is often measured for example by the false alarmprobability or the average run length. We suggest an adjustment of themonitoring schemes to overcome these problems. It guarantees, with a certainprobability, a conditional performance given the estimated in-control state.The suggested method is based on bootstrapping the data used to estimate thein-control state. The method applies to different types of control charts, andalso works with charts based on regression models, survival models, etc. If anonparametric bootstrap is used, the method is robust to model errors. We showlarge sample properties of the adjustment. The usefulness of our approach isdemonstrated through simulation studies.
Gandy A, Lau FD-H, 2013, Non-restarting cumulative sum charts and control of the false discovery rate, BIOMETRIKA, Vol: 100, Pages: 261-268, ISSN: 0006-3444
Henrion M, Mortlock DJ, Hand DJ, et al., 2013, Classification and Anomaly Detection for Astronomical Survey Data, Springer Series in Astrostatistics, Pages: 149-184, ISBN: 9781461435075
© Springer Science+Business Media New York 2013. We present two statistical techniques for astronomical problems: a star-galaxy separator for the UKIRT Infrared Deep Sky Survey (UKIDSS) and a novel anomaly detection method for cross-matched astronomical datasets. The star-galaxy separator is a statistical classification method which outputs class membership probabilities rather than class labels and allows the use of prior knowledge about the source populations. Deep Sloan Digital Sky Survey (SDSS) data from the multiply imaged Stripe 82 region are used to check the results from our classifier, which compares favourably with the UKIDSS pipeline classification algorithm. The anomaly detection method addresses the problem posed by objects having different sets of recorded variables in cross-matched datasets. This prevents the use of methods unable to handle missing values and makes direct comparison between objects difficult. For each source, our method computes anomaly scores in subspaces of the observed feature space and combines them to an overall anomaly score. The proposed technique is very general and can easily be used in applications other than astronomy. The properties and performance of our method are investigated using both real and simulated datasets.
Lee MLT, Gail M, Pfeiffer R, et al., 2013, Preface, Pages: V-VI, ISSN: 0930-0325
Gandy A, Trotta R, 2013, Special Issue on Astrostatistics, STATISTICAL ANALYSIS AND DATA MINING, Vol: 6, Pages: 1-+, ISSN: 1932-1864
Henrion M, Hand DJ, Gandy A, et al., 2013, CASOS: a Subspace Method for Anomaly Detection in High Dimensional Astronomical Databases, STATISTICAL ANALYSIS AND DATA MINING, Vol: 6, Pages: 53-72, ISSN: 1932-1864
Gandy A, Veraart L, 2012, THE EFFECT OF ESTIMATION IN HIGH-DIMENSIONAL PORTFOLIOS, Mathematical Finance
Ashby D, Bird SM, Hunt I, et al., 2012, Discussion on the paper by Spiegelhalter, Sherlaw-Johnson, Bardsley, Blunt, Wood and Grigg, JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES A-STATISTICS IN SOCIETY, Vol: 175, Pages: 25-47, ISSN: 0964-1998
Gandy A, 2012, Performance monitoring of credit portfolios using survival analysis, International Journal of Forecasting, Vol: 28, Pages: 139-144, ISSN: 0169-2070
Gandy A, Rubin-Delanchy P, 2011, An algorithm to compute the power of Monte Carlo tests with guaranteed precision
This article presents an algorithm that generates an exact (conservative)confidence interval of a specified length and coverage probability for thepower of a Monte Carlo test (such as a bootstrap or permutation test). It isthe first method that achieves this aim for almost any Monte Carlo test. Theexisting research on power estimation for Monte Carlo tests has focused onobtaining as accurate a result as possible for a fixed computational effort.However, the methods proposed do not provide any guarantee of precision, in thesense that they cannot report a confidence interval to accompany their estimateof the power. Conversely in this article the computational effort is random.The algorithm operates until a confidence interval can be constructed thatmeets the requirements of the user, in terms of length and coverageprobability. We show that, surprisingly, by generating two more datasets thatwhat might have been assumed to be sufficient, the expected number of stepsrequired by the algorithm is finite in many cases of practical interest. Theseinclude, for instance, any situation where the distribution of the p-value isabsolutely continuous or if it is discrete with finite support. The algorithmis implemented in the R package simctest.
Star–galaxy classification is one of the most fundamental data-processing tasks in survey astronomy and a critical starting point for the scientific exploitation of survey data. Star–galaxy classification for bright sources can be done with almost complete reliability, but for the numerous sources close to a survey’s detection limit each image encodes only limited morphological information about the source. In this regime, from which many of the new scientific discoveries are likely to come, it is vital to utilize all the available information about a source, both from multiple measurements and from prior knowledge about the star and galaxy populations. This also makes it clear that it is more useful and realistic to provide classification probabilities than decisive classifications. All these desiderata can be met by adopting a Bayesian approach to star–galaxy classification, and we develop a very general formalism for doing so. An immediate implication of applying Bayes’s theorem to this problem is that it is formally impossible to combine morphological measurements in different bands without using colour information as well; however, we develop several approximations that disregard colour information as much as possible. The resultant scheme is applied to data from the UKIRT Infrared Deep Sky Survey (UKIDSS) and tested by comparing the results to deep Sloan Digital Sky Survey (SDSS) Stripe 82 measurements of the same sources. The Bayesian classification probabilities obtained from the UKIDSS data agree well with the deep SDSS classifications both overall (a mismatch rate of 0.022 compared to 0.044 for the UKIDSS pipeline classifier) and close to the UKIDSS detection limit (a mismatch rate of 0.068 compared to 0.075 for the UKIDSS pipeline classifier). The Bayesian formalism developed here can be applied to improve the reliability of any star–galaxy classification schemes based on the measured values of morphology statistics alo
Phinikettos I, Gandy A, 2011, Fast computation of high-dimensional multivariate normal probabilities, COMPUTATIONAL STATISTICS & DATA ANALYSIS, Vol: 55, Pages: 1521-1529, ISSN: 0167-9473
This data is extracted from the Web of Science and reproduced under a licence from Thomson Reuters. You may not copy or re-distribute this data in whole or in part without the written consent of the Science business of Thomson Reuters.