122 results found
Misener R, Biegler L, 2023, Formulating data-driven surrogate models for process optimization, COMPUTERS & CHEMICAL ENGINEERING, Vol: 179, ISSN: 0098-1354
Campos JS, Misener R, Parpas P, 2023, Partial lasserre relaxation for sparse max-cut, Optimization and Engineering, Vol: 24, Pages: 1983-2004, ISSN: 1389-4420
A common approach to solve or find bounds of polynomial optimization problems like Max-Cut is to use the first level of the Lasserre hierarchy. Higher levels of the Lasserre hierarchy provide tighter bounds, but solving these relaxations is usually computationally intractable. We propose to strengthen the first level relaxation for sparse Max-Cut problems using constraints from the second order Lasserre hierarchy. We explore a variety of approaches for adding a subset of the positive semidefinite constraints of the second order sparse relaxation obtained by using the maximum cliques of the graph’s chordal extension. We apply this idea to sparse graphs of different sizes and densities, and provide evidence of its strengths and limitations when compared to the state-of-the-art Max-Cut solver BiqCrunch and the alternative sparse relaxation CS-TSSOS.
Addis B, Castel C, Macali A, et al., 2023, Data augmentation driven by optimization for membrane separation process synthesis, COMPUTERS & CHEMICAL ENGINEERING, Vol: 177, ISSN: 0098-1354
Odgers J, Kappatou C, Misener R, et al., 2023, Probabilistic predictions for partial least squares using bootstrap, AIChE Journal, Vol: 69, Pages: 1-16, ISSN: 0001-1541
Modeling the uncertainty in partial least squares (PLS) is made difficult because of the nonlinear effect of the observed data on the latent space that the method finds. We present an approach, based on bootstrapping, that automatically accounts for these nonlinearities in the parameter uncertainty, allowing us to equally well represent confidence intervals for points lying close to or far away from the latent space. To show the opportunities of this approach, we develop applications in determining the Design Space for industrial processes and model the uncertainty of spectroscopy data. Our results show the benefits of our method for accounting for uncertainty far from the latent space for the purposes of Design Space identification, and match the performance of well established methods for spectroscopy data.
Kappatou C, Odgers J, García-Muñoz S, et al., 2023, An optimization approach coupling pre-processing with model regression for enhanced chemometrics, Industrial and Engineering Chemistry Research, Vol: 62, Pages: 6196-6213, ISSN: 0888-5885
Chemometric methods are broadly used in the chemical and biochemical sectors. Typically, derivation of a regression model follows data preprocessing in a sequential manner. Yet, preprocessing can significantly influence the regression model and eventually its predictive ability. In this work, we investigate the coupling of preprocessing and model parameter estimation by incorporating them simultaneously in an optimization step. Common model selection techniques rely almost exclusively on the performance of some accuracy metric, yet having a quantitative metric for model robustness can prolong model up-time. Our approach is applied to optimize for model accuracy and robustness. This requires the introduction of a novel mathematical definition for robustness. We test our method in a simulated set up and with industrial case studies from multivariate calibration. The results highlight the importance of both accuracy and robustness properties and illustrate the potential of the proposed optimization approach toward automating the generation of efficient chemometric models.
Folch JP, Lee RM, Shafei B, et al., 2023, Combining multi-fidelity modelling and asynchronous batch Bayesian Optimization, COMPUTERS & CHEMICAL ENGINEERING, Vol: 172, ISSN: 0098-1354
Zhang S, Lee RM, Shafei B, et al., 2023, Dependence in constrained Bayesian optimization: When do we need it and how does it help?, Optimization Letters, ISSN: 1862-4472
Constrained Bayesian optimization optimizes a black-box objective function subject to black-box constraints. For simplicity, most existing works assume that multiple constraints are independent. To ask, when and how does dependence between constraints help?, we remove this assumption and implement probability of feasibility with dependence (Dep-PoF) by applying multiple output Gaussian processes (MOGPs) as surrogate models and using expectation propagation to approximate the probabilities. We compare Dep-PoF and the independent version PoF. We propose two new acquisition functions incorporating Dep-PoF and test them on synthetic and practical benchmarks. Our results are largely negative: incorporating dependence between the constraints does not help much. Empirically, incorporating dependence between constraints may be useful if: (i) the solution is on the boundary of the feasible region(s) or (ii) the feasible set is very small. When these conditions are satisfied, the predictive covariance matrix from the MOGP may be poorly approximated by a diagonal matrix and the off-diagonal matrix elements may become important. Dep-PoF may apply to settings where (i) the constraints and their dependence are totally unknown and (ii) experiments are so expensive that any slightly better Bayesian optimization procedure is preferred. But, in most cases, Dep-PoF is indistinguishable from PoF.
Thebelt A, Tsay C, Lee RM, et al., 2022, Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces, 36th Conference on Neural Information Processing Systems (NeurIPS 2022), ISSN: 1049-5258
Tree ensembles can be well-suited for black-box optimization tasks such as algorithm tuning and neural architecture search, as they achieve good predictive performance with little or no manual tuning, naturally handle discrete feature spaces, and are relatively insensitive to outliers in the training data. Two well-known challenges in using tree ensembles for black-box optimization are (i) effectively quantifying model uncertainty for exploration and (ii) optimizing over the piece-wise constant acquisition function. To address both points simultaneously, we propose using the kernel interpretation of tree ensembles as a Gaussian Process prior to obtain model variance estimates, and we develop a compatible optimization formulation for the acquisition function. The latter further allows us to seamlessly integrate known constraints to improve sampling efficiency by considering domain-knowledge in engineering settings and modeling search space symmetries, e.g., hierarchical relationships in neural architecture search. Our framework performs as well as state-of-the-art methods for unconstrained black-box optimization over continuous/discrete features and outperforms competing methods for problems combining mixed-variable feature spaces and known input constraints.
Folch JP, Lee RM, Shafei B, et al., 2022, Combining multi-fidelity modelling and asynchronous batch bayesian optimization, Publisher: arXiv
Bayesian Optimization is a useful tool for experiment design. Unfortunately,the classical, sequential setting of Bayesian Optimization does not translatewell into laboratory experiments, for instance battery design, wheremeasurements may come from different sources and their evaluations may requiresignificant waiting times. Multi-fidelity Bayesian Optimization addresses thesetting with measurements from different sources. Asynchronous batch BayesianOptimization provides a framework to select new experiments before the resultsof the prior experiments are revealed. This paper proposes an algorithmcombining multi-fidelity and asynchronous batch methods. We empirically studythe algorithm behavior, and show it can outperform single-fidelity batchmethods and multi-fidelity sequential methods. As an application, we considerdesigning electrode materials for optimal performance in pouch cells usingexperiments with coin cells to approximate battery performance.
Ceccon F, Jalving J, Haddad J, et al., 2022, OMLT: Optimization & Machine Learning Toolkit, Journal of Machine Learning Research, Vol: 23, ISSN: 1532-4435
The optimization and machine learning toolkit (OMLT) is an open-source software package incorporating neural network and gradient-boosted tree surrogate models, which have been trained using machine learning, into larger optimization problems. We discuss the advances in optimization technology that made OMLT possible and show how OMLT seamlessly integrates with the algebraic modeling language Pyomo. We demonstrate how to use OMLT for solving decision-making problems in both computer science and engineering.
Thebelt A, Wiebe J, Kronqvist JPF, et al., 2022, Maximizing information from chemical engineering data sets: Applications to machine learning, Chemical Engineering Science, Vol: 252, Pages: 1-14, ISSN: 0009-2509
It is well-documented how artificial intelligence can have (and already is having) a big impact on chemical engineering. But classical machine learning approaches may be weak for many chemical engineering applications. This review discusses how challenging data characteristics arise in chemical engineering applications. We identify four characteristics of data arising in chemical engineering applications that make applying classical artificial intelligence approaches difficult: (1) high variance, low volume data, (2) low variance, high volume data, (3) noisy / corrupt / missing data, and (4) restricted data with physics-based limitations. For each of these four data characteristics, we discuss applications where these data characteristics arise and show how current chemical engineering research is extending the fields of data science and machine learning to incorporate these challenges. Finally, we identify several challenges for future research.
Thebelt A, Tsay C, Lee R, et al., 2022, Multi-objective constrained optimization for energy applications via tree ensembles, Applied Energy, Vol: 306, Pages: 1-15, ISSN: 0306-2619
Energy systems optimization problems are complex due to strongly non-linear system behavior and multiple competing objectives, e.g. economic gain vs. environmental impact. Moreover, a large number of input variables and different variable types, e.g. continuous and categorical, are challenges commonly present in real-world applications. In some cases, proposed optimal solutions need to obey explicit input constraints related to physical properties or safety-critical operating conditions. This paper proposes a novel data-driven strategy using tree ensembles for constrained multi-objective optimization of black-box problems with heterogeneous variable spaces for which underlying system dynamics are either too complex to model or unknown. In an extensive case study comprised of synthetic benchmarks and relevant energy applications we demonstrate the competitive performance and sampling efficiency of the proposed algorithm compared to other state-of-the-art tools, making it a useful all-in-one solution for real-world applications with limited evaluation budgets.
Ceccon F, Jalving J, Haddad J, et al., 2022, Presentation abstract: Optimization formulations for machine learning surrogates, Computer Aided Chemical Engineering, Pages: 57-58
In many process systems engineering applications, we seek to integrate surrogate models, e.g. already-trained neural network and gradient-boosted tree models, into larger decision-making problems. This presentation explores different ways to automatically take the machine learning surrogate model and produce an optimization formulation. Our goal is to automate the entire workflow of decision-making with surrogate models from input data to optimization formulation. This presentation discusses our progress towards this goal, gives examples of previous successes, and elicits a conversation with colleagues about the path forward.
Folch JP, Zhang S, Lee RM, et al., 2022, SnAKe: Bayesian Optimization via Pathwise Exploration, ISSN: 1049-5258
Bayesian Optimization is a very effective tool for optimizing expensive black-box functions. Inspired by applications developing and characterizing reaction chemistry using droplet microfluidic reactors, we consider a novel setting where the expense of evaluating the function can increase significantly when making large input changes between iterations. We further assume we are working asynchronously, meaning we have to select new queries before evaluating previous experiments. This paper investigates the problem and introduces 'Sequential Bayesian Optimization via Adaptive Connecting Samples' (SnAKe), which provides a solution by considering large batches of queries and preemptively building optimization paths that minimize input costs. We investigate some convergence properties and empirically show that the algorithm is able to achieve regret similar to classical Bayesian Optimization algorithms in both synchronous and asynchronous settings, while reducing input costs significantly. We show the method is robust to the choice of its single hyper-parameter and provide a parameter-free alternative.
Kronqvist J, Misener R, 2021, A disjunctive cut strengthening technique for convex MINLP, OPTIMIZATION AND ENGINEERING, Vol: 22, Pages: 1315-1345, ISSN: 1389-4420
Mistry M, Letsios D, Krennrich G, et al., 2021, Mixed-integer convex nonlinear optimization with gradient-boosted trees embedded, Informs Journal on Computing, Vol: 33, Pages: 837-1257, C2, ISSN: 1091-9856
Decision trees usefully represent sparse, high-dimensional, and noisy data. Having learned a function from these data, we may want to thereafter integrate the function into a larger decision-making problem, for example, for picking the best chemical process catalyst. We study a large-scale, industrially relevant mixed-integer nonlinear nonconvex optimization problem involving both gradient-boosted trees and penalty functions mitigating risk. This mixed-integer optimization problem with convex penalty terms broadly applies to optimizing pretrained regression tree models. Decision makers may wish to optimize discrete models to repurpose legacy predictive models or they may wish to optimize a discrete model that accurately represents a data set. We develop several heuristic methods to find feasible solutions and an exact branch-and-bound algorithm leveraging structural properties of the gradient-boosted trees and penalty functions. We computationally test our methods on a concrete mixture design instance and a chemical catalysis industrial instance.
Wiebe J, Misener R, 2021, ROmodel: Modeling robust optimization problems in Pyomo, Publisher: arXiv
This paper introduces ROmodel, an open source Python package extending themodeling capabilities of the algebraic modeling language Pyomo to robustoptimization problems. ROmodel helps practitioners transition fromdeterministic to robust optimization through modeling objects which allowformulating robust models in close analogy to their mathematical formulation.ROmodel contains a library of commonly used uncertainty sets which can begenerated using their matrix representations, but it also allows users todefine custom uncertainty sets using Pyomo constraints. ROmodel supportsadjustable variables via linear decision rules. The resulting models can besolved using ROmodels solvers which implement both the robust reformulation andcutting plane approach. ROmodel is a platform to implement and compare customuncertainty sets and reformulations. We demonstrate ROmodel's capabilities byapplying it to six case studies. We implement custom uncertainty sets based on(warped) Gaussian processes to show how ROmodel can integrate data-drivenmodels with optimization.
Ceccon F, Misener R, 2021, Solving the pooling problem at scale with extensible solver GALINI, Publisher: arXiv
This paper presents a Python library to model pooling problems, a class ofnetwork flow problems with many engineering applications. The libraryautomatically generates a mixed-integer quadratically-constrained quadraticoptimization problem from a given network structure. The library additionallyuses the network structure to build 1) a convex linear relaxation of thenon-convex quadratic program and 2) a mixed-integer linear restriction of theproblem. We integrate the pooling network library with galini, an open-sourceextensible global solver for quadratic optimization. We demonstrate galini'sextensible characteristics by using the pooling library to develop two galiniplug-ins: 1) a cut generator plug-in that adds valid inequalities in the galinicut loop and 2) a primal heuristic plug-in that uses the mixed-integer linearrestriction. We test galini on large scale pooling problems and show that,thanks to the good upper bound provided by the mixed-integer linear restrictionand the good lower bounds provided by the convex relaxation, we obtainoptimality gaps that are competitive with Gurobi 9.1 on the largest probleminstances.
Pistikopoulos EN, Barbosa-Povoa A, Lee JH, et al., 2021, Process systems engineering - <i>The generation next?</i>, COMPUTERS & CHEMICAL ENGINEERING, Vol: 147, ISSN: 0098-1354
Letsios D, Bradley JT, Suraj G, et al., 2021, Approximate and robust bounded job start scheduling for Royal Mail delivery offices, Journal of Scheduling, Vol: 24, Pages: 237-258, ISSN: 1094-6136
Motivated by mail delivery scheduling problems arising in Royal Mail, we study a generalization of the fundamental makespan scheduling P||Cmax problem which we call the bounded job start scheduling problem. Given a set of jobs, each specified by an integer processing time pj, that have to be executed non-preemptively by a set of m parallel identical machines, the objective is to compute a minimum makespan schedule subject to an upper bound g≤m on the number of jobs that may simultaneously begin per unit of time. With perfect input knowledge, we show that Longest Processing Time First (LPT) algorithm is tightly 2-approximate. After proving that the problem is strongly NP-hard even when g=1, we elaborate on improving the 2-approximation ratio for this case. We distinguish the classes of long and short instances satisfying pj≥m and pj<m, respectively, for each job j. We show that LPT is 5/3-approximate for the former and optimal for the latter. Then, we explore the idea of scheduling long jobs in parallel with short jobs to obtain tightly satisfied packing and bounded job start constraints. For a broad family of instances excluding degenerate instances with many very long jobs, we derive a 1.985-approximation ratio. For general instances, we require machine augmentation to obtain better than 2-approximate schedules. In the presence of uncertain job processing times, we exploit machine augmentation and lexicographic optimization, which is useful for P||Cmax under uncertainty, to propose a two-stage robust optimization approach for bounded job start scheduling under uncertainty aiming in a low number of used machines. Given a collection of schedules of makespan ≤D, this approach allows distinguishing which are the more robust. We substantiate both the heuristics and our recovery approach numerically using Royal Mail data. We show that for the Royal Mail application, machine augmentation, i.e., short-term van rental, is especially relevant.
Tsay C, Kronqvist J, Thebelt A, et al., 2021, Partition-based formulations for mixed-integer optimization of trained ReLU neural networks, Publisher: arXiv
This paper introduces a class of mixed-integer formulations for trained ReLUneural networks. The approach balances model size and tightness by partitioningnode inputs into a number of groups and forming the convex hull over thepartitions via disjunctive programming. At one extreme, one partition per inputrecovers the convex hull of a node, i.e., the tightest possible formulation foreach node. For fewer partitions, we develop smaller relaxations thatapproximate the convex hull, and show that they outperform existingformulations. Specifically, we propose strategies for partitioning variablesbased on theoretical motivations and validate these strategies using extensivecomputational experiments. Furthermore, the proposed scheme complements knownalgorithmic approaches, e.g., optimization-based bound tightening capturesdependencies within a partition.
Olofsson S, Schultz ES, Mhamdi A, et al., 2021, Design of dynamic experiments for black-box model discrimination, Publisher: arXiv
Diverse domains of science and engineering require and use mechanisticmathematical models, e.g. systems of differential algebraic equations. Suchmodels often contain uncertain parameters to be estimated from data. Consider adynamic model discrimination setting where we wish to chose: (i) what is thebest mechanistic, time-varying model and (ii) what are the best model parameterestimates. These tasks are often termed modeldiscrimination/selection/validation/verification. Typically, several rivalmechanistic models can explain data, so we incorporate available data and alsorun new experiments to gather more data. Design of dynamic experiments formodel discrimination helps optimally collect data. For rival mechanistic modelswhere we have access to gradient information, we extend existing methods toincorporate a wider range of problem uncertainty and show that our proposedapproach is equivalent to historical approaches when limiting the types ofconsidered uncertainty. We also consider rival mechanistic models as dynamicblack boxes that we can evaluate, e.g. by running legacy code, but wheregradient or other advanced information is unavailable. We replace theseblack-box models with Gaussian process surrogate models and thereby extend themodel discrimination setting to additionally incorporate rival black-box model.We also explore the consequences of using Gaussian process surrogates toapproximate gradient-based methods.
Kronqvist J, Misener R, Tsay C, 2021, Between steps: Intermediate relaxations between big-M and convex hull formulations, Publisher: arXiv
This work develops a class of relaxations in between the big-M and convexhull formulations of disjunctions, drawing advantages from both. The proposed"P-split" formulations split convex additively separable constraints into Ppartitions and form the convex hull of the partitioned disjuncts. Parameter Prepresents the trade-off of model size vs. relaxation strength. We examine thenovel formulations and prove that, under certain assumptions, the relaxationsform a hierarchy starting from a big-M equivalent and converging to the convexhull. We computationally compare the proposed formulations to big-M and convexhull formulations on a test set including: K-means clustering, P_ball problems,and ReLU neural networks. The computational results show that the intermediateP-split formulations can form strong outer approximations of the convex hullwith fewer variables and constraints than the extended convex hullformulations, giving significant computational advantages over both the big-Mand convex hull.
Wiebe J, Misener R, 2021, ROmodel: A Python Robust Optimization Modeling Toolbox, Computer Aided Chemical Engineering, Pages: 683-688
We introduce ROmodel, a Python package that extends the modeling capabilities of the popular modeling language Pyomo to robust optimization problems. ROmodel contains a library of commonly used uncertainty sets which can be generated using their matrix representations, but it also allows the definition of custom uncertainty sets using Pyomo constraints. The resulting models can be solved using ROmodels solvers which implement both the robust reformulation and cutting plane approach. We apply the problem to a number of instances of three case studies and show some results.
Tsay C, Kronqvist J, Thebelt A, et al., 2021, Partition-Based Formulations for Mixed-Integer Optimization of Trained ReLU Neural Networks, 35th Conference on Neural Information Processing Systems (NeurIPS), Publisher: NEURAL INFORMATION PROCESSING SYSTEMS (NIPS), ISSN: 1049-5258
Wiebe J, Cecílio I, Dunlop J, et al., 2020, A robust approach to warped Gaussian process-constrained optimization, Publisher: arXiv
Optimization problems with uncertain black-box constraints, modeled by warpedGaussian processes, have recently been considered in the Bayesian optimizationsetting. This work introduces a new class of constraints in which the sameblack-box function occurs multiple times evaluated at different domain points.Such constraints are important in applications where, e.g., safety-criticalmeasures are aggregated over multiple time periods. Our approach, which usesrobust optimization, reformulates these uncertain constraints intodeterministic constraints guaranteed to be satisfied with a specifiedprobability, i.e., deterministic approximations to a chance constraint. Thisapproach extends robust optimization methods from parametric uncertainty touncertain functions modeled by warped Gaussian processes. We analyze convexityconditions and propose a custom global optimization strategy for non-convexcases. A case study derived from production planning and an industriallyrelevant example from oil well drilling show that the approach effectivelymitigates uncertainty in the learned curves. For the drill scheduling example,we develop a custom strategy for globally optimizing integer decisions.
Čyras K, Karamlou A, Lee M, et al., 2020, AI-assisted schedule explainer for nurse rostering, AAMAS, Pages: 2101-2103, ISSN: 1548-8403
We present an argumentation-supported explanation generating system, called Schedule Explainer, that assists with makespan scheduling. Our stand-alone generic tool explains to a lay user why a resource allocation schedule is good or not, and offers actions to improve the schedule given the user's constraints. Schedule Explainer provides actionable textual explanations via an interactive graphical interface. We illustrate our system with a proof-of-concept application tool in a nurse rostering scenario whereby a shift-lead nurse aims to account for unexpected events by rescheduling some patient procedures to nurses and is aided by the system to do so.
Thebelt A, Kronqvist J, Mistry M, et al., 2020, ENTMOOT: A framework for optimization over ensemble tree models, Publisher: arXiv
Gradient boosted trees and other regression tree models perform well in awide range of real-world, industrial applications. These tree models (i) offerinsight into important prediction features, (ii) effectively manage sparsedata, and (iii) have excellent prediction capabilities. Despite theiradvantages, they are generally unpopular for decision-making tasks andblack-box optimization, which is due to their difficult-to-optimize structureand the lack of a reliable uncertainty measure. ENTMOOT is our new frameworkfor integrating (already trained) tree models into larger optimizationproblems. The contributions of ENTMOOT include: (i) explicitly introducing areliable uncertainty measure that is compatible with tree models, (ii) solvingthe larger optimization problems that incorporate these uncertainty aware treemodels, (iii) proving that the solutions are globally optimal, i.e. no bettersolution exists. In particular, we show how the ENTMOOT approach allows asimple integration of tree models into decision-making and black-boxoptimization, where it proves as a strong competitor to commonly-usedframeworks.
Letsios D, Baltean-Lugojan R, Ceccon F, et al., 2020, Approximation algorithms for process systems engineering, COMPUTERS & CHEMICAL ENGINEERING, Vol: 132, ISSN: 0098-1354
This data is extracted from the Web of Science and reproduced under a licence from Thomson Reuters. You may not copy or re-distribute this data in whole or in part without the written consent of the Science business of Thomson Reuters.