Publications
42 results found
Qiu D, Baig AM, Wang Y, et al., 2024, Market design for ancillary service provisions of inertia and frequency response via virtual power plants: A non-convex bi-level optimisation approach, Applied Energy, Vol: 361, ISSN: 0306-2619
The Great Britain (GB) government is paving the path to decarbonisation by actively promoting the integration of wind power into its generation mix. This sharp transition to renewable energy, however, introduces specific challenges. The characteristics of non-synchronous wind turbines have the potential to impact grid stability and frequency security due to the reduction in system inertia. In response to these challenges, the deployment of virtual power plants (VPPs) is envisioned in deregulated power systems, which allow for the aggregation and coordinated control of diverse distributed energy resources, enhancing grid flexibility, reliability, and efficiency. Moreover, VPPs offer the provision of system inertia and frequency response services at the national level. This study drops this assumption and leverages the localised flexibility inherent in VPPs to meet the ancillary service requirements of the low-carbon power system. The formulated problem is constructed as a non-convex bi-level optimisation problem, where the upper-level problem represents the operation of a frequency-constrained unit commitment model, and the lower-level problem embodies the energy dispatches and frequency responses of a group of VPPs, guided by the dual price signals of cleared energy and ancillary services. To address the non-convex nature of the unit commitment problem, a two-fold approach is employed. First, binary commitment status decision variables are relaxed to continuous versions. Subsequently, the duality gap between the original non-convex unit commitment problem and its relaxed dual form is minimised, yielding a solution closely approximating the optimal solution of the original problem. Second, the relaxed bi-level optimisation problem is transformed into a single-level mathematical programs with equilibrium constraints by replacing the lower-level VPP problems with their equivalent Karush-Kuhn–Tucker optimality conditions. The case studies conducted in this work en
Wang Y, Qiu D, Sun X, et al., 2024, Coordinating multi-energy microgrids for integrated energy system resilience: a multi-task learning approach, IEEE Transactions on Sustainable Energy, Vol: 15, Pages: 920-937, ISSN: 1949-3029
High-impact and low-probability events have occurred more frequently than before, which can seriously damage energy supply infrastructures. As localized small energy systems, multi-energy microgrids (MEMGs) can provide a viable solution for the system- wise load restoration of integrated energy systems (IESs), due to their enhanced flexibility and controllability. However, existing literature tends to realize MEMGs as corrective response rather than load restoration resource after extreme events, which cannot fully exploit the benefits of multi-MEMGs on IES resilience. This paper introduces a decentralized operating paradigm for the real-time coordination of local multi-MEMGs towards system- wise IES load restoration, while a novel topology-aware multi-task reinforcement learning method with soft modularization is proposed to solve it. The multi-task learning framework enables MEMGs to simultaneously learn scheduling decisions across different network topologies and better adapt to unanticipated contingencies. Additionally, to avoid insecure MEMG operations, a physics-informed safety layer is embedded on top of the multi-task learning framework for action corrections. Case studies have been conducted on two IESs (33- bus power, 20-node gas, and 20-node heat network as well as 69- bus power, 40-node gas, and 62-node heat network) to evaluate the effectiveness of the proposed method in enabling effective coordination among multi-MEMGs towards system- wise IES load restoration.
Qiu D, Wang Y, Wang J, et al., 2024, Resilience-oriented coordination of networked microgrids: a shapley Q-value learning approach, IEEE Transactions on Power Systems, Vol: 39, Pages: 3401-3416, ISSN: 0885-8950
High-impact and low-probability extreme events have occurred more frequently than before because of rapid climate change, which can seriously damage distribution systems. However, conventional distribution management can be dysfunctional after an event, destroying its centralized supervision towards resilience enhancement. In this context, networked microgrids (NMGs) with distributed energy resources provide a viable solution for the resilience enhancement of distribution systems. Existing literature tends to employ model-based optimization approaches for resilient operations of NMGs, which require complete system models and can be time-consuming. To address these challenges, this article suggests a decentralized framework for resilience-oriented coordination of NMGs and proposes a novel multi-agent reinforcement learning (MARL) method to solve it. Specifically, the proposed MARL method develops an efficient credit assignment scheme for NMGs to learn their contributions to the distribution system resilience via the Shapley Q-value technique with more efficient resilience enhancement. Case studies based on two modified IEEE 15- and 69-bus distribution networks are conducted to validate the effectiveness of the proposed MARL method in enabling effective coordination among NMGs and providing a high resilience level.
Wang Y, Qiu D, Wang Y, et al., 2024, Graph Learning-Based Voltage Regulation in Distribution Networks With Multi-Microgrids, IEEE Transactions on Power Systems, Vol: 39, Pages: 1881-1895, ISSN: 0885-8950
Microgrids (MGs), as localized small power systems, can effectively provide voltage regulation services for distribution networks by integrating and managing various distributed energy resources. Existing literature employs model-based optimization approaches to formulate the voltage regulation problem of multi-MGs, which require complete system models. However, this assumption is normally impractical due to time-varying environment and privacy issues. To fill this research gap, this paper suggests a data-driven decentralized framework for the cost-effective voltage regulation of a distribution network with multi-MGs. A novel multi-agent reinforcement learning method featuring an augmented graph convolutional network and a proximal policy optimization algorithm is proposed to solve this problem. Furthermore, the techniques of critical bus and electrical distance enhance the capability of feature extractions from the distribution network, allowing for the decentralized training with privacy preserving. Simulation results based on modified IEEE 33-bus, 69-bus, and 123-bus networks are developed to validate the effectiveness of the proposed method in enabling multi-MGs to provide distribution network voltage regulation.
Wang Y, Qiu D, Teng F, et al., 2024, Towards microgrid resilience enhancement via mobile power sources and repair crews: a multi-agent reinforcement learning approach, IEEE Transactions on Power Systems, Vol: 39, Pages: 1329-1345, ISSN: 0885-8950
Mobile power sources (MPSs) have been gradually deployed in microgrids as critical resources to coordinate with repair crews (RCs) towards resilience enhancement owing to their flexibility and mobility in handling the complex coupled power-transport systems. However, previous work solves the coordinated dispatch problem of MPSs and RCs in a centralized manner with the assumption that the communication network is still fully functioning after the event. However, there is growing evidence that certain extreme events will damage or degrade communication infrastructure, which makes centralized decision making impractical. To fill this gap, this paper formulates the resilience-driven dispatch problem of MPSs and RCs in a decentralized framework. To solve this problem, a hierarchical multi-agent reinforcement learning method featuring a two-level framework is proposed, where the high-level action is used to switch decision-making between power and transport networks, and the low-level action constructed via a hybrid policy is used to compute continuous scheduling and discrete routing decisions in power and transport networks, respectively. The proposed method also uses an embedded function encapsulating system dynamics to enhance learning stability and scalability. Case studies based on IEEE 33-bus and 69-bus power networks are conducted to validate the effectiveness of the proposed method in load restoration.
Qiu D, Wang Y, Ding Z, et al., 2024, Graph Reinforcement Learning for Carbon-Aware Electric Vehicles in Power-Transport Networks, IEEE Transactions on Smart Grid, Pages: 1-1, ISSN: 1949-3053
Zhang T, Sun M, Qiu D, et al., 2023, A Bayesian Deep Reinforcement Learning-Based Resilient Control for Multi-Energy Micro-Gird, IEEE Transactions on Power Systems, Vol: 38, Pages: 5057-5072, ISSN: 0885-8950
Aiming at a cleaner future power system, many regimes in the world have proposed their ambitious decarbonizing plan, with increasing penetration of renewable energy sources (RES) playing an alternative role to conventional energy. As a result, power system tends to have less backup capacity and operate near their designed limit, thus exacerbating system vulnerability against extreme events. Under this reality, resilient control for the multi-energy micro-grid is facing the following challenges, which are: 1) the effect from the stochastic uncertainties of RES; 2) the need for a model-free and fast-reacting control scheme under extreme events; and 3) efficient exploration and robust performance with limited extreme events data. To deal with the aforementioned challenges, this paper proposes a novel Bayesian Deep Reinforcement Learning-based resilient control approach for multi-energy micro-grid. In particular, the proposed approach replaces the deterministic network in traditional Reinforcement Learning with a Bayesian probabilistic network in order to obtain an approximation of the value function distribution, which effectively solves the Q-value overestimation issue. Compared with the naive Deep Deterministic Policy Gradient (DDPG) method and optimization method, the effectiveness and importance of employing the Bayesian Reinforcement Learning approach are investigated and illustrated across different operating scenarios. Case studies have shown that by using the Monte Carlo posterior mean of the Bayesian value function distribution instead of a deterministic estimation, the proposed Bayesian Deep Deterministic Policy Gradient (BDDPG) method achieves a near-optimum policy in a more stable process, which verifies the robustness and the practicability of the proposed approach.
Qiu D, Wang Y, Wang J, et al., 2023, Personalized retail pricing design for smart metering consumers in electricity market, APPLIED ENERGY, Vol: 348, ISSN: 0306-2619
Qiu D, Wang J, Dong Z, et al., 2023, Mean-Field Multi-Agent Reinforcement Learning for Peer-to-Peer Multi-Energy Trading, IEEE TRANSACTIONS ON POWER SYSTEMS, Vol: 38, Pages: 4853-4866, ISSN: 0885-8950
- Author Web Link
- Cite
- Citations: 1
Wang Y, Rousis AO, Qiu D, et al., 2023, A stochastic distributed control approach for load restoration of networked microgrids with mobile energy storage systems, INTERNATIONAL JOURNAL OF ELECTRICAL POWER & ENERGY SYSTEMS, Vol: 148, ISSN: 0142-0615
- Author Web Link
- Cite
- Citations: 3
Qiu D, Wang Y, Zhang T, et al., 2023, Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience, APPLIED ENERGY, Vol: 336, ISSN: 0306-2619
- Author Web Link
- Cite
- Citations: 3
Wang Y, Qiu D, Sun M, et al., 2023, Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach, APPLIED ENERGY, Vol: 335, ISSN: 0306-2619
- Author Web Link
- Cite
- Citations: 2
Qiu D, Chen T, Strbac G, et al., 2023, Coordination for Multienergy Microgrids Using Multiagent Reinforcement Learning, IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, Vol: 19, Pages: 5689-5700, ISSN: 1551-3203
- Author Web Link
- Cite
- Citations: 4
Qiu D, Xue J, Zhang T, et al., 2023, Federated reinforcement learning for smart building joint peer-to-peer energy and carbon allowance trading, Applied Energy, Vol: 333, ISSN: 0306-2619
The multi-energy system (MES), which is regarded as an optimum solution to a high-efficiency, green energy system and a crucial shift towards the future low-carbon energy system, has attracted great attention at the district building level. However, the current exploration of flexible MES operation has been hampered by (1) the increasing penetration of renewable energies and the complicated operation of coupling multi-energy sectors; (2) the privacy concern in the decentralization of the energy system; and (3) the lack of integration of the energy market and carbon emission trading scheme. To address the aforementioned challenges, this paper proposes a joint peer-to-peer energy and carbon allowance trading mechanism for a building community, and then models it as a multi-agent reinforcement learning (MARL) paradigm. In this setting, the flexibility of building local trading and the decarbonization of building energy management can both be fully utilized. To stabilize the training performance, an abstract critic network capturing system dynamics is introduced based on a deep deterministic policy gradient method. The technique of federated learning (FL) is also applied to speed up the training and safeguard the private information of each building in the community. Empirical results on a real-world test case evaluate its superior performance in terms of achieving both economic and environmental benefits, resulting in 5.87% and 8.02% lower total energy and environment costs than the two baseline mechanisms of peer-to-grid energy trading and peer-to-peer energy trading, respectively.
Qiu D, Wang Y, Hua W, et al., 2023, Reinforcement learning for electric vehicle applications in power systems: A critical review, RENEWABLE & SUSTAINABLE ENERGY REVIEWS, Vol: 173, ISSN: 1364-0321
- Author Web Link
- Cite
- Citations: 8
Wang Y, Qiu D, Strbac G, et al., 2023, Coordinated Electric Vehicle Active and Reactive Power Control for Active Distribution Networks, IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, Vol: 19, Pages: 1611-1622, ISSN: 1551-3203
- Author Web Link
- Cite
- Citations: 9
Wang J, Qiu D, Wang Y, et al., 2023, Cost-effective and Resilient Operation of Distribution Grids and 5G Telecommunication, ISSN: 1944-9925
5G base stations have growing importance in an integrated electric power and telecommunication system, for mobile user equipment mobile data supply and demand response in distribution grids. However, demand response through base station's flexibility can have a growing impact on the power flow of the grid. Additionally, in extreme events, if a power outage occurs at the physical base station, data loads need to be first reconnected to the 5G network, which is essential for the grid to further recover the electricity loads. In this paper, a cost-effective and resilient operation method is proposed to optimally utilize the flexibility of renewable-based 5G base stations and the data load shedding to recover the data transmission. The flexibility from batteries equipped in the base stations and the flexible associations between user equipments and base stations are considered. The simulation results verify the proposed method can achieve lower energy costs and power losses of the grid in normal operation and a resilient operation in an extreme event.
Qiu D, Dong Z, Wang Y, et al., 2023, Decarbonising the GB Power System Via Numerous Electric Vehicle Coordination, IEEE Transactions on Power Systems, ISSN: 0885-8950
The increasing penetration of renewable energy has promoted fast decarbonisation of the GB power system. However, low-carbon transitions should be considered from a whole energy system perspective, such as through localised vehicle-to-grid (V2G) techniques. Despite the potential benefits of utilising decentralised V2G flexibility for decarbonisation, the absence of an appropriate incentive mechanism has limited its effectiveness. This paper aims at studying the carbon-aware electric vehicle (EV) power scheduling problem by introducing a carbon emission flow model. The model enables EVs to be cognisant of carbon intensity (CI) signals, thereby enabling them to provide carbon services. To solve this problem, a novel decentralised control approach is proposed to coordinate numerous EVs in a computationally efficient manner with privacy perseverance. Case studies on a 14-node GB power network with a large population of 556,733 EVs are conducted to validate its efficacy in simultaneously maximising the EVs' carbon service provision and decarbonising the GB power system by intelligently linking local behaviours with global interest. Finally, the proposed control approach shows its generalisation performance for various day and season scenarios as well as evidence for realising the GB's decarbonisation ambitions by 2030 and 2050.
Qiu D, Chrysanthopoulos N, Strbac G, 2023, Tariff Design for Local Energy Communities Through Strategic Retail Pricing, 19th International Conference on the European Energy Market (EEM), Publisher: IEEE, ISSN: 2165-4077
Wang J, Qiu D, Strbac G, et al., 2023, Market-Based Generation Planning with Carbon Target, 19th International Conference on the European Energy Market (EEM), Publisher: IEEE, ISSN: 2165-4077
Qiu D, Wang Y, Zhang T, et al., 2022, Hybrid Multiagent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition, IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, Vol: 18, Pages: 8258-8269, ISSN: 1551-3203
- Author Web Link
- Cite
- Citations: 9
Zeng L, Qiu D, Sun M, 2022, Resilience enhancement of multi-agent reinforcement learning-based demand response against adversarial attacks, Applied Energy, Vol: 324, ISSN: 0306-2619
Demand response improves grid security by adjusting the flexibility of consumers meanwhile maintaining their demand–supply balance in real-time. With the large-scale deployment of distributed digital communication technologies and advanced metering infrastructures, data-driven approaches such as multi-agent reinforcement learning (MARL) are being widely employed to solve demand response problems. Nevertheless, the massive interaction of data inside and outside the demand response management system may lead to severe threats from the perspective of cyber-attacks. The cyber security requirements of MARL-based demand response problems are less discussed in the existing studies. To this end, this paper proposes a robust adversarial multi-agent reinforcement learning framework for demand response (RAMARL-DR) with an enhanced resilience against adversarial attacks. In particular, the proposed RAMARL-DR first constructs an adversary agent that aims to cause the worst-case performance via formulating an adversarial attack; and then adopts periodic alternating robust adversarial training scenarios with the optimal adversary aiming to diminish the severe impacts induced by adversarial attacks. Case studies are conducted based on an OpenAI Gym environment CityLearn, which provides a standard evaluation platform of MARL algorithms for demand response problems. Empirical results indicate that the MARL-based demand response management system is vulnerable when the adversary agent occurs, and its performance can be significantly improved after periodic alternating robust adversarial training. It can be found that the adversary agent can result in a 41.43% higher metric value of Ramping than the no adversary case, whereas the proposed RAMARL-DR can significantly enhance the system resilience with an approximately 38.85% reduction in the ramping of net demand.
Qiu D, Dong Z, Ruan G, et al., 2022, Strategic retail pricing and demand bidding of retailers in electricity market: A data-driven chance-constrained programming, ADVANCES IN APPLIED ENERGY, Vol: 7, ISSN: 2666-7924
- Author Web Link
- Cite
- Citations: 2
Qiu D, Wang Y, Sun M, et al., 2022, Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach, APPLIED ENERGY, Vol: 313, ISSN: 0306-2619
- Author Web Link
- Cite
- Citations: 9
Bellizio F, Xu W, Qiu D, et al., 2022, Transition to Digitalized Paradigms for Security Control and Decentralized Electricity Market, PROCEEDINGS OF THE IEEE, ISSN: 0018-9219
Wang Y, Qiu D, Strbac G, 2022, Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems, APPLIED ENERGY, Vol: 310, ISSN: 0306-2619
- Author Web Link
- Cite
- Citations: 21
Qiu D, Dong Z, Zhang X, et al., 2022, Safe reinforcement learning for real-time automatic control in a smart energy-hub, APPLIED ENERGY, Vol: 309, ISSN: 0306-2619
- Author Web Link
- Cite
- Citations: 27
Wang Y, Qiu D, Strbac G, 2022, Multi-agent reinforcement learning for electric vehicles joint routing and scheduling strategies, IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Publisher: IEEE, Pages: 3044-3049, ISSN: 2153-0009
- Author Web Link
- Cite
- Citations: 1
Qiu D, Ye Y, Papadaskalopoulos D, et al., 2021, Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach, APPLIED ENERGY, Vol: 292, ISSN: 0306-2619
- Author Web Link
- Cite
- Citations: 47
Ye Y, Qiu D, Wang H, et al., 2021, Real-Time Autonomous Residential Demand Response Management Based on Twin Delayed Deep Deterministic Policy Gradient Learning, ENERGIES, Vol: 14
- Author Web Link
- Cite
- Citations: 19
This data is extracted from the Web of Science and reproduced under a licence from Thomson Reuters. You may not copy or re-distribute this data in whole or in part without the written consent of the Science business of Thomson Reuters.