- Physical Review Journals
All JournalsPhysics Magazine

PRX Energy

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Defending Smart Electrical Power Grids against Cyberattacks with Deep Q-Learning

Mohammadamin Moradi1, Yang Weng1, and Ying-Cheng Lai1,2,*

  • 1School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, Arizona 85287, USA
  • 2Department of Physics, Arizona State University, Tempe, Arizona 85287, USA

  • *Ying-Cheng.Lai@asu.edu
PDF

PRX Energy 1, 033005 – Published 28 November, 2022

DOI: https://doi.org/10.1103/PRXEnergy.1.033005

Abstract

A key to ensuring the security of smart electrical power grids is to devise and deploy effective defense strategies against cyberattacks. To achieve this goal, an essential task is to simulate and understand the dynamic interplay between the attacker and defender, for which stochastic game theory and reinforcement learning stand out as a powerful mathematical and computational framework. Existing works are based on conventional Q-learning to find the critical sections of a power grid to choose an effective defense strategy, but the methodology is only applicable to small systems. Additional issues with Q-learning are the difficulty in considering the timings of cascading failures in the reward function and deterministic modeling of the game, while attack success depends on various parameters and typically has a stochastic nature. Our solution for overcoming these difficulties is to develop a deep Q-learning-based stochastic zero-sum Nash strategy solution. We demonstrate the workings of our deep Q-learning solution using the benchmark Wood and Wollenberg 6-bus and the IEEE 30-bus systems; the latter is a relatively large-scale power-grid system that defies the conventional Q-learning approach. Comparison with alternative reinforcement learning methods provides further support for the general applicability of our deep Q-learning framework in ensuring secure operation of modern power-grid systems.

View figure in article

Physics Subject Headings (PhySH)

Popular Summary

Electric power grids, a critical infrastructure, are vulnerable to random failures and, more alarmingly, to hostile physical and/or cyberattacks that can often trigger large-scale, cascading breakdowns. The US-Canadian blackout in 2003 affected approximately 50 million people in eight US states and two Canadian provinces. In the same year, there were two significant blackouts in Europe. The gigantic impacted geophysical area of these events and the economic consequences highlight the need to develop effective defense strategies against attacks on the power grid. Generally, cybersecurity is determined by the dynamic interplay between the attacker and defender, where the former seeks to maximize while the latter strives to minimize damage to the power grid. Another recent trend is to incorporate machine learning, especially reinforcement learning, into cybersecurity of power grids, but a fundamental shortcoming is the exponentially growing state space with the system size.

Here, a general deep learning framework is developed to solve the stochastic game problem in arbitrarily large power-grid systems. The algorithm typically leads to a Nash equilibrium and the best protection sets are based on the defined objectives, taking into consideration the defender’s policy. The work opens the door to applying reinforcement learning to large-scale smart grid cybersecurity problems to significantly enhance the security of the system.

Article Text

References (46)

  1. P. Pourbeik, P. S. Kundur, and C. W. Taylor, The anatomy of a power grid blackout—root causes and dynamics of recent major blackouts, IEEE Power Energy Mag. 4, 22 (2006).
  2. J. Xie, A. Stefanov, and C.-C. Liu, Physical and cyber security in a smart grid environment, Wiley Interdis. Rev. Energy Envir. 5, 519 (2016).
  3. R. Langner, Stuxnet: Dissecting a cyberwarfare weapon, IEEE Secu. Priv. 9, 49 (2011).
  4. G. Liang, S. R. Weller, J. Zhao, F. Luo, and Z. Y. Dong, The 2015 Ukraine blackout: Implications for false data injection attacks, IEEE Trans. Power Sys. 32, 3317 (2017).
  5. Y. Liu, P. Ning, and M. K. Reiter, False data injection attacks against state estimation in electric power grids, ACM Trans. Info. Sys. Secu. 14, 1 (2011).
  6. L. Xie, Y. Mo, and B. Sinopoli, Integrity data attacks in power market operations, IEEE Trans. Smart Grid 2, 659 (2011).
  7. M. Mohammadpourfard, Y. Weng, and M. Tajdinian, Benchmark of machine learning algorithms on capturing future distribution network anomalies, IET Gene. Transmi. Distri. 13, 1441 (2019).
  8. A. Shefaei, M. Mohammadpourfard, B. Mohammadi-ivatloo, and Y. Weng, Revealing a new vulnerability of distributed state estimation: A data integrity attack and an unsupervised detection algorithm, IEEE Trans. Cont. Net. Sys. (2021),.
  9. N. Enriquez and Y. Weng, in Asian Conference on Machine Learning (ACML), PMLR 157 (2021) p. 1333.
  10. J. F. Nash, Equilibrium points in n-person games, Proc. Nat. Aca. Sci. (USA) 36, 48 (1950).
  11. T. Başar and G. J. Olsder, Dynamic Non-Cooperative Game Theory (SIAM, Philadelphia, United States, 1998), 2nd ed.
  12. W. Saad, Z. Han, H. V. Poor, and T. Basar, Game-theoretic methods for the smart grid: An overview of microgrid systems, demand-side management, and smart grid communications, IEEE Sig. Proc. Mag. 29, 86 (2012).
  13. S. Poudel, Z. Ni, X. Zhong, and H. He, in 2016 International Joint Conference on Neural Networks (IJCNN), p. 2730.
  14. N. I. Haque, M. H. Shahriar, M. G. Dastgir, A. Debnath, I. Parvez, A. Sarwat, and M. A. Rahman, Machine learning in generation, detection, and mitigation of cyberattacks in smart grid: A survey, arXiv preprint (2020), ArXiv:2010.00661.
  15. Y. Chen, S. Huang, F. Liu, Z. Wang, and X. Sun, Evaluation of reinforcement learning-based false data injection attack to automatic voltage control, IEEE Trans. Smart Grid 10, 2158 (2019).
  16. B. Ning and L. Xiao, in 2021 40th Chinese Control Conference (CCC) (IEEE), p. 8598.
  17. C. J. C. H. Watkins and P. Dayan, Q-learning, Mach. Learn. 8, 279 (1992).
  18. J. Yan, H. He, X. Zhong, and Y. Tang, Q-Learning-based vulnerability analysis of smart grid against sequential topology attacks, IEEE Trans. Info. Foren. Secu. 12, 200 (2017).
  19. Z. Wang, H. He, Z. Wan, and Y. Sun, Coordinated topology attacks in smart grid using deep reinforcement learning, IEEE Trans. Indust. Info. 17, 1407 (2020).
  20. C. Roberts, S.-T. Ngo, A. Milesi, S. Peisert, D. Arnold, S. Saha, A. Scaglione, N. Johnson, A. Kocheturov, and D. Fradkin, in 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) (IEEE), p. 1.
  21. Y. Li and J. Wu, Low latency cyberattack detection in smart grids with deep reinforcement learning, Int. J. Electr. Power Energy Syst. 142, 108265 (2022).
  22. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller, Playing Atari with deep reinforcement learning, (2013), ArXiv:1312.5602.
  23. J. Hu and M. P. Wellman, in ICML ’98 Proceedings of the Fifteenth International Conference on Machine Learning (1998), p. 242.
  24. S. R. Etesami and T. Basar, Dynamic games in cyber-physical security: An overview, Dyn. Games Appl. 9, 884 (2019).
  25. D. Vrabie and F. Lewis, in The 2010 International Joint Conference on Neural Networks (IJCNN), p. 1.
  26. Q. Zhu, H. Tembine, and T. Bbar, Heterogeneous learning in zero-sum stochastic games with incomplete information, 49th IEEE Conference on Decision and Control (CDC), 219 (2010).
  27. P. Bommannavar, T. Alpcan, and N. Bambos, Security risk management via dynamic games with learning, 2011 IEEE International Conference on Communications (ICC), 1 (2011).
  28. A. Truong, S. R. Etesami, J. Etesami, and N. Kiyavash, Optimal attack strategies against predictors - learning from expert advice, IEEE Trans. Info. Foren. Secu. 13, 6 (2018).
  29. Q. Zhu, H. Tembine, and T. Başar, in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control (2013), p. 303.
  30. K.-W. Chung, C. A. Kamhoua, K. A. Kwiat, Z. T. Kalbarczyk, and R. K. Iyer, in 2016 IEEE 17th International Symposium on High Assurance Systems Engineering (HASE) (2016), p. 1.
  31. X. He, H. Dai, and P. Ning, Improving learning and adaptation in security games by exploiting information asymmetry, 2015 IEEE Conference on Computer Communications (INFOCOM), 17872015.
  32. K. K. Trejo, J. B. Clempner, and A. S. Poznyak, in 2016 IEEE 55th Conference on Decision and Control (CDC), p. 5484.
  33. M. J. Eppstein and P. D. H. Hines, A “random chemistry” algorithm for identifying collections of multiple contingencies that initiate cascading failure, IEEE Trans. Power Sys. 27, 1698 (2012).
  34. P. Rezaei, P. D. H. Hines, and M. J. Eppstein, Estimating cascading failure risk with random chemistry, IEEE Trans. Power Sys. 30, 2726 (2015).
  35. G. Liang, S. R. Weller, J. Zhao, F. Luo, and Z. Y. Dong, A framework for cyber-topology attacks: Line-switching and new attack scenarios, IEEE Trans. Smart Grid 10, 1704 (2019).
  36. S. Paul and Z. Ni, in 2018 International Joint Conference on Neural Networks (IJCNN), p. 1.
  37. Z. Ni and S. Paul, A multistage game in smart grid security: A reinforcement learning solution, IEEE Trans. Neural Netw. Learn. Syst. 30, 2684 (2019).
  38. D. S. H. van Hasselt and A. Guez, in Proceedings of the thirtieth AAAI Conference on Artificial Intelligence (2016), p. 1928.
  39. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, Human-level control through deep reinforcement learning, Nature 518, 529 (2015).
  40. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (The MIT Press, London, United Kingdom, 2018), 2nd ed.
  41. R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Mach. Learn. 8, 229 (1992).
  42. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, in Proceedings of the 33rd International Conference on Machine Learning Research 2016, Vol. 48 (2016), p. 1928.
  43. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, (2017), ArXiv:1707.06347.
  44. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning, (2015), arXiv preprint ArXiv:1509.02971.
  45. S. Fujimoto, H. van Hoof, and D. Meger, Addressing function approximation error in actor-critic methods, (2018), arXiv preprint ArXiv:1802.09477.
  46. https://github.com/AminMoradiXL/DQN_grid.

Outline

Information

PRX Energy 1, 033005– Published 28 November, 2022
  • Received 6 June 2022
  • Revised 23 September 2022
  • Accepted 10 October 2022
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation