Journal of the Korea Institute of Military Science and Technology (한국군사과학기술학회지)

The Korea Institute of Military Science and Technology (한국군사과학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Two Circle-based Aircraft Head-on Reinforcement Learning Technique using Curriculum

커리큘럼을 이용한 투서클 기반 항공기 헤드온 공중 교전 강화학습 기법 연구

  • Insu Hwang (Institute of Defense Advanced Technology Research Interlligence & Artificial Intelligence Autonomous Center, Agency for Defense Development) ;
  • Jungho Bae (Institute of Defense Advanced Technology Research Interlligence & Artificial Intelligence Autonomous Center, Agency for Defense Development)
  • 황인수 (국방과학연구소 국방첨단과학기술연구원 인공지능자율센터) ;
  • 배정호 (국방과학연구소 국방첨단과학기술연구원 인공지능자율센터)
  • Received : 2023.04.06
  • Accepted : 2023.07.04
  • Published : 2023.08.05
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, AI pilots using reinforcement learning are developing to a level that is more flexible than rule-based methods and can replace human pilots. In this paper, a curriculum was used to help head-on combat with reinforcement learning. It is not easy to learn head-on with a reinforcement learning method without a curriculum, but in this paper, through the two circle-based head-on air combat learning technique, ownship gradually increase the difficulty and become good at head-on combat. On the two-circle, the ATA angle between the ownship and target gradually increased and the AA angle gradually decreased while learning was conducted. By performing reinforcement learning with and w/o curriculum, it was engaged with the rule-based model. And as the win ratio of the curriculum based model increased to close to 100 %, it was confirmed that the performance was superior.

Keywords

Acknowledgement

이 논문은 2023년 정부의 재원으로 수행된 연구 결과임.

References

  1. Z. Wang, H. Li, H. Wu and Z. Wu, "Improving Maneuver Strategy in Air Combat by Alternate Freeze Games with a Deep Reinforcement Learning Algorithm," Hindawi Mathematical Problems in Engineering, 2023. 
  2. H. S. Inc., "Heron Systems at DARPA Alpha Dogfight Trials," (Sept. 25, 2020). Accessed: Dec. 06, 2022.,[Online Video]. Available: https://www.youtube.com/watch?v=lldE5XFTA88. 
  3. J. Oh, C. Kim, S. Ro, W. C. Choi and Y. Kim, "Air-to-air BFM Engagement Simulator for AI Engagement Model," in Proc. Korea Inst. Mil. Sci. Technol. Conf., pp. 1753-1754. 2022. 
  4. B. Vlahov, E. Squires, L. Strickland, and C. Pippin, "On Developing a UAV Pursuit-evasion Policy Using Reinforcement Learning," in Proc. IEEE Int. Conf. Mach. Learn. Appl. (ICMLA), pp. 859-864, 2018. 
  5. J. Bae, H. Jung, S. Kim, S. Kim and Y. Kim, "Deep Reinforcement Learning-Based Air-to-Air Combat Maneuver Generation in a Realistic Environment," in IEEE Access, Vol. 11, pp. 26427-26440, 2023.  https://doi.org/10.1109/ACCESS.2023.3257849
  6. M. Wiering and M. Van Otterlo, "Reinforcement Learning", Adaptation, Learning and Optimization, Vol. 12, p. 3, 2012. 
  7. T. Haarnoja, A. Zhou, K. Hartikainen and G. Tucker, "Soft Actor-Critic Algorithms and Applications," 2019, arxiv:1812.05905v2. 
  8. S. Hochreiter and J. Schmidhuber, "Long Short-term Memory," Neural Comput., Vol. 9, No. 8, pp. 1735-1780, 1997.  https://doi.org/10.1162/neco.1997.9.8.1735
  9. R. Portelas, C. Romac, K. Hofmann, "Automatic Curriculum Learning for Deep RL : A Short Survey," IJCAI, 2021. 
  10. J. Berndt, "Jsbsim: An Open Source Flight Dynamics Model in c++," in Modeling and Simulation Technologies Conference and Exhibit. American Institute of Aeronautics and Astronautics, 2004. 
  11. A. Pope, J. Jaime, D. Mi'covi'c, H. Diaz, D. Rosenbluth, L. Ritholtz, J. Twedt, T. Waler, K. Alcedo, and D. Javorsek, "Hierarchical Reinforcement Learning for Air-to-air Combat," 2021 International Conference on Unmanned Aircraft Systems(ICUAS), pp. 275-284, 2021.