Comparative Analysis of Artificial Intelligence-Based Learning Systems Used in Air Combat Simulation Environments

Emre Şafak; Tolga Erol; Hüseyin Furkan Ceran; Emre Ayvaz

doi:10.65834/jdsi.12.30

Authors

Emre Şafak HAVELSAN Inc. https://orcid.org/0000-0001-7579-3410
Tolga Erol HAVELSAN Inc. https://orcid.org/0000-0002-3015-0910
Hüseyin Furkan Ceran HAVELSAN Inc. https://orcid.org/0009-0006-2169-4680
Emre Ayvaz Turkish Air Force Command https://orcid.org/0009-0003-7876-609X

DOI:

https://doi.org/10.65834/jdsi.12.30

Keywords:

air combat simulation, reinforcement learning, military simulation training, deep reinforcement learning, multi-agent systems, human-in-the-loop, AI-driven tactical decision making

Abstract

The advancement of Artificial Intelligence (AI) technology is also manifesting itself in the defence sector. This study comprehensively reviews and comparatively analyses AI-based learning systems used in air combat simulation environments. The studies were categorised according to within-visual-range and beyond-visual-range scenarios and compared based on the methods used and the results obtained. These studies demonstrate that AI agents can develop realistic tactics, share situational awareness, and compete effectively with human pilots. In this study, methods such as Proximal Policy Optimisation (PPO) and Hierarchical Multi-Agent Reinforcement Learning (HMARL) were observed to be particularly prominent in terms of performance. Significant problems such as long training durations, high computational costs, partial observability, and sensitivity to sensor noise are the major challenges encountered. As a result of the study, it is concluded that intelligent agents developed in air combat simulation environments can enhance training efficiency by serving as teammates and opponents in pilot training.

References

Bae, J. H., Jung, H., Kim, S., Kim, S., & Kim, Y.-D. (2023). Deep reinforcement learning-based air-to-air combat maneuver generation in a realistic environment. IEEE Access, 11, 29249–29265. https://doi.org/10.1109/ACCESS.2023.3257849

Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv. https://doi.org/10.48550/arXiv.1909.07528

Chang, K.-C., Chu, K.-C., Wang, H.-C., Lin, Y.-C., & Pan, J.-S. (2020). Agent-based middleware framework using distributed CPS for improving resource utilization in smart city. Future Generation Computer Systems, 108, 445–453. https://doi.org/10.1016/j.future.2020.03.006

Dantas, J. P. A., Medeiros, F. L. L., Samersla, A. R., Botelho, P. L. R., Gomes, V. C. F., Silva, S. R., Ferreira, Y. D., Arantes, A. O., Aquino, M. R. C., & Maximo, M. R. O. A. (2025). Deep reinforcement learning agents with collective situational awareness for beyond visual range air combat. IEEE Access, 13, 143052–143069. https://doi.org/10.1109/ACCESS.2025.3597199

Francois-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., & Pineau, J. (2018). An introduction to deep reinforcement learning. arXiv. https://doi.org/10.48550/arXiv.1811.12560

Fujimoto, S., van Hoof, H., & Meger, D. (2018). Addressing function approximation error in actor-critic methods. arXiv. https://doi.org/10.48550/arXiv.1802.09477

Grondman, I., Busoniu, L., Lopes, G. A. D., & Babuska, R. (2012). A survey of actor critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6), 1291–1307. https://doi.org/10.1109/TSMCC.2012.2218595

Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv. https://doi.org/10.48550/arXiv.1801.01290

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

Huh, D., & Mohapatra, P. (2023). Multi-agent reinforcement learning: A comprehensive survey. arXiv. https://doi.org/10.48550/arXiv.2312.10256

Israelsen, B., Ahmed, N., Center, K., Green, R., & Bennett, W. Jr. (2018). Adaptive simulation-based training of artificial-intelligence decision makers using Bayesian optimization. Journal of Aerospace Information Systems, 15(2), 38–56. https://doi.org/10.2514/1.I010553

Jang, B., Kim, M., Harerimana, G., & Kim, J. W. (2019). Q-learning algorithms: A comprehensive classification and applications. IEEE Access, 7, 133653-133667. https://doi.org/10.1109/ACCESS.2019.2941229

Kong, W., Zhou, D., Du, Y., Zhou, Y., & Zhao, Y. (2023). Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat. IET Control Theory & Applications, 17(13), 1840–1862. https://doi.org/10.1049/cth2.12413

Mei, J., Li, G., & Huang, H. (2024). Deep reinforcement-learning-based air-combat maneuver generation framework. Mathematics, 12(19), 3020. https://doi.org/10.3390/math12193020

Mohsen, S., Elkaseer, A., & Scholz, S. G. (2021). Industry 4.0-oriented deep learning models for human activity recognition. IEEE Access, 9, 150508–150521. https://doi.org/10.1109/ACCESS.2021.3125733

Neville, K. J., Thom McLean III, A. L., Sherwood, S., Kaste, K., Walwanis, M., & Bolton, A. (2015). Live-Virtual-Constructive (LVC) Training in Air Combat: Emergent Training Opportunities and Fidelity Ripple Effects. In International Conference on Augmented Cognition (pp. 82-90). Springer. https://doi.org/10.1007/978-3-319-20816-9_9

Ning, Z., & Xie, L. (2024). A survey on multi-agent reinforcement learning and its application. Journal of Automation and Intelligence, 3(2), 73–91. https://doi.org/10.1016/j.jai.2024.02.003

Piao, H., Sun, Z., Meng, G., Chen, H., Qu, B., Lang, K., Sun, Y., Yang, S., & Peng, X. (2020). Beyond-visual-range air combat tactics auto-generation by reinforcement learning. In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8). IEEE. https://doi.org/10.1109/IJCNN48605.2020.9207088

Piao, H. Y., Yang, S., Chen, H., Li, J., Yu, J., Peng, X., Yang, X., Yang, Z., Sun, Z., & Chang, Y. (2024). Discovering expert-level air combat knowledge via deep excitatory-inhibitory factorized reinforcement learning. ACM Transactions on Intelligent Systems and Technology, 15(4), Article 65, 1–28. https://doi.org/10.1145/3653979

Pope, A. P., Ide, J. S., Mićović, D., Diaz, H., Twedt, J. C., Alcedo, K., Walker, T. T., Rosenbluth, D., Ritholtz, L., & Javorsek, D. II. (2023). Hierarchical reinforcement learning for air combat at DARPA's AlphaDogfight Trials. IEEE Transactions on Artificial Intelligence, 4(6), 1371–1385. https://doi.org/10.1109/TAI.2022.3222143

Saldiran, E., Hasanzade, M., Inalhan, G., & Tsourdos, A. (2024). Towards global explainability of artificial intelligence agent tactics in close air combat. Aerospace, 11(6), 415. https://doi.org/10.3390/aerospace11060415

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv. https://doi.org/10.48550/arXiv.1707.06347

Selmonaj, A., Antonucci, A., Schneider, A., Rüegsegger, M., & Sommer, M. (2025). Explaining strategic decisions in multi-agent reinforcement learning for aerial combat tactics. arXiv. https://doi.org/10.48550/arXiv.2505.11311

Selmonaj, A., Del Rio, G., Schneider, A., & Antonucci, A. (2025). Towards human engagement with realistic AI combat pilots. In Proceedings of the 13th International Conference on Human-Agent Interaction (HAI ’25) (pp. 1–3). ACM. https://doi.org/10.1145/3765766.3765827

Selmonaj, A., Szehr, O., Del Rio, G., Antonucci, A., Schneider, A., & Rüegsegger, M. (2023). Hierarchical multi-agent reinforcement learning for air combat maneuvering. arXiv. https://doi.org/10.48550/arXiv.2309.11247

Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023). Reinforcement learning algorithms: A brief survey. Expert Systems with Applications, 231, 120495. https://doi.org/10.1016/j.eswa.2023.120495

Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022). Curriculum learning: A survey. International Journal of Computer Vision, 130(6), 1526–1565. https://doi.org/10.1007/s11263-022-01611-x

Sivamayil, K., Rajasekar, E., Aljafari, B., Nikolovski, S., Vairavasundaram, S., & Vairavasundaram, I. (2023). A systematic study on reinforcement learning based applications. Energies, 16(3), 1512. https://doi.org/10.3390/en16031512

Sun, Z., Piao, H., Yang, Z., Zhao, Y., Zhan, G., Zhou, D., Meng, G., Chen, H., Chen, X., Qu, B., & Lu, Y. (2021). Multi-agent hierarchical policy gradient for Air Combat Tactics emergence via self-play. Engineering Applications of Artificial Intelligence, 98, 104112. https://doi.org/10.1016/j.engappai.2020.104112

Teng, T.-H., Tan, A.-H., Ong, W.-S., & Lee, K.-L. (2012). Adaptive CGF for pilots training in air combat simulation. In Proceedings of the 2012 15th International Conference on Information Fusion (FUSION) (pp. 2263–2270). IEEE.

Terven, J. (2025). Deep reinforcement learning: A chronological overview and methods. AI, 6(3), 46. https://doi.org/10.3390/ai6030046

Toubman, A., Roessingh, J. J. M., Spronck, P., Plaat, A., & van den Herik, H. J. (2015). Rewarding air combat behavior in training simulations. In 2015 IEEE International Conference on Systems, Man, and Cybernetics (pp. 1397–1402). IEEE. https://doi.org/10.1109/SMC.2015.248

Toubman, A., Roessingh, J. J. M., Spronck, P., Plaat, A., & van den Herik, H. J. (2015). Transfer learning of air combat behavior. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA) (pp. 226–231). IEEE. https://doi.org/10.1109/ICMLA.2015.61

Zhu, J., Kuang, M., Zhou, W., Shi, H., & Han, X. (2024). Mastering air combat game with deep reinforcement learning. Defence Technology, 34, 295–312. https://doi.org/10.1016/j.dt.2023.08.019

Comparative Analysis of Artificial Intelligence-Based Learning Systems Used in Air Combat Simulation Environments

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License

Make a Submission