Military Operations Research and Mission Engineering with Reinforcement Learning-based Virtual Pilots

Öz

This study explores the use of reinforcement learning algorithms trained in mission-oriented simulation environments, not only as systems executing operational tasks but also as decision-support tools for military operations analysis. Unlike traditional approaches such as linear programming, dynamic programming, or Monte Carlo simulations, which rely on predefined rules and probabilities and thus struggle to reflect real-world uncertainties, RL-based virtual pilots are trained across diverse scenarios and evaluated through repeated task executions, producing rich behavioral datasets. Post-operation analyses yield tactical insights into hit-loss ratios, ammunition usage, route preferences, enemy threat levels, and allied vulnerabilities, while statistical and explainable methods such as heatmaps, feature importance scores, and decision trees enable predictive assessments and cross-scenario comparisons. The findings demonstrate that combining RL policies with explainable analysis techniques facilitates the development of next-generation military analysis infrastructures that support decision-making as well as mission execution.

Anahtar Kelimeler

• Simulation based Training
• Behaviour Models
• Reinforcement Learning

Pekiştirmeli Öğrenme Tabanlı Sanal Pilotlar ile Askeri Harekat Analizi ve Görev Mühendisliği

Öz

Bu çalışmada görev odaklı simülasyonlarda pekiştirmeli öğrenme ile eğitilmiş sanal ajanların, yalnızca operasyonel görev icracısı değil, aynı zamanda askeri harekat analizinde karar destek aracı olarak kullanılabilirliği incelenmektedir. Doğrusal, dinamik programlama, Monte Carlo simülasyonları gibi geleneksel yöntemler, görev kurallarının önceden tanımlanmasını gerektirdiğinden, gerçek dünyadaki belirsizlikleri yeterince yansıtamamaktadır. Buna karşılık, farklı senaryolarda eğitilen RL tabanlı sanal pilotlar çok sayıda görev icrasıyla davranışsal veri üretmekte; isabet-zaiyat oranları, mühimmat kullanımı, rota tercihleri, tehdit düzeyleri ve müttefik zafiyetleri üzerinden taktiksel çıkarımlar yapılabilmektedir. Isı haritaları, karar ağaçları ve öznitelik önem analizleri gibi yöntemlerle öngörüsel değerlendirmeler gerçekleştirilebilmekte, senaryolar arası farklar sayısal olarak ortaya konmaktadır. Sonuç olarak, RL ajanlarının öğrenilmiş politikalarının istatistiksel ve açıklanabilir analiz teknikleriyle birleştirilmesi, yalnızca görev icra eden değil aynı zamanda karar destek süreçlerinde etkin rol üstlenen yeni nesil askeri analiz altyapılarının geliştirilmesini mümkün kılmaktadır.

Anahtar Kelimeler

• Simülasyon Tabanlı Eğitim
• Davranış Modelleri
• Pekiştirmeli Öğrenme

Kaynakça

1. [1] Kılıç M., “Komuta Kontrol Fonksiyon Alanı ve Türk Silahlı Kuvvetleri İçin Öneriler”, Savunma ve Güvenlik Araştırmaları Dergisi, 1: 217–242, (2024).
2. [2] Türkkan B. Ö., Asker C., “Personel Atama Problemi İçin Çok Kriterli ve Çok Amaçlı Karar Verme Yöntemleri ile Hibrit Bir Model Önerisi”, Gümüşhane Üniversitesi Sosyal Bilimler Dergisi (GUSBID), 14: 1–15, (2023).
3. [3] Dağıstanlı H. A., “Çok ürünlü çok depolu araç rotalama problemi: askerî ilaç fabrikası örneği”, Politeknik Dergisi, 1–1, (2023).
4. [4] Çetinkaya A. F., Demirel N., “Askerî operasyonlarda araç seçimi: karar verme sürecinde kullanılabilecek analitik yaklaşımlar”, Uluslararası Türk Dünyası Araştırmaları Dergisi, 6: 35–74, (2023).
5. [5] StrandA., PatrickR.G., KarstenB., “Modeling air combat behaviorfor simulation-based pilot training: A survey of machine learning approaches”, IEEE Access, (2025).
6. [6] Selmonaj A., et al., “Explaining Strategic Decisions in Multi-Agent Reinforcement Learning for Aerial Combat Tactics”, arXiv preprint arXiv:2505.11311, (2025).
7. [7] Fan W., et al., “Multi-agent modeling and simulation in the AI age”, Tsinghua Science and Technology, 26: 608–624, (2021).
8. [8] Hitch C., “Economics and Military Operations Research”, The Review of Economics and Statistics, 40: 199–209, (1958).

9. [9] Lepingwell J. W. R., “The Laws of Combat? Lanchester Reexamined”, International Security, 12: 89–134, (1987).
10. [10] Hitch C., “Uncertainties in Operations Research”, Operations Research, 8: 437–445, (1960).
11. [11] Royset J. O., et al., “Routing Military Aircraft With A Constrained Shortest-Path Algorithm”, Military Operations Research, 14: 31–52, (2009).
12. [12] Rogers M. B., et al., “A Military Logistics Network Planning System”, Military Operations Research, 23: 5–24, (2018).
13. [13] Caballero W. N., et al., “Leveraging Behavioral Game Theory to Inform Military Operations Planning”, Military Operations Research, 25: 5–22, (2020).
14. [14] McKenna R. S., et al., “Approximate dynamic programming for the military inventory routing problem”, Annals of Operations Research, 288: 391–416, (2020).
15. [15] Bertsimas D., Alex P., “Solving LargeScale Weapon Target Assignment Problems in Seconds Using Branch-Price-And-Cut”, Naval Research Logistics (NRL), (2025).
16. [16] GhoshA., etal., “Learning Action Representations for Reinforcement Learning”, International Conference on Machine Learning (ICML), (2020).
17. [17] Kim H., et al., “Multi-Agent Reinforcement Learning for C2 Planning in Digital Twin Battlefields”, Journal of Defense Modeling and Simulation, (2022).
18. [18] Yu C., et al., “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games”, arXiv preprint arXiv:2103.01955, (2021).
19. [19] Soleyman S., et al., “Multi-agent autonomous battle management using deep neuroevolution”, Unmanned Systems Technology, 23: 11758, (2021).
20. [20] Pope A. P., et al., “Hierarchical reinforcement learning for air combat at DARPA’s AlphaDogfight trials”, IEEE Transactions on Artificial Intelligence, 4: 1371–1385, (2022).
21. [21] Möbius M., et al., “Incorporation of militarydoctrinesandobjectivesintoanAIagent via natural language and reward in reinforcement learning”, Winter Simulation Conference (WSC), (2023).
22. [22] Dimitriu A., et al., “A Reinforcement Learning Approach to Military Simulations in Command: Modern Operations”, IEEE Access, (2024).
23. [23] Henslee A., et al., “Data-Driven Reinforcement Learning for Mission Engineering and Combat Simulation”, IUTAM Symposium on Optimal Guidance and Control for Autonomous Systems, Switzerland, (2023).
24. [24] Yüksek B., et al., “Intelligent wargaming approach to increase course of action effectiveness in military operations”, AIAA SciTech Forum, (2023).
25. [25] Schlabach J., “Operations Research Is Key to Fulfilling the Promise of Battlefield Artificial Intelligence”, Military Operations Research, 26: 9–23, (2021).
26. [26] de Reus N., et al., "Generating Explainable Military Courses of Action.", International Conference on Military Communication and Information Systems (ICMCIS), IEEE, (2024).
27. [27] Shah K., et al., "XAI-BattleOps: Explainable AI-based Energy Consumption Analytical Framework for UAV-assisted Battlefield Operations.", IEEE International Conference on Blockchain and Distributed Systems Security (ICBDS), IEEE, (2024).
28. [28] Selmonaj A., et al., "Explainability in MultiAgent Reinforcement Learning for Air Combat Tactics.", M&S as Enabler for Digital Transformation in NATO and Nations, 217, (2024).
29. [29] Kazhdan D., Zohreh S., Pietro L., "Marleme: A multi-agent reinforcement learning model extraction library.", International Joint Conference on Neural Networks (IJCNN), IEEE, (2020).
30. [30] Basak A., et al., “Utility of doctrine with multi-agent RL for military engagements.”, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications IV, Vol. 12113, SPIE, (2022).
31. [31] Narendra A., Samuel F. R., Chloe A. R., “Navigational efficiency of nocturnal Myrmecia ants suffers at low light levels”, PLoS One, 8: e58801, (2013).
32. [32] Altun H. O., et al., “Strategic Implementation of Super-agents in Heterogeneous Multi-Agent Training for Advanced Military Simulation Adaptability”, IEEE Access, (2025).