Pekiştirmeli Öğrenme Yoluyla Bir Etmenin Kentsel Çevrede Yol Bulma Değerlendirilmesi

Abstract

Bu çalışma kentte yön bulma odağındaki davranış kalıplarını etmen tabanlı model üzerinden analiz eder. Çalışmanın hedefinde pekiştirmeli öğrenme ile yön bulmayı öğrenen etmenlerin davranışlarını anlamak bulunmaktadır. Çalışmanın hedefi doğrultusunda kullanılan yöntem etmen tabanlı modelleme olmuştur. Etmenler pekiştirmeli öğrenme ile Unity ML-Agents kullanılarak eğitilmiştir. Çalışma Sultan Ahmet Camii ve çevresini etmenin eğitimi için örnek bir çevre olarak almaktadır. Etmenler Sultan Ahmet Camii’yi bulmak hedefinde eğitilmiştir. Karmaşık ve basit öğrenme olarak iki farklı yöntemle eğitilen ve farklı başlangıç noktalarında yön bulma görevlerine başlatılan etmenlerin davranışları karşılaştırılmıştır. Bu çalışma pekiştirmeli öğrenme ile eğitilen etmenlerin yön bulma davranışlarını anlamaya çalışması bakımından literatürden farklılaşmaktadır. Bir diğer yönden bu araştırma yön bulma algısını merak kavramı üzerinden tartışmakta ve nöroşehircilikte önemli kavramlardan olan kent algısını etmenler üzerinden anlamaya çalışmaktadır. Etmenlerin eğitilmesi sonrasında etmenlerin yön bulma sürecinde hedefe ulaşırken ürettikleri alternatif güzergahlar analiz edilmiştir. Analiz sonucunda, etmenlerin güzergah dışına çıkarak, kentsel mekanda algıladığı farklı konumlarda gezebildiği ve sonrasında hedefe ulaştığı görülmüştür. Bu durum pekiştirmeli öğrenme ile eğitilen etmenin merakı olarak açıklanmıştır. Etmenlerin merak odaklı keşiflerinden ve geçmiş deneyimlerden elde edilen yön bulma davranışları, kentsel tasarımı destekleyecek nöroşehirciliğin bir girdisi olabilmesi yönünde çalışma literatüre katkıda bulunmaktadır. Bu çalışma, kentte yön bulma davranış kalıplarının; kullanıcı odaklı ve kentsel algı açısından zengin kentsel alanların geliştirilebilmesinde pekiştirmeli öğrenme yöntemi ile katkıda bulunmaktadır.

Keywords

• Davranışsal Kalıplar
• Motivasyon Merakı
• nöroşehircilik
• pekiştirmeli öğrenme
• Kentsel Tasarım.

Assessment of an Agent’s Wayfinding of the Urban Environment Through Reinforcement Learning

Abstract

This simulation study explores wayfinding motivated behavioral patterns in the city through agent-based modelling. Agents were trained using Unity’s ML-Agents toolkit with reinforcement learning. The study uses the Sultan Ahmet Mosque and its surrounding boundary as a model environment for the training of an agent’s wayfinding. Agents are trained to locate the Sultan Ahmet Mosque target. The behaviors of agents trained with two different methods, “Complex” and “Simple” learning, comparing their navigation quests at various difficulty levels featuring respawn points. After the training of the agents, the alternative routes produced while attaining the target during the wayfinding process were analyzed. As an outcome of the analysis, it was observed that the agents were prone to go off-route, navigate to different locations they perceived in the urban space, and then would reach the target. This occurrence is justified as an agent’s curiosity trained through reinforcement learning. This study differs from the literature in a way that it attempts to understand the navigational behavior of agents that were trained with reinforcement learning. Moreover, this research discusses the perception of wayfinding through curiosity and aims to make a comprehension of the perception of the city, which is one of the key ideas in neurourbanism. The study contributes to the literature by showing that wayfinding behaviors acquired from agents’ curiosity-driven explorations and past experiences can be an input for neurourbanism, supporting urban design. It informs urban enhancements that are user-centric and rich in urban perception using the reinforcement learning method.

Keywords

• Behavioral Patterns
• Motivational Curiosity
• Neurourbanism
• Reinforcement learning
• Urban Design.

References

1. Adli, M., Berger, M., Brakemeier, E.-L., Engel, L., Fingerhut, J., Gomez-Carrillo, A., Hehl, R., Heinz, A., Mayer, J. H., Mehran, N., Tolaas, S., Walter, H., Weiland, U., & Stollmann, J. (2017). Neurourbanism: towards a new discipline. The Lancet Psychiatry, 4(3), pp. 183–185. https://doi.org/10.1016/s2215-0366(16)30371-6
2. Arbib, M. A. (2021). When brains meet buildings. Oxford University Press.
3. As, I., Basu, P., & Talwar, P. (Eds.). (2022). Artificial intelligence in urban planning and design: technologies, implementation, and impacts. Elsevier.
4. Banczyk, M., & Potts, J. (2018). City as Neural Platform-Toward New Economics of a City. https://dx.doi.org/10.2139/ssrn.3233686
5. Baumann, P.S., Söderström, O., Abrahamyan Empson, L. (2020). Urban remediation: a new recovery-oriented strategy to manage urban stress after first-episode psychosis. Social Psychiatry and Psychiatric Epidemiology 55, 273–283. https://doi.org/10.1007/s00127-019-01795-7
6. Bibri, S. E., Krogstie, J., Kaboli, A., & Alahi, A. (2024). Smarter eco-cities and their leading-edge artificial intelligence of things solutions for environmental sustainability: A comprehensive systematic review. Environmental Science and Ecotechnology, 19, 100330. https://doi.org/10.1016/j.ese.2023.100330
7. Botteghi, N., Sirmacek, B., Poel, M., Brune, C., & Schulte, R. (2021). Curiosity-driven reinforcement learning agent for mapping unknown indoor environments. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 5(1), 129-136. https://doi.org/10.5194/isprs-annals-V-1-2021-129-2021
8. Bouton, M., Nakhaei, A., Fujimura, K., & Kochenderfer, M. J. (2019, June). Safe reinforcement learning with scene decomposition for navigating complex urban environments. 2019 IEEE Intelligent Vehicles Symposium (IV), 1469-1476. https://doi.org/10.1109/IVS.2019.8813803

9. Buttazzoni, A., Doherty, S., & Minaker, L. (2022). How do urban environments affect young people’s mental health? A novel conceptual framework to bridge public health, planning, and neurourbanism. Public Health Reports, 137(1), 48-61. https://doi.org/10.1177/0033354920982088
10. Cutitoi, A. C. (2022). Smart city analytics, digital twin simulation and visualization modeling technologies, and cognitive data mining algorithms in sustainable urban governance networks. Geopolitics, History, and International Relations, 14(1), pp. 107-122. https://doi.org/10.22381/GHIR14120227
11. Deshpande, N., Vaufreydaz, D., & Spalanzani, A. (2021, September). Navigation in urban environments amongst pedestrians using multi-objective deep reinforcement learning. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 923-928.
12. Engelbrecht, D. (2023). Unity ml-agents. Introduction to Unity ML-Agents: Understand the Interplay of Neural Networks and Simulation Space Using the Unity ML-Agents Package (pp. 87-135). Apress. https://doi.org/10.1007/978-1-4842-8998-3_6
13. Ghazal, T. M., Hasan, M. K., Alshurideh, M. T., Alzoubi, H. M., Ahmad, M., Akbar, S. S., ... & Akour, I. A. (2021). IoT for smart cities: Machine learning approaches in smart healthcare—A review. Future Internet, 13(8), 218. https://doi.org/10.3390/fi13080218
14. Görgül, E., & Yıldız Özkan, D. (2024). Neuro-urbanism: Measurement of the street enclosure and its influence on human physiology through wearable sensors. Journal of Design, Planning and Aesthetics Research 3(1), 56-72. https://doi.org/10.55755/deparch.2024.27
15. Han, Z., Yan, W., & Liu, G. (2021). A performance-based urban block generative design using Deep Reinforcement Learning and Computer Vision. Proceedings of the 2020 Digital FUTURES, pp. 134–143. https://doi.org/10.1007/978-981-33-4400-6_13
16. Heino, J. (2020). Creating connections to land through art: allowing curiosity to take the lead in urban spaces. Journal of childhood studies 45(1), 48-55. https://doi.org/10.18357/jcs00019400
17. Huang, C., Zhang, R., Ouyang, M., Wei, P., Lin, J., Su, J., & Lin, L. (2021). Deductive reinforcement learning for visual autonomous urban driving navigation. IEEE Transactions on Neural Networks and Learning Systems, 32(12), 5379-5391. https://doi.org/10.1109/TNNLS.2021.3109284
18. Huang, X., Deng, H., Zhang, W., Song, R., & Li, Y. (2021). Towards multi-modal perception-based navigation: a deep reinforcement learning method. IEEE Robotics and Automation Letters, 6(3), 4986-4993. https://doi.org/10.1109/LRA.2021.3064461
19. Intrator, O. & Intrator, N. (2001). Interpreting neural-network results: a simulation study. Computational Statistics & Data Analysis, 37(3), 373-393. https://doi.org/10.1016/S0167-9473(01)00016-0
20. Jeffery, K. (2019). Urban architecture: a cognitive neuroscience perspective. The Design Journal, 22(6), 853-872. https://doi.org10.1080/14606925.2019.1662666
21. Juliani, A., Berges, V. P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao Y., Henry, H., Mattar, M., & Lange, D. (2018). Unity: a general platform for intelligent agents. arXiv:1809.02627. https://doi.org/10.48550/arxiv.1809.02627
22. Kee, T.; Ho, W. K. (2024). Optimizing machine learning models for urban studies: A comparative analysis of hyperparameter tuning methods. Preprints 2024, 2024060264. https://doi.org/10.20944/preprints202406.0264.v1
23. Küçük, S., & Yüceer, H. (2022). Neuroscience and the cities: Neurourbanism. Gazi University Journal of Science Part B: Art Humanities Design and Planning, 10(3), 287-301. https://dergipark.org.tr/en/pub/gujsb/issue/72742/1140128#article_cite
24. La Rocca, F. (2017). A theoretical approach to the flâneur and the sensitive perception of the metropolis. Sociétés, (1), 9-17. https://doi.org/10.3917/soc.135.0009
25. Lanham, M. (2018). Learn unity ml-agents—fundamentals of unity machine learning: incorporate new powerful ml algorithms such as deep reinforcement learning for games. Packt Publishing.
26. Leomi, E. (2015). Tokyo flâneur: A study of urban experience in narrative [Master's thesis, Aalto University]. Aalto University. https://urn.fi/URN:NBN:fi:aalto-201602161305
27. Macal, C. M., & North, M. J. (2009). Agent-based modeling and simulation. Proceedings of the 2009 Winter Simulation Conference (WSC). https://doi.org/10.1109/wsc.2009.5429318
28. Makanadar, A. (2024). Neuro-adaptive architecture: Buildings and city design that respond to human emotions, cognitive states. Research in Globalization, 100222. https://doi.org/10.1016/j.resglo.2024.100222
29. Murail, E. (2017). A body passes by: the flâneur and the senses in nineteenth-century London and Paris. The Senses and Society, 12(2), pp. 162-176. https://doi.org/10.1080/17458927.2017.1310454
30. Nilsson, T. (2011). Curio-Urbia: A curiosity exploration of hidden urban interactions [Master's thesis, Malmö University]. Malmö University. https://mau.diva-portal.org/smash/get/diva2:1482935/FULLTEXT01.pdf
31. Ndaguba, E., Cilliers, J., Mbanga, S., Brown, K., & Ghosh, S. (2022). Re-imaging the future in urban studies and built environment discourse: A neurourbanism perspective. Buildings, 12(12), 2056. https://doi.org/10.3390/buildings12122056
32. Olszewska-Guizzo, A. (2021). Neuroscience-based urban design for mentally healthy cities. Urban Health and Wellbeing Programme (pp. 13–16). Springer. https://doi.org/10.1007/978-981-33-6036-5_3
33. Phillips, R., Evans, B., & Muirhead, S. (2015). Curiosity, place and wellbeing: Encouraging place-specific curiosity as a ‘way to wellbeing’. Environment and Planning A, 47(11), 2339-2354. https://doi.org/10.1177/0308518X15599290
34. Portugali, J., & Haken, H. (2018). Movement, cognition and the city. Built Environment, 44(2), 136-161. http://www.jstor.org/stable/45173822
35. Pykett, J., Osborne, T., & Resch, B. (2020). From urban stress to neurourbanism: How should we research city well-being? Annals of the American Association of Geographers, 110(6), 1936–1951. https://doi.org/10.1080/24694452.2020.1736982
36. Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: mapping the inequality of urban perception. PloS one, 8(7), e68400. https://doi.org/10.1371/journal.pone.0119352
37. Silvia, P. J. (2012). Curiosity and motivation. The Oxford Handbook of Human Motivation (pp. 157-166). https://doi.org/10.1093/oxfordhb/9780195399820.013.0010
38. Son, T. H., Weedon, Z., Yigitcanlar, T., Sanchez, T., Corchado, J. M., & Mehmood, R. (2023). Algorithmic urban planning for smart and sustainable development: Systematic review of the literature. Sustainable Cities and Society, 104562 https://doi.org/10.1016/j.scs.2023.104562
39. Sutton, R. S. (1992). Introduction: the challenge of reinforcement learning. Reinforcement Learning (pp. 1-3). Springer.
40. Sutton, R. S. & Barto, A. G. (1999). Reinforcement learning. Journal of Cognitive Neuroscience, 11(1), pp. 126-134.
41. Tewari, K., Tewari, M. & Niyogi, D. (2023). Need for considering urban climate change factors on stroke, neurodegenerative diseases, and mood disorders studies. Computational Urban Science, 3(4). https://doi.org/10.1007/s43762-023-00079-w
42. Tolunay, F. (2022). The neurological and psychological effects of human and nature interaction: Walking in natural landscape and landscaped garden environments [Doctoral dissertation, Bilkent Universitesi].
43. Williamson, B. (2019). Computing brains: learning algorithms and neurocomputation in the smart city. The Social Power of Algorithms (pp. 81-99). Routledge.
44. Xu, J., Liu, N., Polemiti, E. (2023). Effects of urban living environments on mental health in adults. Nature Medicine 29, 1456–1467. https://doi.org/10.1038/s41591-023-02365-w
45. Yao, Z. (2023). A Framework Study of the Effects of Neurourbanism Perspectives on the Impact of Neighbourhood Green Space Exposure on Adolescent Mental Health. Academic Journal of Environment & Earth Science, 5(9), 1-7. https://dx.doi.org/10.25236/AJEE.2023.050901
46. Ye, Q., Feng, Y., Candela, E., Escribano Macias, J., Stettler, M., & Angeloudis, P. (2021). Spatial-temporal flows-adaptive street layout control using reinforcement learning. Sustainability, 14(1), 107. https://doi.org/10.3390/su14010107
47. Zhang, X., Huang, T., Gulakhmadov, A., Song, Y., Gu, X., Zeng, J., ... & Niyogi, D. (2022). Deep learning-based 500 m spatio-temporally continuous air temperature generation by fusing multi-source data. Remote Sensing, 14(15), 3536. https://doi.org/10.3390/rs14153536
48. Zheng, Y., Lin, Y., Zhao, L., Wu, T., Jin, D., & Li, Y. (2023). Spatial planning of urban communities via Deep Reinforcement Learning. Nature Computational Science, 3(9), 748–762. https://doi.org/10.1038/s43588-023-00503-5