Assessment of an Agent’s Wayfinding of the Urban Environment Through Reinforcement Learning

Year 2024, Volume: 5 Issue: 2, 259 - 278, 30.09.2024

Mahad Imhemed , Can Uzun

https://doi.org/10.53710/jcode.1512798

Abstract

This simulation study explores wayfinding motivated behavioral patterns in the city through agent-based modelling. Agents were trained using Unity’s ML-Agents toolkit with reinforcement learning. The study uses the Sultan Ahmet Mosque and its surrounding boundary as a model environment for the training of an agent’s wayfinding. Agents are trained to locate the Sultan Ahmet Mosque target. The behaviors of agents trained with two different methods, “Complex” and “Simple” learning, comparing their navigation quests at various difficulty levels featuring respawn points. After the training of the agents, the alternative routes produced while attaining the target during the wayfinding process were analyzed. As an outcome of the analysis, it was observed that the agents were prone to go off-route, navigate to different locations they perceived in the urban space, and then would reach the target. This occurrence is justified as an agent’s curiosity trained through reinforcement learning. This study differs from the literature in a way that it attempts to understand the navigational behavior of agents that were trained with reinforcement learning. Moreover, this research discusses the perception of wayfinding through curiosity and aims to make a comprehension of the perception of the city, which is one of the key ideas in neurourbanism. The study contributes to the literature by showing that wayfinding behaviors acquired from agents’ curiosity-driven explorations and past experiences can be an input for neurourbanism, supporting urban design. It informs urban enhancements that are user-centric and rich in urban perception using the reinforcement learning method.

Keywords

Behavioral Patterns, Motivational Curiosity, Neurourbanism, Reinforcement learning, Urban Design.

Pekiştirmeli Öğrenme Yoluyla Bir Etmenin Kentsel Çevrede Yol Bulma Değerlendirilmesi

Abstract

Bu çalışma kentte yön bulma odağındaki davranış kalıplarını etmen tabanlı model üzerinden analiz eder. Çalışmanın hedefinde pekiştirmeli öğrenme ile yön bulmayı öğrenen etmenlerin davranışlarını anlamak bulunmaktadır. Çalışmanın hedefi doğrultusunda kullanılan yöntem etmen tabanlı modelleme olmuştur. Etmenler pekiştirmeli öğrenme ile Unity ML-Agents kullanılarak eğitilmiştir. Çalışma Sultan Ahmet Camii ve çevresini etmenin eğitimi için örnek bir çevre olarak almaktadır. Etmenler Sultan Ahmet Camii’yi bulmak hedefinde eğitilmiştir. Karmaşık ve basit öğrenme olarak iki farklı yöntemle eğitilen ve farklı başlangıç noktalarında yön bulma görevlerine başlatılan etmenlerin davranışları karşılaştırılmıştır. Bu çalışma pekiştirmeli öğrenme ile eğitilen etmenlerin yön bulma davranışlarını anlamaya çalışması bakımından literatürden farklılaşmaktadır. Bir diğer yönden bu araştırma yön bulma algısını merak kavramı üzerinden tartışmakta ve nöroşehircilikte önemli kavramlardan olan kent algısını etmenler üzerinden anlamaya çalışmaktadır. Etmenlerin eğitilmesi sonrasında etmenlerin yön bulma sürecinde hedefe ulaşırken ürettikleri alternatif güzergahlar analiz edilmiştir. Analiz sonucunda, etmenlerin güzergah dışına çıkarak, kentsel mekanda algıladığı farklı konumlarda gezebildiği ve sonrasında hedefe ulaştığı görülmüştür. Bu durum pekiştirmeli öğrenme ile eğitilen etmenin merakı olarak açıklanmıştır. Etmenlerin merak odaklı keşiflerinden ve geçmiş deneyimlerden elde edilen yön bulma davranışları, kentsel tasarımı destekleyecek nöroşehirciliğin bir girdisi olabilmesi yönünde çalışma literatüre katkıda bulunmaktadır. Bu çalışma, kentte yön bulma davranış kalıplarının; kullanıcı odaklı ve kentsel algı açısından zengin kentsel alanların geliştirilebilmesinde pekiştirmeli öğrenme yöntemi ile katkıda bulunmaktadır.

Keywords

Davranışsal Kalıplar, Motivasyon Merakı, nöroşehircilik, pekiştirmeli öğrenme, Kentsel Tasarım.

References

• Adli, M., Berger, M., Brakemeier, E.-L., Engel, L., Fingerhut, J., Gomez-Carrillo, A., Hehl, R., Heinz, A., Mayer, J. H., Mehran, N., Tolaas, S., Walter, H., Weiland, U., & Stollmann, J. (2017). Neurourbanism: towards a new discipline. The Lancet Psychiatry, 4(3), pp. 183–185. https://doi.org/10.1016/s2215-0366(16)30371-6
• Arbib, M. A. (2021). When brains meet buildings. Oxford University Press.
• As, I., Basu, P., & Talwar, P. (Eds.). (2022). Artificial intelligence in urban planning and design: technologies, implementation, and impacts. Elsevier.
• Banczyk, M., & Potts, J. (2018). City as Neural Platform-Toward New Economics of a City. https://dx.doi.org/10.2139/ssrn.3233686
• Baumann, P.S., Söderström, O., Abrahamyan Empson, L. (2020). Urban remediation: a new recovery-oriented strategy to manage urban stress after first-episode psychosis. Social Psychiatry and Psychiatric Epidemiology 55, 273–283. https://doi.org/10.1007/s00127-019-01795-7
• Bibri, S. E., Krogstie, J., Kaboli, A., & Alahi, A. (2024). Smarter eco-cities and their leading-edge artificial intelligence of things solutions for environmental sustainability: A comprehensive systematic review. Environmental Science and Ecotechnology, 19, 100330. https://doi.org/10.1016/j.ese.2023.100330
• Botteghi, N., Sirmacek, B., Poel, M., Brune, C., & Schulte, R. (2021). Curiosity-driven reinforcement learning agent for mapping unknown indoor environments. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 5(1), 129-136. https://doi.org/10.5194/isprs-annals-V-1-2021-129-2021
• Bouton, M., Nakhaei, A., Fujimura, K., & Kochenderfer, M. J. (2019, June). Safe reinforcement learning with scene decomposition for navigating complex urban environments. 2019 IEEE Intelligent Vehicles Symposium (IV), 1469-1476. https://doi.org/10.1109/IVS.2019.8813803
• Buttazzoni, A., Doherty, S., & Minaker, L. (2022). How do urban environments affect young people’s mental health? A novel conceptual framework to bridge public health, planning, and neurourbanism. Public Health Reports, 137(1), 48-61. https://doi.org/10.1177/0033354920982088
• Cutitoi, A. C. (2022). Smart city analytics, digital twin simulation and visualization modeling technologies, and cognitive data mining algorithms in sustainable urban governance networks. Geopolitics, History, and International Relations, 14(1), pp. 107-122. https://doi.org/10.22381/GHIR14120227
• Deshpande, N., Vaufreydaz, D., & Spalanzani, A. (2021, September). Navigation in urban environments amongst pedestrians using multi-objective deep reinforcement learning. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 923-928.
• Engelbrecht, D. (2023). Unity ml-agents. Introduction to Unity ML-Agents: Understand the Interplay of Neural Networks and Simulation Space Using the Unity ML-Agents Package (pp. 87-135). Apress. https://doi.org/10.1007/978-1-4842-8998-3_6
• Ghazal, T. M., Hasan, M. K., Alshurideh, M. T., Alzoubi, H. M., Ahmad, M., Akbar, S. S., ... & Akour, I. A. (2021). IoT for smart cities: Machine learning approaches in smart healthcare—A review. Future Internet, 13(8), 218. https://doi.org/10.3390/fi13080218
• Görgül, E., & Yıldız Özkan, D. (2024). Neuro-urbanism: Measurement of the street enclosure and its influence on human physiology through wearable sensors. Journal of Design, Planning and Aesthetics Research 3(1), 56-72. https://doi.org/10.55755/deparch.2024.27
• Han, Z., Yan, W., & Liu, G. (2021). A performance-based urban block generative design using Deep Reinforcement Learning and Computer Vision. Proceedings of the 2020 Digital FUTURES, pp. 134–143. https://doi.org/10.1007/978-981-33-4400-6_13
• Heino, J. (2020). Creating connections to land through art: allowing curiosity to take the lead in urban spaces. Journal of childhood studies 45(1), 48-55. https://doi.org/10.18357/jcs00019400
• Huang, C., Zhang, R., Ouyang, M., Wei, P., Lin, J., Su, J., & Lin, L. (2021). Deductive reinforcement learning for visual autonomous urban driving navigation. IEEE Transactions on Neural Networks and Learning Systems, 32(12), 5379-5391. https://doi.org/10.1109/TNNLS.2021.3109284
• Huang, X., Deng, H., Zhang, W., Song, R., & Li, Y. (2021). Towards multi-modal perception-based navigation: a deep reinforcement learning method. IEEE Robotics and Automation Letters, 6(3), 4986-4993. https://doi.org/10.1109/LRA.2021.3064461
• Intrator, O. & Intrator, N. (2001). Interpreting neural-network results: a simulation study. Computational Statistics & Data Analysis, 37(3), 373-393. https://doi.org/10.1016/S0167-9473(01)00016-0
• Jeffery, K. (2019). Urban architecture: a cognitive neuroscience perspective. The Design Journal, 22(6), 853-872. https://doi.org10.1080/14606925.2019.1662666
• Juliani, A., Berges, V. P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao Y., Henry, H., Mattar, M., & Lange, D. (2018). Unity: a general platform for intelligent agents. arXiv:1809.02627. https://doi.org/10.48550/arxiv.1809.02627
• Kee, T.; Ho, W. K. (2024). Optimizing machine learning models for urban studies: A comparative analysis of hyperparameter tuning methods. Preprints 2024, 2024060264. https://doi.org/10.20944/preprints202406.0264.v1
• Küçük, S., & Yüceer, H. (2022). Neuroscience and the cities: Neurourbanism. Gazi University Journal of Science Part B: Art Humanities Design and Planning, 10(3), 287-301. https://dergipark.org.tr/en/pub/gujsb/issue/72742/1140128#article_cite
• La Rocca, F. (2017). A theoretical approach to the flâneur and the sensitive perception of the metropolis. Sociétés, (1), 9-17. https://doi.org/10.3917/soc.135.0009
• Lanham, M. (2018). Learn unity ml-agents—fundamentals of unity machine learning: incorporate new powerful ml algorithms such as deep reinforcement learning for games. Packt Publishing.
• Leomi, E. (2015). Tokyo flâneur: A study of urban experience in narrative [Master's thesis, Aalto University]. Aalto University. https://urn.fi/URN:NBN:fi:aalto-201602161305
• Macal, C. M., & North, M. J. (2009). Agent-based modeling and simulation. Proceedings of the 2009 Winter Simulation Conference (WSC). https://doi.org/10.1109/wsc.2009.5429318
• Makanadar, A. (2024). Neuro-adaptive architecture: Buildings and city design that respond to human emotions, cognitive states. Research in Globalization, 100222. https://doi.org/10.1016/j.resglo.2024.100222
• Murail, E. (2017). A body passes by: the flâneur and the senses in nineteenth-century London and Paris. The Senses and Society, 12(2), pp. 162-176. https://doi.org/10.1080/17458927.2017.1310454
• Nilsson, T. (2011). Curio-Urbia: A curiosity exploration of hidden urban interactions [Master's thesis, Malmö University]. Malmö University. https://mau.diva-portal.org/smash/get/diva2:1482935/FULLTEXT01.pdf
• Ndaguba, E., Cilliers, J., Mbanga, S., Brown, K., & Ghosh, S. (2022). Re-imaging the future in urban studies and built environment discourse: A neurourbanism perspective. Buildings, 12(12), 2056. https://doi.org/10.3390/buildings12122056
• Olszewska-Guizzo, A. (2021). Neuroscience-based urban design for mentally healthy cities. Urban Health and Wellbeing Programme (pp. 13–16). Springer. https://doi.org/10.1007/978-981-33-6036-5_3
• Phillips, R., Evans, B., & Muirhead, S. (2015). Curiosity, place and wellbeing: Encouraging place-specific curiosity as a ‘way to wellbeing’. Environment and Planning A, 47(11), 2339-2354. https://doi.org/10.1177/0308518X15599290
• Portugali, J., & Haken, H. (2018). Movement, cognition and the city. Built Environment, 44(2), 136-161. http://www.jstor.org/stable/45173822
• Pykett, J., Osborne, T., & Resch, B. (2020). From urban stress to neurourbanism: How should we research city well-being? Annals of the American Association of Geographers, 110(6), 1936–1951. https://doi.org/10.1080/24694452.2020.1736982
• Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: mapping the inequality of urban perception. PloS one, 8(7), e68400. https://doi.org/10.1371/journal.pone.0119352
• Silvia, P. J. (2012). Curiosity and motivation. The Oxford Handbook of Human Motivation (pp. 157-166). https://doi.org/10.1093/oxfordhb/9780195399820.013.0010
• Son, T. H., Weedon, Z., Yigitcanlar, T., Sanchez, T., Corchado, J. M., & Mehmood, R. (2023). Algorithmic urban planning for smart and sustainable development: Systematic review of the literature. Sustainable Cities and Society, 104562 https://doi.org/10.1016/j.scs.2023.104562
• Sutton, R. S. (1992). Introduction: the challenge of reinforcement learning. Reinforcement Learning (pp. 1-3). Springer.
• Sutton, R. S. & Barto, A. G. (1999). Reinforcement learning. Journal of Cognitive Neuroscience, 11(1), pp. 126-134.
• Tewari, K., Tewari, M. & Niyogi, D. (2023). Need for considering urban climate change factors on stroke, neurodegenerative diseases, and mood disorders studies. Computational Urban Science, 3(4). https://doi.org/10.1007/s43762-023-00079-w
• Tolunay, F. (2022). The neurological and psychological effects of human and nature interaction: Walking in natural landscape and landscaped garden environments [Doctoral dissertation, Bilkent Universitesi].
• Williamson, B. (2019). Computing brains: learning algorithms and neurocomputation in the smart city. The Social Power of Algorithms (pp. 81-99). Routledge.
• Xu, J., Liu, N., Polemiti, E. (2023). Effects of urban living environments on mental health in adults. Nature Medicine 29, 1456–1467. https://doi.org/10.1038/s41591-023-02365-w
• Yao, Z. (2023). A Framework Study of the Effects of Neurourbanism Perspectives on the Impact of Neighbourhood Green Space Exposure on Adolescent Mental Health. Academic Journal of Environment & Earth Science, 5(9), 1-7. https://dx.doi.org/10.25236/AJEE.2023.050901
• Ye, Q., Feng, Y., Candela, E., Escribano Macias, J., Stettler, M., & Angeloudis, P. (2021). Spatial-temporal flows-adaptive street layout control using reinforcement learning. Sustainability, 14(1), 107. https://doi.org/10.3390/su14010107
• Zhang, X., Huang, T., Gulakhmadov, A., Song, Y., Gu, X., Zeng, J., ... & Niyogi, D. (2022). Deep learning-based 500 m spatio-temporally continuous air temperature generation by fusing multi-source data. Remote Sensing, 14(15), 3536. https://doi.org/10.3390/rs14153536
• Zheng, Y., Lin, Y., Zhao, L., Wu, T., Jin, D., & Li, Y. (2023). Spatial planning of urban communities via Deep Reinforcement Learning. Nature Computational Science, 3(9), 748–762. https://doi.org/10.1038/s43588-023-00503-5