Reinforcement Learning-Based Kalman Filtering for Glucose Prediction

Öz

Accurate prediction of glucose levels in diabetes patients is critical for preventing complications such as hypoglycemia and hyperglycemia. In recent years, the implementation of continuous glucose monitoring (CGM) systems has enabled the development of prediction models. Among these models, Kalman filtering (KF) and its variants, the extended Kalman filter (EKF) and the unscented Kalman filter (UKF), have been widely applied for modeling both linear and nonlinear systems. However, these filtering models depend on fixed parameters, which limits their adaptability to changing physiological conditions and reduces their performance for long-term prediction. Recent advancements in machine learning enable continuous dynamic adaptation to changing conditions, providing an effective solution to these limitations. In particular, Q-Learning (QL), a reinforcement learning algorithm, can dynamically update model parameters based on environmental feedback, thereby enabling more accurate and personalized glucose predictions. This study investigates the glucose prediction performance of hybrid models that integrate KF, EKF, and UKF with QL algorithm. Experimental evaluations were conducted on the OhioT1DM dataset, using various parameter configurations across multiple prediction horizons ranging from 5 to 90 minutes. Results demonstrate that the standard KF provides high accuracy for short-term predictions, while the UKF shows superior performance for long-term prediction.

Anahtar Kelimeler

• Kalman Filter
• Q-Learning
• Reinforcement Learning
• Glucose Prediction
• Diabetes Management
• Continuous Glucose Monitoring

Glikoz Tahmini için Pekiştirmeli Öğrenme Tabanlı Kalman Filtre

Öz

Diyabet hastalarında glikoz seviyelerinin doğru bir şekilde tahmin edilmesi, hipoglisemi ve hiperglisemi gibi komplikasyonların önlenmesi için çok önemlidir. Son yıllarda, sürekli glikoz izleme (CGM) sistemlerinin uygulanması, tahmin modellerinin geliştirilmesini mümkün kılmıştır. Bu modeller arasında Kalman filtreleme (KF) ve onun varyantları olan genişletilmiş Kalman filtresi (EKF) ve kokusuz Kalman filtresi (UKF), hem doğrusal hem de doğrusal olmayan sistemlerin modellenmesinde yaygın olarak kullanılmaktadır. Ancak, bu filtreleme modelleri sabit parametrelere bağlıdır, bu da değişen fizyolojik koşullara uyum sağlama yeteneklerini sınırlar ve uzun vadeli tahminlerde performanslarını düşürür. Makine öğrenimindeki son gelişmeler, değişen koşullara sürekli dinamik uyum sağlamayı mümkün kılarak bu sınırlamalara etkili bir çözüm sunmaktadır. Özellikle, pekiştirmeli öğrenme algoritması olan Q-Öğrenme (QL), çevresel geri bildirimlere dayalı olarak model parametrelerini dinamik olarak güncelleyerek daha doğru ve kişiselleştirilmiş glikoz tahminleri sağlar. Bu çalışma, KF, EKF ve UKF'yi QL algoritmasıyla entegre eden hibrit modellerin glikoz tahmin performansını incelemektedir. Deney değerlendirmeleri, 5 ila 90 dakika arasında değişen çoklu tahmin ufuklarında çeşitli parametre yapılandırmaları kullanılarak OhioT1DM veri seti üzerinde gerçekleştirilmiştir. Sonuçlar, standart KF'nin kısa vadeli tahminlerde yüksek doğruluk sağlarken, UKF'nin uzun vadeli tahminlerde üstün performans gösterdiğini ortaya koymaktadır.

Anahtar Kelimeler

• Kalman Filtre
• Q-Öğrenme
• Pekiştirmeli Öğrenme
• Glikoz Tahmini
• Diyabet Yönetimi
• Sürekli Glikoz İzleme

Kaynakça

1. D. Control, C. T. E. o. D. Interventions, and C. S. R. Group, "Intensive diabetes treatment and cardiovascular disease in patients with type 1 diabetes," New England Journal of Medicine, vol. 353, no. 25, pp. 2643-2653, 2005.
2. A. Katsarou et al., "Type 1 diabetes mellitus," Nature reviews Disease primers, vol. 3, no. 1, pp. 1-17, 2017.
3. V. Kılıç, "Deep gated recurrent unit for smartphone-based image captioning," Sakarya University Journal of Computer and Information Sciences, vol. 4, no. 2, pp. 181-191, 2021.
4. Z. Palaz, V. Doğan, and V. Kılıç, "Smartphone-based multi-parametric glucose prediction using recurrent neural networks," Avrupa Bilim ve Teknoloji Dergisi, no. 32, pp. 1168-1174, 2021.
5. M. Şen et al., "Colorimetric detection of H2O2 with Fe3O4@ Chi nanozyme modified µPADs using artificial intelligence," Microchimica Acta, vol. 189, no. 10, p. 373, 2022.
6. R. W. Beck et al., "Effect of continuous glucose monitoring on glycemic control in adults with type 1 diabetes using insulin injections: the DIAMOND randomized clinical trial," Jama, vol. 317, no. 4, pp. 371-378, 2017.
7. E. J. Knobbe and B. Buckingham, "The extended Kalman filter for continuous glucose monitoring," Diabetes technology & therapeutics, vol. 7, no. 1, pp. 15-27, 2005.
8. E. Matsinos, "The Kalman Filter: a didactical overview," arXiv preprint arXiv:1607.05590, 2016.

9. G. F. Welch, "Kalman filter," in Computer vision: a reference guide: Springer, 2021, pp. 721-723.
10. B. Fetiler, Ö. Çaylı, and V. Kılıç, "Leveraging Pre-trained 3D-CNNs for Video Captioning," European Journal of Science and Technology, no. 53, pp. 58-63, 2024.
11. B. Fetiler, Ö. Çaylı, Ö. T. Moral, V. Kılıç, and A. Onan, "Video captioning based on multi-layer gated recurrent unit for smartphones," Avrupa Bilim ve Teknoloji Dergisi, no. 32, pp. 221-226, 2021.
12. Ö. A. Koca and V. Kılıç, "Trend-weighted multi-resolution transformer for multi-parametric glucose prediction," Biomedical Signal Processing and Control, vol. 113, p. 108885, 2026.
13. V. Doǧan, T. Isık, V. Kılıç, and N. Horzum, "A field-deployable water quality monitoring with machine learning-based smartphone colorimetry," Analytical Methods, vol. 14, no. 35, pp. 3458-3466, 2022.
14. Ö. B. Mercan, V. Doğan, and V. Kılıç, "Time Series Analysis based Machine Learning Classification for Blood Sugar Levels," in 2020 Medical Technologies Congress (TIPTEKNO), 2020: IEEE, pp. 1-4.
15. A. Y. Mutlu and V. Kılıç, "Machine learning based smartphone spectrometer for harmful dyes detection in water," in 2018 26th Signal Processing and Communications Applications Conference (SIU), 2018: IEEE, pp. 1-4.
16. F. AlMahamid and K. Grolinger, "Reinforcement learning algorithms: An overview and classification," in 2021 IEEE canadian conference on electrical and computer engineering (CCECE), 2021: IEEE, pp. 1-7.
17. C. J. Watkins and P. Dayan, "Q-learning," Machine learning, vol. 8, pp. 279-292, 1992.
18. K. Tang, X. Luan, F. Ding, and F. Liu, "Q‐learning based adaptive Kalman filtering for partial model‐free dynamic systems," International Journal of Adaptive Control and Signal Processing, vol. 38, no. 3, pp. 954-967, 2024.
19. P. Yadav, J. Piri, A. Sengupta, and M. Sengupta, "Q‐Learning‐Based Adaptive Student's t t Maximum Correntropy Cubature Kalman Filter for Non‐Gaussian Noise With Unknown Noise Covariances," International Journal of Adaptive Control and Signal Processing, 2025.
20. O. S. Al-Heety et al., "Traffic Control Based on Integrated Kalman Filtering and Adaptive Quantized Q-Learning Framework for Internet of Vehicles," CMES-Computer Modeling in Engineering & Sciences, vol. 138, no. 3, 2024.
21. C. Marling and R. Bunescu, "The OhioT1DM dataset for blood glucose level prediction," in CEUR workshop proceedings, 2020, vol. 2675, p. 71.
22. M. Halvorsen, K. D. Benam, H. Khoshamadi, and A. L. Fougner, "Blood glucose level prediction using subcutaneous sensors for in vivo study: Compensation for measurement method slow dynamics using kalman filter approach," in 2022 IEEE 61st conference on decision and control (CDC), 2022: IEEE, pp. 6034-6039.
23. Q. Li, R. Li, K. Ji, and W. Dai, "Kalman filter and its application," in 2015 8th international conference on intelligent networks and intelligent systems (ICINIS), 2015: IEEE, pp. 74-77.
24. M. I. Ribeiro, "Kalman and extended kalman filters: Concept, derivation and properties," Institute for Systems and Robotics, vol. 43, no. 46, pp. 3736-3741, 2004.
25. E. A. Wan and R. Van Der Merwe, "The unscented Kalman filter," Kalman filtering and neural networks, pp. 221-280, 2001.
26. H. V. Dudukcu, M. Taskiran, and T. Yildirim, "Blood glucose prediction with deep neural networks using weighted decision level fusion," Biocybernetics and Biomedical Engineering, vol. 41, no. 3, pp. 1208-1223, 2021.
27. K. Li, C. Liu, T. Zhu, P. Herrero, and P. Georgiou, "GluNet: A deep learning framework for accurate glucose forecasting," IEEE journal of biomedical and health informatics, vol. 24, no. 2, pp. 414-423, 2019.
28. T. Yang, X. Yu, N. Ma, R. Wu, and H. Li, "An autonomous channel deep learning framework for blood glucose prediction," Applied Soft Computing, vol. 120, p. 108636, 2022.
29. J. Jeon, P. J. Leimbigler, G. Baruah, M. H. Li, Y. Fossat, and A. J. Whitehead, "Predicting glycaemia in type 1 diabetes patients: experiments in feature engineering and data imputation," Journal of healthcare informatics research, vol. 4, no. 1, pp. 71-90, 2020.
30. J. Chen, K. Li, P. Herrero, T. Zhu, and P. Georgiou, "Dilated Recurrent Neural Network for Short-time Prediction of Glucose Concentration," in KDH@ IJCAI, 2018, pp. 69-73.
31. P. Domanski, A. Ray, F. Firouzi, K. Lafata, K. Chakrabarty, and D. Pflüger, "Blood glucose prediction for type-1 diabetics using deep reinforcement learning," in IEEE International Conference on Digital Health (ICDH), 2023: IEEE, pp. 339-347.
32. M. M. H. Shuvo and S. K. Islam, "Deep multitask learning by stacked long short-term memory for predicting personalized blood glucose concentration," IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 3, pp. 1612-1623, 2023.