HealthCraft: A Precision Model for Smart Resource Optimisation in Dynamic Big Data Healthcare Environments

Abstract

Cloud computing provides scalable computing and storage resources for big healthcare data. Efficient resource utilisation is the most critical factor in processing large-scale data in a reasonable time. Due to the complexity and heterogeneity of distributed computing frameworks, resource utilisation is often lower than expected. Moreover, predicting resource usage under real-world errors in such large and complex systems is quite challenging. In this study, we propose an online resource utilisation prediction model using machine learning (ML) methods combined with an automated log data preprocessing technique to forecast future resource consumption to automatically provision resources for big cloud-based big data systems where common errors occur, including CPU, memory, network, and data locality. Our experiments using the Hadoop framework in the cloud environment show that our ML-based models predict resource usage with a high accuracy rate in environments where different faults coincidentally occur. The model can easily locate the resource bottlenecks for inefficient resource utilisation in big data systems with high accuracy.

Keywords

• big data
• MapReduce
• Machine learning
• Resource utilisation
• Prediction

References

1. Koppad S, Gkoutos GV, Acharjee A. Cloud computing enabled big multi-omics data analytics. Bioinform Biol Insights. 2021;15.
2. Wu PY, Cheng CW, Kaddi CD, Venugopalan J, Hoffman R, Wang MD. –omic and electronic health record big data analytics for precision medicine. IEEE Trans Biomed Eng. 2016;64(2):263–73.
3. Kumar J, Singh AK. Workload prediction in cloud using artificial neural network and adaptive differential evolution. Future Gener Comput Syst. 2018;81:41–52.
4. Gui B, Wei X, Shen Q, Qi J, Guo L. Financial time series forecasting using support vector machine. In: 2014 Tenth International Conference on Computational Intelligence and Security. IEEE; 2014. p. 39–43.
5. Kim KJ, Han I. Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index. Expert Syst Appl. 2000;19(2):125–32.
6. Ullrich M, Lässig J. Current challenges and approaches for resource demand estimation in the cloud. In: 2013 International Conference on Cloud Computing and Big Data. IEEE; 2013. p. 387–94.
7. Alwasel K, Calheiros RN, Garg S, Buyya R, Pathan M, Georgakopoulos D, et al. Bigdatasdnsim: A simulator for analyzing big data applications in software-defined cloud data centers. Softw Pract Exp. 2021;51(5):893–920.
8. Jung J, Kim H. Mr-cloudsim: Designing and implementing mapreduce computing model on cloudsim. In: 2012 International Conference on ICT Convergence (ICTC). IEEE; 2012. p. 504–9.

9. Calcaterra C, Carmenini A, Marotta A, Bucci U, Cassioli D. Maxhadoop: an efficient scalable emulation tool to test sdn protocols in emulated hadoop environments. J Netw Syst Manage. 2020;28(4):1610–38.
10. Datadog [Internet]. New York: Datadog Inc.; [cited 2020 Jul 13]. Available from: https://www.datadoghq.com/
11. Apache Chukwa [Internet]. [cited 2020 Jul 14]. Available from: https://chukwa.apache.org/
12. Demirbaga U, Wen Z, Noor A, Mitra K, Alwasel K, Garg S, et al. Autodiagn: An automated real-time diagnosis framework for big data systems. IEEE Trans Comput. 2021;71(5):1035–48.
13. Zhao K, Li S, Kang Z. Takagi-sugeno fuzzy modeling and control of nonlinear system with adaptive clustering algorithms. In: 2018 10th International Conference on Modelling, Identification and Control (ICMIC). IEEE; 2018. p. 1–6.
14. Er MJ, Deng C. Obstacle avoidance of a mobile robot using hybrid learning approach. IEEE Trans Ind Electron. 2005;52(3):898–905.
15. Al-Asaly MS, Bencherif MA, Alsanad A, Hassan MM. A deep learning-based resource usage prediction model for resource provisioning in an autonomic cloud computing environment. Neural Comput Appl. 2021;1–18.
16. Royal College of Physicians. National early warning score (NEWS): standardising the assessment of acute-illness severity in the NHS. London: Report of working party; 2012.
17. Ross M, Wei W, Ohno-Machado L. “Big data” and the electronic health record. Yearb Med Inform. 2014;23(1):97–104.
18. Li S, Kang L, Zhao XM, et al. A survey on evolutionary algorithm based hybrid intelligence in bioinformatics. Biomed Res Int. 2014;2014:1–12.
19. World Health Organization. Partnering for health early warning systems [Internet]. [cited 2025 Jun 21]. Available from: https://public-old.wmo.int/en/bulletin/partnering-health-early-warning-systems
20. McGinley A, Pearse RM. A national early warning score for acutely ill patients: A new standard should help identify patients in need of critical care. BMJ. 2012;345(7869):9–9.
21. Aujla GS, Jindal A. A decoupled blockchain approach for edge-envisioned IoT-based healthcare monitoring. IEEE J Sel Areas Commun. 2021;39(2):491–9.
22. Mehta N, Pandit A. Concurrence of big data analytics and healthcare: A systematic review. Int J Med Inform. 2018;114:57–65.
23. Dean J, Ghemawat S. Mapreduce: simplified data processing on large clusters. Commun ACM. 2008;51(1):107–13.
24. Sun X, Ansari N, Wang R. Optimizing resource utilization of a data center. IEEE Commun Surv Tutor. 2016;18(4):2822–46.
25. Ikhlasse H, Benjamin D, Vincent C, Hicham M. Multimodal cloud resources utilization forecasting using a bidirectional gated recurrent unit predictor based on a power efficient stacked denoising autoencoders. Alex Eng J. 2022;61(12):11565–77.
26. Meng Y, Rao R, Zhang X, Hong P. Crupa: A container resource utilization prediction algorithm for autoscaling based on time series analysis. In: 2016 International Conference on Progress in Informatics and Computing (PIC). IEEE; 2016. p. 468–72.
27. Margara A, Urbani J, Van Harmelen F, Bal H. Streaming the web: Reasoning over dynamic data. J Web Semant. 2014;25:24–44.
28. Hameed A, Khoshkbarforoushha A, Ranjan R, Jayaraman PP, Kolodziej J, Balaji P, et al. A survey and taxonomy on energy efficient resource allocation techniques for cloud computing systems. Comput. 2016;98:751–74.
29. Armstrong B, Eigenmann R. Performance forecasting: Towards a methodology for characterizing large computational applications. In: Proceedings of 1998 International Conference on Parallel Processing (Cat. No. 98EX205). IEEE; 1998. p. 518–25.
30. Benaim M, Le Boudec JY. A class of mean field interaction models for computer and communication systems. Perform Eval. 2008;65(11–12):823–38.
31. Demirbaga U, Noor A, Wen Z, James P, Mitra K, Ranjan R. Smartmonit: Real-time big data monitoring system. In: 2019 38th Symposium on Reliable Distributed Systems (SRDS). IEEE; 2019. p. 357–72.
32. Mehmood T, Latif S, Malik S. Prediction of cloud computing resource utilization. In: 2018 15th International Conference on Smart Cities: Improving Quality of Life Using ICT & IoT (HONET-ICT). IEEE; 2018. p. 38–42.
33. Al-Asaly MS, Bencherif MA, Alsanad A, Hassan MM. A deep learning-based resource usage prediction model for resource provisioning in an autonomic cloud computing environment. Neural Comput Appl. 2021;1–18.
34. Rahmanian AA, Ghobaei-Arani M, Tofighy S. A learning automata-based ensemble resource usage prediction algorithm for cloud computing environment. Future Gener Comput Syst. 2018;79:54–71.
35. Gao S, Xiao H, Zhou E, Chen W. Robust ranking and selection with optimal computing budget allocation. Automatica. 2017;81:30–6.
36. Khan MA, Jan MA, He X. Blockchain-based edge computing frameworks for IoT applications: A comprehensive survey. IEEE Internet Things J. 2021;8(1):22–39.
37. Aujla M, Jindal R. Blockchain-based healthcare monitoring for edge computing environments: Performance evaluation and analysis. IEEE Trans Ind Inform. 2020;16(3):2204–13.
38. Nguyen DC, Pathirana PN, Ding M, Seneviratne A. BEdgeHealth: A decentralized architecture for edge-based IoMT networks using blockchain. arXiv. 2021;arXiv:2109.14295.
39. Akbari Zarkesh M, Dastani E, Safaei B, Movaghar A. EdgeLinker: Practical blockchain-based framework for healthcare fog applications to enhance security in edge-IoT data communications. arXiv. 2024;arXiv:2408.15838.
40. Cheikhrouhou O, Mershad K, Jamil F, Mahmud R, Koubaa A, Moosavi SR. A lightweight blockchain and fog-enabled secure remote patient monitoring system. arXiv. 2023;arXiv:2301.03551.