From Wait Time to Slowdown: A Hurdle Framework for Performance-Centric Metric Prediction in HPC Clusters

Abstract

The demand for High-Performance Computing (HPC) has been growing since the rise of AI in the last few years. Supercomputers are in demand more than ever across many fields for running generative models, bitcoin mining, gaming services, and other computing applications that are increasingly run in the cloud. Scheduling in HPC has always been one of the most interesting research areas, especially in the 2000s, when grid computing emerged as a promising new technology in the domain. However, research in this area has been in the shadows for a while as the grid has matured into cloud-based systems in practice. Scheduling for HPC seemed to be a saturated research area, while new research trends have taken off in all directions. The recent big leap in AI models has revived this domain for two main reasons. Many classic, heuristic, and metaheuristic scheduling algorithms can be reconceptualized by integrating them with machine learning-based algorithms or frameworks, likely leading to better performance than the classical models used previously. Additionally, the urgent need for supercomputing for running the AI models that are used widely and daily, almost for every task, urges the area of scheduling algorithms itself. This paper proposes two novel AI models. The first model is based on the hurdle framework. In contrast, the second model is based on a stacking classifier followed by a stacking regression for two real datasets, using three performance metrics: wait time, slowdown, and response time, compared with the Random Forest solo model. The Hurdle model achieved the best performance on zero-inflated targets, while the Two-Stacking Stage showed better temporal robustness. When zero inflation is absent, simpler models can be more effective.

Keywords

• High-performance computing
• Job scheduling
• Wait time prediction
• Zero-inflated data
• Hurdle model
• Stacking regression

Ethical Statement

Ethics committee approval was not required for this study because there was no study on animals or humans.

References

1. Abraham, A., Liu, H., Grosan, C., & Xhafa, F. (2008). Nature inspired meta-heuristics for grid scheduling: Single and multi-objective optimization approaches. In F. Xhafa & A. Abraham (Eds.), Metaheuristics for scheduling in distributed computing environments (pp. 247–272). Springer.
2. Bailey Lee, C., Schwartzman, Y., Hardy, J., & Snavely, A. (2004). Are user runtime estimates inherently inaccurate? In D. G. Feitelson, L. Rudolph, & U. Schwiegelshohn (Eds.), Job scheduling strategies for parallel processing (pp. 228–243). Springer.
3. Baker, N. (2024). Unlocking a new era for scientific discovery with AI: How Microsoft's AI screened over 32 million candidates to find a better battery. Microsoft Azure Quantum Blog. https://azure.microsoft.com/en-us/blog/unlocking-a-new-era-for-scientific-discovery-with-ai/
4. Bartolini, A., Borghesi, A., Bridi, T., Lombardi, M., & Milano, M. (2014). Proactive workload dispatching on the EURORA supercomputer. In B. O'Sullivan (Ed.), Principles and practice of constraint programming (pp. 744–759). Springer.
5. Berriman, G. B., Laity, A. C., Good, J. C., Katz, D. S., Jacob, J. C., Deelman, E., & Prince, T. A. (2006). Science applications of the Montage image mosaic engine. Proceedings of the International Astronomical Union, 2(S14), 621–621. https://doi.org/10.1017/S174392130600495X
6. Boëzennec, R., Dufossé, F., & Pallez, G. (2023). Optimization metrics for the evaluation of batch schedulers in HPC. In D. Klusáček, W. Cirne, & N. Desai (Eds.), Job scheduling strategies for parallel processing (pp. 3–23). Springer.
7. Boëzennec, R., Dufossé, F., & Pallez, G. (2024). Qualitatively analyzing optimization objectives in the design of HPC resource manager. ACM Transactions on Modeling and Performance Evaluation of Computing Systems, 9(4), 1–28. https://doi.org/10.1145/3675392
8. Borghesi, A., Collina, F., Lombardi, M., Milano, M., & Benini, L. (2015). Power capping in high performance computing systems. In G. Pesant (Ed.), Principles and practice of constraint programming (pp. 531–547). Springer.

9. Borghesi, A., Di Santi, C., Molan, M., Ardebili, M. S., Mauri, A., Guarrasi, M., Cavazzoni, C., & Benini, L. (2023). M100 ExaData: A data collection campaign on the CINECA’s Marconi100 Tier-0 supercomputer. Scientific Data, 10(1), Article 288. https://doi.org/10.1038/s41597-023-02174-3
10. Brevik, J., Nurmi, D., & Wolski, R. (2006). Predicting bounds on queuing delay for batch-scheduled parallel machines. In Proceedings of the Eleventh ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (pp. 110–118). ACM. https://doi.org/10.1145/1122971.1122990
11. Bridi, T., Bartolini, A., Lombardi, M., Milano, M., & Benini, L. (2016). A constraint programming scheduler for heterogeneous high-performance computing machines. IEEE Transactions on Parallel and Distributed Systems, 27(10), 2781–2794. https://doi.org/10.1109/TPDS.2016.2516990
12. Chang, C. (2018). Multiphysics magnetic fusion reactor simulator, from hot core to cold wall [Doctoral dissertation, Princeton University]. Princeton University Repository.
13. Chen, X., Zhang, H., Bai, H., Yang, C., Zhao, X., & Li, B. (2020). Runtime prediction of high-performance computing jobs based on ensemble learning. In Proceedings of the 2020 4th International Conference on High Performance Compilation, Computing and Communications (pp. 28–32). ACM. https://doi.org/10.1145/3391811.3391823
14. Dakkak, O., Arif, S., & Nor, S. A. (2015a). A critical analysis of simulators in grid. Jurnal Teknologi (Sciences & Engineering), 77(4), 111–117. https://doi.org/10.11113/jt.v77.6050
15. Dakkak, O., Arif, A. S. B. C. M., & Nor, S. A. (2015b). Resource allocation mechanisms in computational grid: A survey. ARPN Journal of Engineering and Applied Sciences, 10(15), 6662–6667
16. Dakkak, O., Fazea, Y., Nor, S. A., & Arif, S. (2021). Towards accommodating deadline driven jobs on high performance computing platforms in grid computing environment. Journal of Computational Science, 54, Article 101439. https://doi.org/10.1016/j.jocs.2021.101439
17. DeLucia, A., & Baseman, E. (2018). Work in progress: Topic modeling for HPC job state prediction. In Proceedings of the First Workshop on Machine Learning for Computing Systems (pp. 1–4). ACM.
18. Doll, F., Habbel, B., Liesch, T., Wetzel, M., Kunz, S., & Broda, S. (2026). Validation strategies for deep learning-based groundwater level time series prediction using exogenous meteorological input features. Geoscientific Model Development, 19(7), 2657–2675. https://doi.org/10.5194/gmd-19-2657-2026
19. Du, J., & Leung, J. Y.-T. (1989). Complexity of scheduling parallel task systems. SIAM Journal on Discrete Mathematics, 2(4), 473–487. https://doi.org/10.1137/0402040
20. Ensmenger, N. (2018). The environmental history of computing. Technology and Culture, 59(4), S7–S33. https://doi.org/10.1353/tech.2018.0148
21. Etinski, M., Corbalan, J., Labarta, J., & Valero, M. (2011). Linear programming based parallel job scheduling for power constrained systems. In 2011 International Conference on High Performance Computing & Simulation (pp. 389–398). IEEE. https://doi.org/10.1109/HPCSim.2011.5999847
22. Fan, Y., Lan, Z., Rich, P., Allcock, W. E., Papka, M. E., Austin, B., & Paul, D. (2019). Scheduling beyond CPUs for HPC. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing (pp. 201–212). ACM. https://doi.org/10.1145/3307681.3325409
23. Feitelson, D. G., Rudolph, L., & Schwiegelshohn, U. (2004). Parallel job scheduling—A status report. In D. G. Feitelson, L. Rudolph, & U. Schwiegelshohn (Eds.), Job scheduling strategies for parallel processing (pp. 1–16). Springer.
24. Frachtenberg, E., & Feitelson, D. G. (2005). Pitfalls in parallel job scheduling evaluation. In D. G. Feitelson & L. Rudolph (Eds.), Job scheduling strategies for parallel processing (pp. 257–282). Springer.
25. Gaikwad, B., Simakov, N. A., Furlani, T., White, J. P., & Patra, A. (2025). Predictive modeling of HPC job queue times: Improving user decision-making and resource utilization. In Practice and Experience in Advanced Research Computing 2025: The Power of Collaboration (pp. 1–4). ACM. https://doi.org/10.1145/3708579.3708595
26. Gainaru, A., Aupy, G. P., Sun, H., & Raghavan, P. (2019). Speculative scheduling for stochastic HPC applications. In Proceedings of the 48th International Conference on Parallel Processing (pp. 1–10). ACM. https://doi.org/10.1145/3337821.3337905
27. Galleguillos, C., Sîrbu, A., Kiziltan, Z., Babaoglu, O., Borghesi, A., & Bridi, T. (2017). Data-driven job dispatching in HPC systems. In G. Nicosia, G. Pardalos, & R. Umeton (Eds.), Machine learning, optimization, and big data (pp. 245–257). Springer.
28. Gaussier, E., Glesser, D., Reis, V., & Trystram, D. (2015). Improving backfilling by using machine learning to predict running times. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–10). ACM. https://doi.org/10.1145/2807591.2807641
29. Goponenko, A. V., Lamar, K., Allan, B. A., Brandt, J. M., & Dechev, D. (2024). Job scheduling for HPC clusters: Constraint programming vs. backfilling approaches. In Proceedings of the 18th ACM International Conference on Distributed and Event-based Systems (pp. 154–165). ACM. https://doi.org/10.1145/3629104.3661559
30. Goponenko, A. V., Lamar, K., Peterson, C., Allan, B. A., Brandt, J. M., & Dechev, D. (2022). Metrics for packing efficiency and fairness of HPC cluster batch job scheduling. In 2022 IEEE 34th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) (pp. 41–50). IEEE. https://doi.org/10.1109/SBAC-PAD55451.2022.00016
31. Houssein, E. H., Gad, A. G., Wazery, Y. M., & Suganthan, P. N. (2021). Task scheduling in cloud computing based on meta-heuristics: Review, taxonomy, open challenges, and future trends. Swarm and Evolutionary Computation, 62, Article 100841. https://doi.org/10.1016/j.swevo.2021.100841
32. Hovestadt, M., Kao, O., Keller, A., & Streit, A. (2003). Scheduling in HPC resource management systems: Queuing vs. planning. In D. G. Feitelson, L. Rudolph, & U. Schwiegelshohn (Eds.), Job scheduling strategies for parallel processing (pp. 1–20). Springer.
33. Jackson, D., Snell, Q., & Clement, M. (2001). Core algorithms of the Maui scheduler. In D. G. Feitelson & L. Rudolph (Eds.), Job scheduling strategies for parallel processing (pp. 87–102). Springer.
34. Jette, M. A., & Wickberg, T. (2023). Architecture of the Slurm workload manager. In D. Klusáček, W. Cirne, & N. Desai (Eds.), Job scheduling strategies for parallel processing (pp. 24–41). Springer.
35. Klusáček, D., Rudová, H., Baraglia, R., Pasquali, M., & Capannini, G. (2008). Comparison of multi-criteria scheduling techniques. In T. Priol & M. Vanneschi (Eds.), Grid computing: Achievements and prospects (pp. 173–184). Springer.
36. Klusáček, D., Tóth, Š., & Podolníková, G. (2016). Complex job scheduling simulations with Alea 4. In Proceedings of the 9th EAI International Conference on Simulation Tools and Techniques (pp. 114–121). EAI. https://doi.org/10.4108/eai.31-8-2016.2265039
37. Kolker-Hicks, E., Zhang, D., & Dai, D. (2023). A reinforcement learning based backfilling strategy for HPC batch jobs. In Proceedings of the SC'23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis (pp. 1386–1393). ACM. https://doi.org/10.1145/3624062.3624208
38. Kurth, T., Treichler, S., Romero, J., Mudigonda, M., Luehr, N., Phillips, E., Mahesh, A., Matheson, M., Deslippe, J., Fatica, M., & Prabhat. (2018). Exascale deep learning for climate analytics. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 649–660). IEEE. https://doi.org/10.1109/SC.2018.00057
39. Leung, V. J., Sabin, G., & Sadayappan, P. (2010). Parallel job scheduling policies to improve fairness: A case study. In 2010 39th International Conference on Parallel Processing Workshops (pp. 217–224). IEEE. https://doi.org/10.1109/ICPPW.2010.38
40. Mu'alem, A. W., & Feitelson, D. G. (2001). Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Transactions on Parallel and Distributed Systems, 12(6), 529–543. https://doi.org/10.1109/71.932709
41. Oh, J., & Kim, Y. (2020). Job placement using reinforcement learning in GPU virtualization environment. Cluster Computing, 23(3), 2219–2234. https://doi.org/10.1007/s10586-020-03082-x
42. Park, J.-W. (2019). Queue waiting time prediction for large-scale high-performance computing system. In 2019 International Conference on High Performance Computing & Simulation (HPCS) (pp. 111–117). IEEE. https://doi.org/10.1109/HPCSim40371.2019.00028
43. Pathak, A. R. (2025). Highlights on utilizing machine learning for high performance computing systems. Procedia Computer Science, 258, 1242–1253. https://doi.org/10.1016/j.procs.2025.07.158
44. Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., Hauenstein, S., Lahoz‐Monfort, J. J., Schröder, B., Thuiller, W., & Dormann, C. F. (2017). Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913–929. https://doi.org/10.1111/ecog.02881
45. Rudolph, L., & Smith, P. H. (2000). Valuation of ultra-scale computing systems. In D. G. Feitelson & L. Rudolph (Eds.), Job scheduling strategies for parallel processing (pp. 235–248). Springer.
46. Tanash, M., Dunn, B., Andresen, D., Hsu, W., Yang, H., & Okanlawon, A. (2019). Improving HPC system performance by predicting job resources via supervised machine learning. In Practice and Experience in Advanced Research Computing 2019: Rise of the Machines (Learning) (pp. 1–8). ACM. https://doi.org/10.1145/3332186.3333157
47. Tirmazi, M., Barker, A., Deng, N., Haque, M. E., Qin, Z. G., Hand, S., Harchol-Balter, M., & Wilkes, J. (2020). Borg: The next generation. In Proceedings of the Fifteenth European Conference on Computer Systems (pp. 1–14). ACM. https://doi.org/10.1145/3342195.3387517
48. Tsai, C.-W., & Rodrigues, J. J. (2013). Metaheuristic scheduling for cloud: A survey. IEEE Systems Journal, 8(1), 279–291. https://doi.org/10.1109/JSYST.2013.2268316
49. Verma, A., Korupolu, M., & Wilkes, J. (2014). Evaluating job packing in warehouse-scale computing. In 2014 IEEE International Conference on Cluster Computing (CLUSTER) (pp. 48–56). IEEE. https://doi.org/10.1109/CLUSTER.2014.6968735
50. Wang, L., Rodriguez, M. A., & Lipovetzky, N. (2026). MetaPilot: A DRL-based controller for dynamic adaptation to shifting scheduling objectives in HPC systems. Future Generation Computer Systems, 178, 1–15. https://doi.org/10.1016/j.future.2025.108269