Effort Estimation for AI-Enabled Software Systems

Abstract

The widespread adoption of AI-enabled software in recent years has introduced new challenges in estimating software development efforts. In this exploratory study, we investigate potential solutions to these challenges, with a particular focus on whether functional size—a robust effort estimator in traditional software development—remains valid in AI-enabled software. To this end, an initial dataset comprising 39 AI-enabled software projects has been curated from both the software industry and academic institutions. Effort estimation is approached as a supervised regression problem, utilizing 20 distinct algorithms. To assess the significance of features—including functional size measured in COSMIC Function Points (CFP)—feature aggregation and selection techniques have been applied. The top-performing model was KNeighbors Regressor, while linear regression variants demonstrated limited success. This outcome suggests that the relationship between the target variable and predictor features is not adequately characterized by linearity. Both dataset size and functional size related effort factors emerged among the most important features. While the relatively small dataset size limits broad generalizability, the results of this preliminary investigation provide promising early evidence that CFP may remain a robust indicator for effort estimation, even in scenarios where the software is AI-based. Nevertheless, AI-related factors such as feature engineering also exhibit a significant influence on the effort estimation process, highlighting the need for future large-scale empirical validations.

Keywords

• AI-Enabled Software
• Artificial Intelligence
• COSMIC Function Point
• Effort Estimation
• Machine Learning

Project Number

7210488

Ethical Statement

This study does not involve any human or animal participants and does not require ethical committee approval. It is declared that during the preparation process of this study, scientific and ethical principles were followed, and all the studies benefited from are stated in the bibliography.

Thanks

This research was supported by the Scientific and Technological Research Council of Turkey (TUBITAK). The project number is TEYDEB 7210488. We extend our deepest gratitude to the invaluable contribution and support provided by TUBITAK throughout the course of this research endeavor.

References

1. E. A. Nelson, Management Handbook for the Estimation of Computer Programming Costs. Santa Monica, CA, USA: System Development Corp., 1967.
2. D. Galin, Software Quality Assurance: From Theory to Implementation. Harlow, U.K.: Pearson Education, 2004.
3. V. Garousi, A. Coşkunçay, A. Betin-Can, and O. Demirörs, “A survey of software engineering practices in Turkey (extended version),” arXiv preprint arXiv:1412.4648, 2014.
4. T. Hacaloglu and O. Demirors, “Measureability of functional size in Agile software projects: Multiple case studies with COSMIC FSM,” in Proc. 2019 45th Euromicro Conf. Softw. Eng. Adv. Appl. (SEAA), 2019, pp. 204–211.
5. C. Commeyne, A. Abran, and R. Djouab, “Effort estimation with story points and COSMIC function points: An industry case study,” Softw. Meas. News, vol. 21, no. 1, pp. 25–36, 2016.
6. G. Giray, “A software engineering perspective on engineering machine learning systems: State of the art and challenges,” J. Syst. Softw., vol. 180, Art. no. 111031, 2021.
7. A. Lesterhuis and A. Abran, “COSMIC sizing of machine learning image classifier software using neural networks,” in Proc. Int. Workshop Softw. Meas. Int. Conf. Softw. Process Product Meas. (IWSM-Mensura), vol. 2476, 2019, pp. 121–129.
8. S. Vedadi Moghaddam, “Investigation of the COSMIC sizing of real-time embedded and AI software for their usage within a priori and a posteriori contexts for estimation purposes in industry,” Ph.D. dissertation, École de technologie supérieure, Montreal, QC, Canada, 2020.

9. T. Fehlmann and E. Kranich, “Testing artificial intelligence by customers’ needs,” Athens J. Sci., vol. 6, no. 4, pp. 265–286, 2019.
10. K. Ahmad, M. Abdelrazek, C. Arora, M. Bano, and J. Grundy, “Requirements engineering for artificial intelligence systems: A systematic mapping study,” Inf. Softw. Technol., vol. 158, Art. no. 107176, 2023.
11. L. E. Holmquist, “Intelligence on tap: Artificial intelligence as a new design material,” Interactions, vol. 24, no. 4, pp. 28–33, 2017.
12. E. Duman and M. H. Ozcelik, “Detecting credit card fraud by genetic algorithm and scatter search,” Expert Syst. Appl., vol. 38, no. 10, pp. 13057–13063, 2011.
13. Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development practices?” IEEE Trans. Softw. Eng., vol. 47, no. 9, pp. 1857–1871, 2019.
14. M. Shaw and L. Zhu, “Can software engineering harness the benefits of advanced AI?” IEEE Softw., vol. 39, no. 6, pp. 99–104, 2022.
15. I. Ozkaya, “What is really different in engineering AI-enabled systems?” IEEE Softw., vol. 37, no. 4, pp. 3–6, 2020.
16. S. S. Gautam and V. Singh, “The state-of-the-art in software development effort estimation,” J. Softw. Evol. Process, vol. 30, no. 12, Art. no. e1983, 2018.
17. Common Software Measurement International Consortium, “Non-functional & project requirements with COSMIC: Experts guide,” COSMIC Organization. [Online]. Available: https://cosmic-sizing.org
18. Common Software Measurement International Consortium, “Guideline for sizing data warehouse and big data software,” COSMIC Organization, 2018. [Online]. Available: https://cosmic-sizing.org
19. Common Software Measurement International Consortium, “Non-functional requirements and COSMIC sizing practitioner’s guide,” COSMIC Organization, 2020.
20. Common Software Measurement International Consortium, “Early software sizing with COSMIC: Expert’s guide,” COSMIC Organization, 2020.
21. Common Software Measurement International Consortium, “Early software sizing with COSMIC: Practitioners guide,” COSMIC Organization, 2020.
22. A. Sellami et al., “Case study: Sizing natural language/UML requirements for web and mobile applications using COSMIC FSM,” 2016.
23. A. Sellami et al., “The COSMIC functional size measurement method version 4.0.2,” 2019.
24. F. Vogelezang, “Guideline for early or rapid COSMIC functional size measurement: By using approximation approaches,” 2015.
25. S. Bagriyanik and A. Karahoca, “Automated COSMIC function point measurement using a requirements engineering ontology,” Inf. Softw. Technol., vol. 72, pp. 189–203, 2016.
26. C. Jones, Estimating Software Costs: Bringing Realism to Estimating. New York, NY, USA: McGraw-Hill, 2007.
27. S. McConnell, Software Estimation: Demystifying the Black Art. Redmond, WA, USA: Microsoft Press, 2006.
28. S. Bağrıyanık, A. Karahoca, and E. Ersoy, “Selection of a functional sizing methodology: A telecommunications company case study,” Glob. J. Technol., vol. 7, pp. 98–108, 2015.
29. A. Deveci, S. Caliskan, S. Bagriyanik, O. Gazdağı, and A. Karahoca, “Yazılım geliştirme hizmetlerinin satın alınması için bir maliyet modeli önerisi,” in Proc. CEUR Workshop, Kadir Has Univ., Istanbul, Turkey, 2018.
30. K. Öztürk et al., “Yazılım geliştirme hizmetlerinin satın alma yönetiminde COSMIC işlev puan kullanımı,” in Proc. Turkish Nat. Softw. Eng. Symp., 2017.
31. M. Salmanoğlu, K. Öztürk, S. Bağrıyanık, E. Ungan, and O. Demirörs, “Benefits and challenges of measuring software size: Early results in a large organization,” in Proc. 25th Int. Workshop Softw. Meas. 10th Int. Conf. Softw. Process Product Meas. (IWSM-Mensura), 2015.
32. T. Hacaloğlu and O. Demirörs, “Challenges of using software size in agile software development: A systematic literature review,” in Proc. CEUR Workshop, 2018.
33. M. Dashti and T. J. Gandomani, “A taxonomy of approaches and methods for software effort estimation,” in Innovations in Computer Science and Engineering: Proc. Ninth ICICSE, 2021. Singapore: Springer, 2022, pp. 97–105.
34. A. Jadhav, M. Kaur, and F. Akter, “Evolution of software development effort and cost estimation techniques: Five decades study using automated text mining approach,” Math. Probl. Eng., vol. 2022, pp. 1–17, 2022.
35. Y. Mahmood, N. Kama, A. Azmi, and M. Ali, “Improving estimation accuracy prediction of software development effort: A proposed ensemble model,” in Proc. 2020 Int. Conf. Electr., Commun., Comput. Eng. (ICECCE), 2020, pp. 1–6.
36. A. Trendowicz and R. Jeffery, Software Project Effort Estimation: Foundations and Best Practices for Success. Cham, Switzerland: Springer, 2014, pp. 277–293.
37. T. Vera, S. F. Ochoa, and D. Perovich, “Survey of software development effort estimation taxonomies,” Dept. Comput. Sci., Univ. Chile, Santiago, Chile, 2017.
38. A. B. Nassif, M. Azzeh, A. Idri, and A. Abran, “Software development effort estimation using regression fuzzy models,” Comput. Intell. Neurosci., vol. 2019, 2019.
39. A. Setiadi, W. F. Hidayat, A. Sinnun, A. Setiawan, M. Faisal, and D. P. Alamsyah, “Analyze the datasets of software effort estimation with particle swarm optimization,” in Proc. 2021 Int. Semin. Intell. Technol. Its Appl. (ISITIA), 2021, pp. 197–201.
40. N. Sreekanth et al., “Evaluation of estimation in software development using deep learning-modified neural network,” Appl. Nanosci., vol. 13, no. 3, pp. 2405–2417, 2023.
41. M. Hammad and A. Alqaddoumi, “Features-level software effort estimation using machine learning algorithms,” in Proc. 2018 Int. Conf. Innov. Intell. Informat., Comput., Technol. (3ICT), 2018, pp. 1–3.
42. S. Hameed, Y. Elsheikh, and M. Azzeh, “An optimized case-based software project effort estimation using genetic algorithm,” Inf. Softw. Technol., vol. 153, Art. no. 107088, 2023.
43. M. Arora, A. Sharma, S. Katoch, M. Malviya, and S. Chopra, “A state of the art regressor model’s comparison for effort estimation of agile software,” in Proc. 2021 2nd Int. Conf. Intell. Eng. Manag. (ICIEM), 2021, pp. 211–215.
44. H. Mustapha and N. Abdelwahed, “Investigating the use of random forest in software effort estimation,” Procedia Comput. Sci., vol. 148, pp. 343–352, 2019.
45. E. I. Mustafa and R. Osman, “A random forest model for early-stage software effort estimation for the SEERA dataset,” Inf. Softw. Technol., Art. no. 107413, 2024.
46. M. Rahman, T. Goncalves, and H. Sarwar, “Review of existing datasets used for software effort estimation,” Int. J. Adv. Comput. Sci. Appl., vol. 14, no. 7, 2023.
47. P. Sharma and J. Singh, “Systematic literature review on software effort estimation using machine learning approaches,” in Proc. 2017 Int. Conf. Next Gener. Comput. Inf. Syst. (ICNGCIS), 2017, pp. 43–47.
48. M. H. Ozcelik and S. Bagriyanik, “Cost estimation and sizing of AI-enabled software systems,” in Proc. 41st Nat. Oper. Res. Ind. Eng. Congr., Pamukkale Univ., Denizli, Turkey, 2022.
49. S. Samoili, M. L. Cobo, E. Gómez, G. De Prato, F. Martínez-Plumed, and B. Delipetrev, “AI Watch. Defining artificial intelligence. Towards an operational definition and taxonomy of artificial intelligence,” Eur. Comm., Brussels, Belgium, 2020.
50. P. Chapman et al., CRISP-DM 1.0: Step-by-Step Data Mining Guide, 2000.
51. The COSMIC Functional Size Measurement Method Measurement Manual (The COSMIC Implementation Guide for ISO/IEC 19761:2017), Version 4.0.2, 2017.
52. O. Turetken, O. Ozcan Top, B. Ozkan, and O. Demirors, “The impact of individual assumptions on functional size measurement,” in Proc. Int. Conf. Softw. Process Product Meas., Berlin, Germany: Springer, 2008, pp. 155–169.
53. F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
54. D. Port and M. Korte, “Comparative studies of the model evaluation criteria MMRE and PRED in software cost estimation research,” in Proc. 2nd ACM-IEEE Int. Symp. Empir. Softw. Eng. Meas., 2008, pp. 51–60.
55. L. L. Minku and X. Yao, “Software effort estimation as a multiobjective learning problem,” ACM Trans. Softw. Eng. Methodol., vol. 22, no. 4, pp. 1–32, 2013.