Leveraging Machine Learning to Predict Future Storage and Compute Needs Based on Usage Trends
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V3I2P109Keywords:
Machine Learning, Predictive Analytics, Storage Management, Compute Resource Allocation, Usage Trends, Cloud ComputingAbstract
As the usage of data-driven applications and cloud infrastructures rapidly expands, organizations are more and more challenged to achieve the management of the storage and compute resources efficiently. Future resource requirements have become imperative in order to streamline the cost, performance as well as to prevent bottlenecks. This paper will provide a detailed discussion on how machine learning (ML) methods can be used to forecast future storage and computation needs by using a history of usage. The proposal will combine data gathering, premodelling and complex predictive modeling to deliver viable data to infrastructure planning. Several ML algorithms are tested on the actual data of vendors of cloud services and internal IT systems, such as regression models, time-series prediction, and ensembles. The standard measures are used to compare the performance of these models in terms of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and R-Sqr (R 2 ) values. Also, in this paper, we will discuss how feature selection, model tuning and analysis of seasonal trends affect the accuracy of prediction. The findings indicate that the prediction systems that employ MLs can be useful in a variety of ways to boost the resource allocation plans, decrease the operational expenses, and increase the quality of the service. In addition, it has been suggested to use a visual dashboard to monitor estimates and real usage of resources to enable the IT managers to make informed decisions in near real time. The results underscore the relevance of using both the information-driven knowledge, as well as the ML-based methods to ensure the proactive management of storage and computational resources in order to be able to furnish the infrastructural facilities that can handle the changing patterns of use in a short period of time
References
1. Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018). Statistical and Machine Learning forecasting methods: Concerns and ways forward. PloS one, 13(3), e0194889.
2. Jannapureddy, R., Vien, Q. T., Shah, P., & Trestian, R. (2019). An auto-scaling framework for analyzing big data in the cloud environment. Applied Sciences, 9(7), 1417.
3. Hyndman, R. J., & Athanasopoulos, G. (2018). Forecasting: principles and practice. OTexts.
4. Sarker, I. H. (2021). Data science and analytics: an overview from data-driven smart computing, decision-making and applications perspective. SN Computer Science, 2(5), 377.
5. Khallouli, W., & Huang, J. (2022). Cluster resource scheduling in cloud computing: literature review and research challenges. The Journal of supercomputing, 78(5), 6898-6943.
6. Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32.
7. Chen, T., & Guestrin, C. (2016, August). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining (pp. 785-794).
8. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
9. Zhan, Z. H., Liu, X. F., Gong, Y. J., Zhang, J., Chung, H. S. H., & Li, Y. (2015). Cloud computing resource scheduling and a survey of its evolutionary approaches. ACM Computing Surveys (CSUR), 47(4), 1-33.
10. Qolomany, B., Al-Fuqaha, A., Gupta, A., Benhaddou, D., Alwajidi, S., Qadir, J., & Fong, A. C. (2019). Leveraging machine learning and big data for smart buildings: A comprehensive survey. IEEE access, 7, 90316-90356.
11. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine learning, 20(3), 273-297.
12. Akgun, I. U., Aydin, A. S., Shaikh, A., Velikov, L., & Zadok, E. (2021, July). A machine learning framework to improve storage system performance. In Proceedings of the 13th ACM Workshop on Hot Topics in Storage and File Systems (pp. 94-102).
13. Dalal, A., Abdul, S., Mahjabeen, F., & Kothamali, P. R. (2019). Leveraging Artificial Intelligence and Machine Learning for Enhanced Application Security. Available at SSRN 5403818.
14. Terry, N., & Palmer, J. (2016). Trends in home computing and energy demand. Building Research & Information, 44(2), 175-187.
15. Fitz-Enz, J., & John Mattox, I. I. (2014). Predictive analytics for human resources. John Wiley & Sons.
16. Nijjer, S., & Raj, S. (2020). Predictive analytics in human resource management: a hands-on approach. Routledge India.
17. Matsunaga, A., & Fortes, J. A. (2010, May). On the use of machine learning to predict the time and resources consumed by applications. In 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (pp. 495-504). IEEE.
18. Verma, S., & Bala, A. (2021). Auto-scaling techniques for IoT-based cloud applications: a review. Cluster Computing, 24(3), 2425-2459.
19. Tan, J., Dube, P., Meng, X., & Zhang, L. (2011, June). Exploiting resource usage patterns for better utilization prediction. In 2011 31st International Conference on Distributed Computing Systems Workshops (pp. 14-19). IEEE.
20. Muccini, H., & Vaidhyanathan, K. (2021, May). Software architecture for ml-based systems: What exists and what lies ahead. In 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN) (pp. 121-128). IEEE.
