Engineering Intelligence Systems Using Big Data and Cloud Architectures for Modern Data Intensive Applications
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V1I2P105Keywords:
Big Data Engineering, Cloud Architectures, Data-Intensive Systems, Distributed Computing, Scalable Data Platforms, Cloud-Native Applications, Data Pipelines, Real-Time Analytics, High-Performance Computing, Data Engineering Frameworks, Intelligent Information Systems, Large-Scale Data Processing, Microservices Architecture, Fault-Tolerant Systems, Data Orchestration, Edge-Cloud Integration, Modern Data InfrastructureAbstract
The rapid expansion of digital ecosystems has intensified the need for scalable and adaptive Data-Intensive Application Engineering frameworks capable of handling complex, high-volume workloads. This paper introduces a combined architectural solution that is used to create Cloud-Based Intelligence Systems that is needed to support the modern enterprise and scientific scenery. The framework proposed focuses on the Large Dataset Processing Systems that are constructed based on a Parallel Data Processing mechanism to guarantee the high throughput and low execution latency of distributed infrastructures. Storage-Compute Decoupling is a fundamental design principle of the architecture, which allows the independent scaling of data storage and computational resources to be rolled out and ensure the flexibility of operations and cost efficiency. The system uses Metadata-Driven Processing to promote data governance, orchestration and lifecycle management and assure the flexibility to non-homogeneous sources of data. The architecture intelligently provisions cloud resources according to workloads requirements through the Elastic Compute Provisioning which enhances responsiveness and utilization of infrastructures. Moreover, the Infrastructure-Aware Application Design and combined Performance Engineering is incorporated at all stages of the development cycle to facilitate the Throughput Optimization and robust Fault Isolation Mechanism. Another feature that is presented by the framework is Application-Embedded Intelligence, which enables predictive scaling, adaptive workload distribution, and continuous performance monitoring. Experimental tests indicate better scalability, processing, and model accuracy and low cloud operational costs. The results can bring an organized approach to the design of smart, non-native cloud-based systems that will serve dynamically changing data-intensive applications.
References
[1] Sangaiah, A. K., Zhang, Z., & Sheng, M. (Eds.). (2018). Computational intelligence for multimedia big data on the cloud with engineering applications. Academic Press.
[2] Nachiappan, R., Javadi, B., Calheiros, R. N., & Matawie, K. M. (2017). Cloud storage reliability for big data applications: A state of the art survey. Journal of Network and Computer Applications, 97, 35-47.
[3] Yu, J. H., & Zhou, Z. M. (2019). Components and development in Big Data system: A survey. Journal of Electronic Science and Technology, 17(1), 51-72.
[4] Kothapalli, M. (2018). Cloud Computing Architectures: Comparing Service Models (IaaS, PaaS, SaaS) and Deployment Models (Public, Private, Hybrid, Community)-Uses and Trade-offs. Journal of Scientific and Engineering Research, 5(10), 334-341.
[5] Safaei, A. A. (2017). Real-time processing of streaming big data. Real-Time Systems, 53(1), 1-44.
[6] Dahiya, P., Chaitra, B., & Kumari, U. (2017). Survey on big data using Apache Hadoop and Spark. International Journal of Computer Engineering In Research Trends, 4(6), 195-201.
[7] Neves, P. C., Schmerl, B., Cámara, J., & Bernardino, J. (2016, April). Big Data in Cloud Computing: features and issues. In International Conference on Internet of Things and Big Data (Vol. 2, pp. 307-314). SCITEPRESS.
[8] Jonnalagadda, V. S., Srikanth, P., Thumati, K., Nallamala, S. H., & Dist, K. (2016). A review study of apache spark in big data processing. International Journal of Computer Science Trends and Technology (IJCST), 4(3), 93-98.
[9] Hussain, T., Sanga, A., & Mongia, S. (2019, October). Big data hadoop tools and technologies: A review. In Proceedings of International Conference on Advancements in Computing & Management (ICACM).
[10] Sriram, I., & Khajeh-Hosseini, A. (2010). Research agenda in cloud technologies. arXiv preprint arXiv:1001.3259.
[11] Hadoop, Storm, Samza, Spark, and Flink: Big Data Frameworks Compared, digitalocean, 2016. online. https://www.digitalocean.com/community/tutorials/hadoop-storm-samza-spark-and-flink-big-data-frameworks-compared
[12] Tang, S., He, B., Yu, C., Li, Y., & Li, K. (2020). A survey on spark ecosystem: Big data processing infrastructure, machine learning, and applications. IEEE Transactions on Knowledge and Data Engineering, 34(1), 71-91.
[13] Perera, S., Perera, A., & Hakimzadeh, K. (2016). Reproducible experiments for comparing apache flink and apache spark on public clouds. arXiv preprint arXiv:1610.04493.
[14] Bennett, C., Grossman, R. L., Locke, D., Seidman, J., & Vejcik, S. (2010, July). Malstone: towards a benchmark for analytics on large data clouds. In Proceedings of the 16th ACM SIGKDD International conference on Knowledge discovery and data mining (pp. 145-152).
[15] Lee, I. (2019). The Internet of Things for enterprises: An ecosystem, architecture, and IoT service business model. Internet of things, 7, 100078.
[16] Bahrami, M., & Singhal, M. (2014). The role of cloud computing architecture in big data. In Information granularity, big data, and computational intelligence (pp. 275-295). Cham: Springer International Publishing.
[17] Kibria, M. G., Nguyen, K., Villardi, G. P., Zhao, O., Ishizu, K., & Kojima, F. (2018). Big data analytics, machine learning, and artificial intelligence in next-generation wireless networks. IEEE access, 6, 32328-32338.
[18] Jones, J. W., Keating, B. A., & Porter, C. H. (2001). Approaches to modular model development. Agricultural Systems, 70(2-3), 421-443.
[19] Iqbal, R., Doctor, F., More, B., Mahmud, S., & Yousuf, U. (2020). Big Data analytics and Computational Intelligence for Cyber–Physical Systems: Recent trends and state of the art applications. Future Generation Computer Systems, 105, 766-778.
[20] Xia, Y., Chen, J., Lu, X., Wang, C., & Xu, C. (2016). Big traffic data processing framework for intelligent monitoring and recording systems. Neurocomputing, 181, 139-146.
