A Cloud-Native Reference Architecture for Data Engineering, Generative AI, and Decision Intelligence Using AWS and Amazon Bedrock
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V5I1P122Keywords:
Cloud-Native Architecture, Data Engineering, Generative AI, Decision Intelligence, Aws, Amazon Bedrock, Machine Learning, Data Lake, Enterprise AnalyticsAbstract
As data in enterprises is growing at an accelerated pace and with the advancement of artificial intelligence, how the organizations are obtaining business intelligence and operational insights has undergone a significant change. Today's companies are increasingly relying on scalable, secure, intelligent, cloud-native architectures that can support high-volume data engineering, generative artificial intelligence (GenAI), and real-time decision intelligence. However, traditional data platforms are not designed to handle data silos, low latency, complex infrastructure, or integration with AI, and these are some of the challenges they face in today's business context. Solutions that leverage elastic infrastructure and elastic services are a good way to overcome these challenges. This paper suggests a cloud-native reference architecture for a data engineering pipeline, AI systems, and decision intelligence integration system with Amazon Web Services (AWS) and Amazon Bedrock. The architecture utilizes AWS services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon EMR, Amazon Kinesis, AWS Lambda, Amazon SageMaker, and Amazon Bedrock to create a scalable intelligence ecosystem for enterprise. The proposed architecture provides the possibility of collecting, processing, transforming and analyzing structured and unstructured data, as well as integrating foundation models for intelligent automation and augmentation of decisions. The architecture is built around four key layers: data ingestion, data processing and storage, AI intelligence and delivery of decision intelligence. The data ingestion layer can ingest data using AWS-native services, such as batch and streaming workloads. The processing layer provides scalable ETL, ELT and data transformation pipelines. The AI intelligence layer includes AI and generative AI features such as natural language analytics, intelligent summarization, anomaly detection, predictive analytics, and recommendation systems. The decision intelligence layer transforms the results of the analysis into meaningful business insights via dashboards, automated workflows, and decision systems supported by artificial intelligence. The research compares the architecture based on the following performance metrics: Pipeline latency, Data processing throughput, Model inference speed, Query response time, Business decision acceleration. The experimental results have shown the substantial enhancements of the operational efficiency, data processing speed and decision quality over the conventional architectures. Results show that AI augmentation can lead to lower processing latency, responsiveness in analytics, and better AI-driven decision support capabilities. This research proposes a practical architecture that integrates modern data engineering with generative AI and intelligent decision-making, creating a seamless and enterprise-ready solution. It offers scalable infrastructure, a secure environment for AI deployment, and high-performance analytics, making it an ideal choice for digital transformation projects. It is a reference model for enterprises looking to modernize their data platforms and implement an AI-driven decision ecosystem with AWS cloud technologies.
References
1. Jamshidi, P., Pahl, C., & Mendonça, N. C. (2017). Pattern‐based multi‐cloud architecture migration. Software: Practice and Experience, 47(9), 1159-1184.
2. Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, omega, and kubernetes. Communications of the ACM, 59(5), 50-57.
3. Blinowski, G., Ojdowska, A., & Przybyłek, A. (2022). Monolithic vs. microservice architecture: A performance and scalability evaluation. IEEE access, 10, 20357-20374.
4. Dean, J., & Ghemawat, S. (2008). MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1), 107-113.
5. White, T. (2012). Hadoop: The definitive guide. " O'Reilly Media, Inc.".
6. Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., ... & Stoica, I. (2016). Apache spark: a unified engine for big data processing. Communications of the ACM, 59(11), 56-65.
7. Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016). Activeclean: Interactive data cleaning for statistical modeling. Proc. VLDB Endow., 9(12), 948-959.
8. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., ... & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in neural information processing systems, 28.
9. Gandomi, A., & Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics. International journal of information management, 35(2), 137-144.
10. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.
11. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).
12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
13. Halevy, A., Norvig, P., & Pereira, F. (2009). The unreasonable effectiveness of data. IEEE intelligent systems, 24(2), 8-12.
14. Balalaie, A., Heydarnoori, A., & Jamshidi, P. (2016, May). Microservices architecture enables DevOps: An experience report on migration to a cloud-native architecture. In 2016 IEEE Software Architecture Workshops (ICSAW) (pp. 201–208). IEEE. https://doi.org/10.1109/ICSAW.2016.31
15. Cases, B. U., & Figueiredo, M. (2023). Generative AI with SAP and Amazon Bedrock. SAP Technical Documentation.
16. Kar, A. K., Varsha, P. S., & Rajan, S. (2023). Unravelling the impact of generative artificial intelligence (GAI) in industrial applications: A review of scientific and grey literature. Global Journal of Flexible Systems Management, 24(4), 659-689.
17. Zohuri, B., & Moghaddam, M. (2020). From business intelligence to artificial intelligence. Journal of Material Sciences & Manufacturing Research, 1(1), 1-10.
18. Varghese, B., & Buyya, R. (2018). Next generation cloud computing: New trends and research directions. Future generation computer systems, 79, 849-861.
19. Lakarasu, P. (2022). AI-Driven Data Engineering: Automating Data Quality, Lineage, And Transformation In Cloud-Scale Platforms. Lineage, and Transformation in Cloud-scale Platforms (December 10, 2022).
20. Beheshti, A., Yang, J., Sheng, Q. Z., Benatallah, B., Casati, F., Dustdar, S., ... & Xue, S. (2023, July). ProcessGPT: transforming business process management with generative artificial intelligence. In 2023 IEEE international conference on web services (ICWS) (pp. 731-739). IEEE.
21. Zhang, H., Chen, G., Ooi, B. C., Tan, K. L., & Zhang, M. (2015). In-memory big data management and processing: A survey. IEEE Transactions on Knowledge and Data Engineering, 27(7), 1920-1948.
