The Role of Intelligent Data Engineering in Enterprise Digital Transformation
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V3I4P111Keywords:
Intelligent Data Engineering, Enterprise Digital Transformation, Data Pipelines, Ai-Driven Analytics, Real-Time Processing, Cloud-Native Architecture, Data GovernanceAbstract
Enterprise digital transformation is increasingly driven by the ability to manage and exploit data at scale, speed, and reliability. As organizations adopt cloud platforms, artificial intelligence, and real-time digital services, traditional data engineering approaches often fail to meet requirements for agility, responsiveness, and operational efficiency. Intelligent Data Engineering (IDE) has emerged as a critical enabler of modern enterprise transformation by embedding automation, machine learning, and adaptive intelligence into data pipelines and architectures. This paper examines the role of intelligent data engineering in supporting enterprise digital transformation, focusing on its impact on data ingestion, processing, orchestration, governance, and analytics. IDE enhances conventional data engineering by enabling self-optimizing pipelines, automated data quality management, and intelligent orchestration capable of handling batch, streaming, and hybrid workloads. The study highlights how cloud-native and lakehouse-based architectures provide scalable foundations for intelligent data processing while supporting real-time and predictive analytics. Furthermore, the paper analyzes how IDE improves integration across enterprise systems such as ERP and CRM, enabling data-driven decision-making and operational automation. Based on industry practices and performance benchmarks reported in 2022, the findings demonstrate that intelligent data engineering delivers significant gains in latency reduction, scalability, reliability, and cost efficiency. The paper concludes that IDE is not merely a technical enhancement but a strategic capability that underpins data-centric enterprise architectures and enables sustainable, insight-driven digital transformation
References
1. Drobot, A. T. (2020, December). Industrial Transformation and the Digital Revolution: A Focus on artificial intelligence, data science and data engineering. In 2020 ITU Kaleidoscope: Industry-Driven Digital Transformation (ITU K) (pp. 1-11). IEEE.
2. Mokhtar, S., Hussin, N., Tokiran, N. S. M., Wahab, H., & Ibrahim, A. (2020). Digital transformation in information management. International Journal of Academic Research in Business and Social Sciences, 10(11), 1453–1460. https://doi.org/10.6007/IJARBSS/v10-i11/9071.
3. Ilin, I., Levina, A., Borremans, A., & Kalyazina, S. (2019). Enterprise architecture modeling in digital transformation era. In Energy management of municipal transportation facilities and transport (pp. 124-142). Cham: Springer International Publishing.
4. Jackson, P., & Carruthers, C. (2019). Data driven business transformation: How to disrupt, innovate and stay ahead of the competition. John Wiley & Sons.
5. Maheshwari, A. (2019). Digital transformation: Building intelligent enterprises. John Wiley & Sons.
6. Chan, Y., Talburt, J., & Talley, T. M. (Eds.). (2009). Data engineering: mining, information and intelligence (Vol. 132). Springer Science & Business Media.
7. Zhu, J., Gong, C., Zhang, S., Zhao, M., & Zhou, W. (2018). Foundation study on wireless big data: Concept, mining, learning and practices. China communications, 15(12), 1-15.
8. Ceci, M., Japkowicz, N., Liu, J., Papadopoulos, G. A., & Ras, Z. W. (2018). Foundations of Intelligent Systems. Springer International Publishing.
9. Korhonen, J. J., & Halén, M. (2017, July). Enterprise architecture for digital transformation. In 2017 IEEE 19th Conference on Business Informatics (CBI) (Vol. 1, pp. 349-358). IEEE.
10. Naskali, J., Kaukola, J., Matintupa, J., Ahtosalo, H., Jaakola, M., & Tuomisto, A. (2018, July). Mapping business transformation in digital landscape: A prescriptive maturity model for small enterprises. In International Conference on Well-Being in the Information Society (pp. 101-116). Cham: Springer International Publishing.
11. Krizanic, S., Sestanj-Peric, T., & Tomicic-Pupek, K. (2019, May). The changing role of ERP and CRM in digital transformation. In Economic and Social Development (Book of Proceedings), 41st International Scientific Conference on Economic and Social Development (p. 248).
12. Zimmermann, A., Schmidt, R., Sandkuhl, K., Jugel, D., Bogner, J., & Möhring, M. (2018, October). Evolution of enterprise architecture for digital transformation. In 2018 IEEE 22nd International Enterprise Distributed Object Computing Workshop (EDOCW) (pp. 87-96). IEEE.
13. Bellini, P., Nesi, P., Paolucci, M., & Zaza, I. (2018, March). Smart city architecture for data ingestion and analytics: Processes and solutions. In 2018 IEEE Fourth International Conference on Big Data Computing Service and Applications (BigDataService) (pp. 137-144). IEEE.
14. Singh, J., Cobbe, J., & Norval, C. (2018). Decision provenance: Harnessing data flow for accountable systems. IEEE Access, 7, 6562-6574.
15. Shah, D., Wang, J., & He, Q. P. (2020). Feature engineering in big data analytics for IoT-enabled smart manufacturing–Comparison between deep learning and statistical learning. Computers & Chemical Engineering, 141, 106970.
16. Turban, E. (2011). Decision support and business intelligence systems. Pearson Education India.
17. Niu, Y., Ying, L., Yang, J., Bao, M., & Sivaparthipan, C. B. (2021). Organizational business intelligence and decision making using big data analytics. Information Processing & Management, 58(6), 102725.
18. Zahid, H., Mahmood, T., & Ikram, N. (2018, December). Enhancing dependability in big data analytics enterprise pipelines. In International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage (pp. 272-281). Cham: Springer International Publishing.
19. Arul, K. (2021). Optimizing data pipelines in cloud-based big data ecosystems: A comparative study of modern ETL tools. International Journal Of Engineering And Computer Science, 10(4).
20. Panetto, H., Zdravkovic, M., Jardim-Goncalves, R., Romero, D., Cecil, J., & Mezgár, I. (2016). New perspectives for the future interoperable enterprise systems. Computers in industry, 79, 47-63.
