A Multi-Agent Generative AI Framework for Automated Data Engineering, Governance, and Analytical Optimization
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V6I4P113Keywords:
Multi-Agent Systems, Generative AI, Data Engineering, Data Governance, Analytical Optimization, Automated Data Pipelines, Reinforcement LearningAbstract
The data explosion in industries has introduced unprecedented issues in the management, governance and insights derived about huge volumes of data. Conventional data engineering culture is usually unstructured laborious processes that are subject to errors and delays. To automate the data engineering activities, mandate administrative policies and maximize the analytical processes, this paper introduces a new Multi-Agent Generative AI (MAGAI) to automate such functions. MAGAI architecture uses several dedicated AI agents that can work independently to perform such tasks as data cleaning, integrating, transforming, metadata management and automated analytics. The framework combines generative AI models and reinforcement learning strategies to streamline the process of data pipelines and decision-making. Through experimental assessment, it is proved that the efficiency of data processing have better levels and reduced errors in determining correct data and adhering to governance norms. This approach proposed has the benefit of minimizing the human factor as well as improving the quality of the information derived using complex data. We outline the promise of multi-agent AI systems in transforming enterprise data management and analytics
References
1. Jaleel, H. Q., Stephan, J. J., & Naji, S. A. (2020). Multi-Agent Systems: A Review Study. Ibn AL-Haitham Journal For Pure and Applied Sciences, 33(3), 188-214.
2. Sambath Narayanan, D. B. G. (2025). AI-Driven Data Engineering Workflows for Dynamic ETL Optimization in Cloud-Native Data Analytics Ecosystems. American International Journal of Computer Science and Technology, 7(3), 99-109. https://doi.org/10.63282/3117-5481/AIJCST-V7I3P108
3. Dorri, A., Kanhere, S. S., & Jurdak, R. (2018). Multi-agent systems: A survey. Ieee Access, 6, 28573-28593.
4. Kossek, M., & Stefanovic, M. (2024). Survey of recent results in privacy-preserving mechanisms for multi-agent systems. Journal of Intelligent & Robotic Systems, 110(3), 129.
5. Ibrahim, M., Al Khalil, Y., Amirrajab, S., Sun, C., Breeuwer, M., Pluim, J., ... & Dumontier, M. (2025). Generative AI for synthetic data across multiple medical modalities: A systematic review of recent developments and challenges. Computers in biology and medicine, 189, 109834.
6. Chen, Y., Yan, Z., & Zhu, Y. (2023). A unified framework for generative data augmentation: A comprehensive survey. arXiv preprint arXiv:2310.00277.
7. Figueira, A., & Vaz, B. (2022). Survey on synthetic data generation, evaluation methods and GANs. Mathematics, 10(15), 2733.
8. Nadal, S., Jovanovic, P., Bilalli, B., & Romero, O. (2022). Operationalizing and automating data governance. Journal of big data, 9(1), 117.
9. Pahune, S., Akhtar, Z., Mandapati, V., & Siddique, K. (2025). The Importance of AI Data Governance in Large Language Models. Big Data and Cognitive Computing, 9(6), 147.
10. Sambath Narayanan, D. B. G. (2024). Data Engineering for Responsible AI: Architecting Ethical and Transparent Analytical Pipelines. International Journal of Emerging Research in Engineering and Technology, 5(3), 97-105. https://doi.org/10.63282/3050-922X.IJERET-V5I3P110
11. Mahi, B. S. (2019). AI-Driven Metadata Management: The Future of Data Governance.
12. Vaddepalli, R. K. (2025). Smart Governance for AI: Can Metadata Automation Keep Up with Real-Time ML Pipelines?. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 6(2), 119-124.
13. Zou, H., Zhao, Q., Bariah, L., Bennis, M., & Debbah, M. (2023). Wireless multi-agent generative AI: From connected intelligence to collective intelligence. arXiv preprint arXiv:2307.02757.
14. Sambath Narayanan, D. B. G. (2025). Generative AI–Enabled Intelligent Query Optimization for Large-Scale Data Analytics Platforms. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 6(2), 153-160. https://doi.org/10.63282/3050-9262.IJAIDSML-V6I2P117
15. Athanasiadis, I. N., Milis, M., Mitkas, P. A., & Michaelides, S. C. (2009). A multi-agent system for meteorological radar data management and decision support. Environmental Modelling & Software, 24(11), 1264-1273.
16. Zhang, H. S., Zhang, Y., Xu, J. H., Xiao, D. Y., & Hu, D. C. (2003, October). Study on ITS data management based on multi-agent systems. In Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems (Vol. 1, pp. 183-187). IEEE.
17. Ladley, J. (2019). Data governance: How to design, deploy, and sustain an effective data governance program. Academic Press.
18. Omicini, A., & Mariani, S. (2013). Agents & multiagent systems: En route towards complex intelligent systems. Intelligenza Artificiale, 7(2), 153-164.
19. Himeur, Y., Elnour, M., Fadli, F., Meskin, N., Petri, I., Rezgui, Y., ... & Amira, A. (2023). AI-big data analytics for building automation and management systems: a survey, actual challenges and future perspectives. Artificial intelligence review, 56(6), 4929-5021.
20. Adeyinka, T. I., & Adeyinka, K. I. (2025). Intelligent, Autonomous, and Multi-Agents. In Humans and Generative AI Tools for Collaborative Intelligence (pp. 301-326). IGI Global Scientific Publishing.
21. Sambath Narayanan, D. B. G. (2025). Semantic Layer Construction in Data Warehouses Using GenAI for Contextualized Analytical Query Processing. American International Journal of Computer Science and Technology, 7(4), 93-102. https://doi.org/10.63282/3117-5481/AIJCST-V7I4P108
22. Yang, W., Fu, R., Bilal Amin, M., & Kang, B. (2025). Impact and influence of modern AI in metadata management. arXiv e-prints, arXiv-2501.
23. Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., ... & Zhang, X. (2024). Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680.
24. Pradeep, A., Rustamov, A., Shokirov, X., Ibragimovna, G. T., Farkhadovna, S. U., & Medetovna, A. F. (2023, August). Enhancing Data Engineering and Accelerating Learning through Intelligent Automation. In 2023 Second International Conference on Trends in Electrical, Electronics, and Computer Engineering (TEECCON) (pp. 104-110). IEEE.
