A Scalable Full-Stack .NET Enterprise Application Framework for Data Integration with Master Data Management Systems
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V3I2P115Keywords:
Data Integration, Master Data Management (MDM), .NET Framework, Microservices Architecture, Enterprise Applications, Data Governance, Distributed Systems, Cloud ComputingAbstract
At the digital transformation age, companies are seeking strong data integration platforms to promote the consistency, quality, and administration of vital business information. Master Data Management (MDM) systems are the foundation of having one source of truth within heterogeneous systems of data. Nevertheless, the incorporation of various sources of data into MDM systems at a scale, reliability, and performance is a major challenge. In this paper, a proposal will be offered of a scalable full stack enterprise application stack targeting a perfect data integration with MDM systems. The construct utilizes the current architectural paradigms such as microservices, event-driven architecture, and cloud-native concepts to enable high-data processing and low-latency data processing. The framework proposed is layered architecture with clearly separated concerns of interests, which provides maintainability and extensibility. It uses ASP.NET Core as the backend service, Angular/Blazor as the frontend interface, and Entity Framework Core in addition to distributed messaging systems based on Kafka and Azure Service Bus asynchronous communication. The system uses RESTful APIs and GraphQL interfaces to achieve interoperability and clusterization deployment with a variety of Docker and orchestration by Kubernetes. Also, rule-based engines are used to implement data validation, deduplication, and golden record management in the MDM integration layer. Role based access control (RBAC), encryption protocols and audit logging are measures that ensure security and governance. They are also based on the framework of real-time and batch data processing pipelines, allowing organizations to process the requirements of data integration both in a stream form and the historical data form. Optimization performance tricks like caching, load balancing and parallel processing are also included to improve the efficiency of the system. As experimental analysis shows, the suggested framework is more sustainable in terms of scalability, decreased latency and achieved greater precision in data than the conventional monolithic schemes of integration. Its ability to manage large volumes of enterprise data has been pointed out by the findings. This study is relevant to the field as it offers a highly detailed and applicable method to the construction of scalable, secure, and efficient systems of data integration based on the Net ecosystem, thus helping organizations gain strong MDM implementation and data management.
References
[1] Kimball, R., & Ross, M. (2013). The data warehouse toolkit: The definitive guide to dimensional modeling. John Wiley & Sons.
[2] Stonebraker, M., Madden, S., Abadi, D. J., Harizopoulos, S., Hachem, N., & Helland, P. (2018). The end of an architectural era: it's time for a complete rewrite. In Making Databases Work: the Pragmatic Wisdom of Michael Stonebraker (pp. 463-489).
[3] Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Khan, S. U. (2015). The rise of “big data” on cloud computing: Review and open research issues. Information systems, 47, 98-115.
[4] Scannapieco, M., & Batini, C. (2006). Data quality: concepts, methodologies and techniques. Springer.
[5] Otto, B. (2011). Organizing data governance: Findings from the telecommunications industry and consequences for large service providers. Communications of the Association for Information Systems, 29(1), 3.
[6] Inmon, W. H. (2005). Building the data warehouse. John wiley & sons.
[7] Dreibelbis, A. (2008). Enterprise master data management: an SOA approach to managing core information. Pearson Education India.
[8] Newman, S. (2021). Building microservices: designing fine-grained systems. " O'Reilly Media, Inc.".
[9] Dragoni, N., Giallorenzo, S., Lafuente, A. L., Mazzara, M., Montesi, F., Mustafin, R., & Safina, L. (2017). Microservices: yesterday, today, and tomorrow. Present and ulterior software engineering, 195-216.
[10] Zhang, Q., Cheng, L., & Boutaba, R. (2010). Cloud computing: state-of-the-art and research challenges. Journal of internet services and applications, 1(1), 7-18.
[11] Morrison, R., Balasubramaniam, D., Greenwood, R. M., Kirby, G. N. C., Mayes, K., Munro, D. S., & Warboys, B. (2000). An approach to compliance in software architectures. Computing & Control Engineering Journal, 11(4), 195–200. https://doi.org/10.1049/cce:20000410
[12] RAHMAN, M. A., & Arifur Rahman, M. (2022). Enterprise Resource Planning and Customer Relationship Management Integration: A Systematic Review of Adoption Models And Organizational Impact.
[13] Talburt, J. R., & Zhou, Y. (2015). Entity information life cycle for big data: Master data management and information integration. Morgan Kaufmann.
[14] Mahmud, D., & Ikbal, M. Z. (2022). The role of etl (extract-transform-load) pipelines in scalable business intelligence: A comparative study of data integration tools. ASRC Procedia: Global Perspectives in Science and Scholarship, 2(1), 89-121.
[15] Nwokeji, J. C., & Matovu, R. (2021). A systematic literature review on big data extraction, transformation and loading (etl). Intelligent computing, 308-324.
[16] Loshin, D. (2010). Master data management. Morgan Kaufmann.
[17] Allen, M., & Cervo, D. (2015). Multi-domain master data management: Advanced MDM and data governance in practice. Morgan Kaufmann.
[18] Bakshi, K. (2017, March). Microservices-based software architecture and approaches. In 2017 IEEE aerospace conference (pp. 1-8). IEEE.
[19] Bergamaschi, S., Beneventano, D., Mandreoli, F., Martoglia, R., Guerra, F., Orsini, M., ... & Magnotta, L. (2017). From data integration to big data integration. In A Comprehensive Guide Through the Italian Database Research Over the Last 25 Years (pp. 43-59). Cham: Springer International Publishing.
[20] Mutlu, O. (2013, May). Memory scaling: A systems architecture perspective. In 2013 5th IEEE International Memory Workshop (pp. 21-25). IEEE.
[21] Hu, H., Wen, Y., Chua, T. S., & Li, X. (2014). Toward scalable systems for big data analytics: A technology tutorial. IEEE access, 2, 652-687.
[22] Rabl, T., Sadoghi, M., Jacobsen, H. A., Gómez-Villamor, S., Muntés-Mulero, V., & Mankowskii, S. (2012). Solving big data challenges for enterprise application performance management. arXiv preprint arXiv:1208.4167.
