Modular Data Pipeline Architecture for Large-Scale Supply Chain Optimization: A Framework for Scalable, Configuration-Driven Input Generation

Abstract

Large-scale supply chain optimization relies on the steady production of accurate, validated input datasets. These datasets feed the mathematical models that handle network design, capacity planning, and cost minimization. In most organizations, the data pipelines behind these models started life as monolithic scripts—single codebases that bundle business logic, configuration parameters, and execution dependencies all in one place. That works fine at small scale, but as the number of scenarios, data sources, and downstream consumers grows, these architectures turn brittle and expensive to maintain. This paper lays out a generalized framework for rebuilding supply chain input generation pipelines around modular, configuration-driven, cloud-native principles. We draw on existing work in data engineering, distributed systems, and operations research to describe the structural problems with monolithic pipeline design, spell out the requirements for a scalable replacement, and propose an architecture built on directed acyclic graph (DAG) execution, externalized configuration management, isolated multi-scenario runs, and formal version control. We evaluate the framework against quantitative benchmarks—end-to-end runtime, scenario throughput, parameter visibility, failure recovery—and show that modular pipeline architectures cut operational overhead, improve data quality, and let organizations scale their optimization capabilities across business units and planning cycles. The implications reach across manufacturing, retail, healthcare, and logistics wherever optimization models depend on complex, multi-source input generation.