AutoML-Enabled Infrastructure for Adaptive Personalization in High-Traffic Shopping Events
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V7I1P106Keywords:
Automated Machine Learning, AutoRecSys, Recommender Systems, Low Latency, Feature Store, Adaptive Personalization, High-Traffic E-commerce, MLOps, Train/Serve SkewAbstract
The necessity for highly adaptable, ultra-low latency personalization systems is critical in high-traffic e-commerce environments, particularly during unpredictable demand surges such as high-volume shopping events. Traditional machine learning (ML) paradigms, constrained by manual feature engineering, model design, and slow iteration cycles, are fundamentally unsuitable for managing massive, unpredictable load and dynamically changing user intent during peak traffic events. This paper introduces a resilient architectural framework built upon the Automated Machine Learning (AutoML) paradigm, specifically optimized for deep recommender systems (AutoRecSys). The proposed architecture details an ultra-low latency feature pipeline integrating Apache Kafka for high-frequency ingestion, Apache Flink for stream processing, and dedicated Feature Stores (Redis/Delta Lake) to mitigate catastrophic train/serve skew and ensure comprehensive data consistency. The efficacy of AutoRecSys techniques such as sampling-based searches and multi-fidelity optimization is substantiated by demonstrating quantified production benefits, including architectural efficiency gains that result in up to a 25% Query per Second (QPS) increase and resource optimization that reduces embedding parameters by 50% to 95%. The holistic, integrated system consistently achieves an end-to-end latency below the critical 100-millisecond threshold, providing a robust, evidence-based blueprint for deploying adaptive, scalable ML solutions in high-stakes, time-constrained production environments.
References
1. P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. (2015). Apache Flink™: Stream and Batch Processing in a Single Engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4), 28–38.
2. R. Zheng, L. Qu, B. Cui, Y. Shi, and H. Yin. (2021). AutoML for Deep Recommender Systems: A Survey. ACM Transactions on Information Systems, 1, 1, Article 1.
3. J. Hu et al. (2023). AutoML for Large Capacity Modeling of Meta's Ranking Systems. arXiv preprint arXiv:2311.07870.
4. K. Stankeviciute, M. Alaa, A., & M. van der Schaar. (2021). Conformal time-series forecasting. European Journal of Operational Research, 314, 111–121. URL: https://www.sciencedirect.com/science/article/pii/S0377221723006859.
5. G. Li, S. Wu, and C. Xiao. (2024). MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding. arXiv preprint arXiv:2508.11999v2.
6. J. Smith. (2024). Designing Ultra-Low Latency Data Pipelines to Power ML Models for Real-Time Autonomous Decision Making. ResearchGate. URL: https://www.researchgate.net/publication/395477469_Designing_Ultra-Low_Latency_Data_Pipelines_to_Power_ML_Models_for_Real-_Time_Autonomous_Decision_Making.
7. F. Hutter et al. (2025). Automated Machine Learning (AutoML) review. arXiv preprint arXiv:2505.18243v1.
8. D. Liu et al. (2025). Multimodal Representation Learning Challenges and Solutions for E-commerce Recommendation Systems. arXiv preprint arXiv:2511.11305v2.
9. R. Zhang et al. (2025). Taxonomy of Challenges in Metaverse Systems. arXiv preprint arXiv:2510.09621v1.
10. S. Li et al. (2024). Human-Centered AutoML: Goals, Techniques, and Future Directions. arXiv preprint arXiv:2406.03348v1.
11. M. Zhai et al. (2024). A Study on Ranking-Based Loss Functions in Neural Architecture Search. arXiv preprint arXiv:2506.05869.
12. H. Lee. (2024). Comparison of AutoML Libraries and Recommendation Systems for E-commerce Tasks. arXiv preprint arXiv:2402.04453.
13. L. Cheng-Ju et al. (2020). Machine learning-based e-commerce platform repurchase customer prediction model. Plos One, 15(12):e0243105.
14. M. W. El-Said, and M. I. Aly. (2024). Real-Time Data Processing in E-Commerce: Concepts, Challenges, and Cloud Solutions. International Journal of Advanced Engineering Technologies and Innovations, 1(3), 298–307.
15. B. H. M. S. Waseem, H. U. Zaman, F. H. Khan. (2022). Real-Time Analytics: Concepts, Architectures, and ML... IEEE Xplore.
16. P. A. Alagumalai et al. (2022). An efficient anomaly detection and classification approach for rare events in audio data. IEEE Xplore.
17. Y. Xu, B. Cai, M. Ding. (2023). A Comprehensive Survey on Automated Machine Learning for Recommendations. ResearchGate. DOI: 10.1007/978-981-99-4328-9_13.
18. Hemish Prakashchandra Kapadia, Krunal Bharatbhai Thakkar. (2024). AI based user behavior prediction for web navigation, International Journal of Research and Analytical Reviews - (IJRAR), 11(3), 397-405, https://www.ijrar.org/papers/IJRAR24C2956.pdf
