A Multi-Cloud Architecture for Distributed Task Processing Using Celery, Docker, and Cloud Services
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V1I2P101Keywords:
Multi-cloud, distributed computing, Celery, Docker, scalability, fault tolerance, cost optimization, cloud services, task processing, containerizationAbstract
In the era of cloud computing, the ability to efficiently manage and distribute tasks across multiple cloud environments is crucial for modern applications. This paper presents a novel multi-cloud architecture designed to facilitate distributed task processing using Celery, Docker, and a variety of cloud services. The proposed architecture leverages the flexibility and scalability of containerization and cloud services to provide a robust, cost-effective, and highly available solution for task management. We detail the design and implementation of the architecture, including the integration of Celery for task queuing, Docker for containerization, and cloud services for resource management. We also present a performance evaluation of the system, demonstrating its effectiveness in handling large-scale distributed tasks. The results show that the proposed architecture can significantly improve task processing efficiency and reduce operational costs
References
1. Celery Documentation. (n.d.). Retrieved from https://docs.celeryproject.org/
2. Docker Documentation. (n.d.). Retrieved from https://docs.docker.com/
3. AWS Documentation. (n.d.). Retrieved from https://docs.aws.amazon.com/
4. Azure Documentation. (n.d.). Retrieved from https://docs.microsoft.com/en-us/azure/
5. Google Cloud Documentation. (n.d.). Retrieved from https://cloud.google.com/docs
6. https://towardsdatascience.com/serving-deep-learning-algorithms-as-a-service-6aa610368fde/
8. https://moldstud.com/articles/p-dockerize-your-celery-app-with-this-step-by-step-guide
9. https://softwaremind.com/blog/multi-cloud-architecture-guide/
10. https://www.dabbleofdevops.com/blog/deploy-a-celery-job-queue-with-docker-part-1-develop
