Distributed RAG Architecture for Enterprise Knowledge QA on Databricks

Abstract

Organizations rely on centralized knowledge repositories like Salesforce Knowledge to support internal teams and external teams. The traditional keyword/rule-based search often doesn’t provide accurate, up-to-date information at scale. This paper presents a unified data-centric and model-centric Retrieval-Augmented Generation (RAG) architecture implemented on Databricks using Apache Spark. The architecture integrates large-scale ETL processing with efficient model execution for vector embedding generation within a single distributed framework. The proposed solution employs open and modular design with separate components for data extraction, data cleansing, workflow orchestration, embedding generation, vector retrieval and LLM inference.  In the proposed approach, knowledge articles are extracted by executing Salesforce Object Query Language (SOQL) queries against a Salesforce Knowledge object REST API endpoint and stored in a Databricks Lakehouse. The RAG pipeline employs stateful model reuse and fine-grained resource allocation coupled with concurrent multithreaded encoding within Databricks to perform large-scale vector embedding generation.  The generated vector embeddings from the extracted content are stored in a vector database. At inference time, user queries from the chatbot are converted into vectors and matched against the vector database to find most relevant information chunks which are then sent along with prompts to an LLM to generate responses. This design ensures that answers are traceable to authoritative knowledge sources, reduces hallucinations, and supports continuous updates as knowledge articles evolve.The RAG pipeline implemented on a Databricks cluster demonstrated 100% CPU utilization with high execution efficiency on a vector embedding pipeline as shown in the experimentation. The proposed architecture provides fine-grained control over resource allocation, concurrency, and batching strategies, enabling enterprises to generate high quality embeddings while leveraging existing infrastructure without significant redesign.