Building an AI Customer Support SaaS with Django, RAG and Self-Hosted LLMs

Over the past few months, the author has been developing AI-Autofy, an AI customer-support SaaS platform. This platform allows businesses to train an assistant using their own website, documents, FAQs, and business data. While it may seem simple to build an AI chatbot, there are several challenges to consider when providing reliable business-specific answers, tenant isolation, live data, product information, images, analytics, and predictable inference costs.

The author outlines the basic architecture of the AI customer-support SaaS, which is built using Python and Django. The Django framework handles various aspects such as customer accounts, subscriptions, AI configuration, knowledge-base management, chat history, analytics, widget configuration, tenant separation, and integrations.

The AI inference layer is separated from the main Django application to prevent the web application from running the language model itself, allowing for independent scaling and easier model updates.

A crucial component of the system is Retrieval-Augmented Generation (RAG). This approach enhances the AI's ability to answer customer support inquiries by leveraging its own information, such as data from the company's website or PDFs. The RAG process involves several steps, including creating embeddings, searching business knowledge, retrieving relevant documents, building an LLM prompt, and generating a grounded answer.

Tenant isolation is vital to ensure that information from one company does not appear in responses generated for another company.

To achieve effective retrieval, a vector database is employed. However, the quality of retrieval depends on various factors, including chunk size, metadata, tenant filtering, similarity thresholds, query rewriting, and prompt construction. The author discovered that relevance filtering is essential for producing accurate answers and minimizing irrelevant context.

Knowledge data and live data are treated differently, with knowledge data consisting of static information such as website content, FAQs, and policies, while live data includes dynamic elements like products, prices, and availability.

The author emphasizes the importance of introducing relevance checks before enriching the prompt with live data. This approach helps maintain response quality and token efficiency. Additionally, the author discusses the benefits of incorporating relevant images into AI responses, such as providing product images in response to relevant queries. However, the system must employ relevance filtering to avoid displaying irrelevant images.

Self-hosting the language model is another significant architectural decision. While hosted AI APIs offer convenience during development, they can make it challenging to maintain predictable SaaS pricing due to variable external API costs. The author opted for self-hosted models running on GPU infrastructure, providing more control over model choice, token limits, capacity, cost per message, and scaling.

The data flow architecture includes a website widget interacting with Django, an AI service, a vector database, live data, and the LLM inference component.

Streaming responses play a crucial role in enhancing the user experience, particularly in chat applications. Perceived latency is almost as important as total generation time, as users appreciate incremental responses rather than waiting for an entire answer. This approach leads to a more responsive and engaging user experience.

In summary, building an AI customer-support SaaS with Django, RAG, and self-hosted LLMs presents several challenges and considerations. By separating AI inference from the main Django application, employing Retrieval-Augmented Generation with tenant isolation, treating static and live data differently, and incorporating relevance checks, the author has created a more effective and efficient AI system for customer support.