Build a RAG-Based AI Assistant in Kotlin with a Vector Database

Retrieval-Augmented Generation, or RAG, is a system that augments an LLM with external knowledge to enable applications to answer questions using private or frequently changing documents. In this tutorial, the focus is on building a Kotlin client for a backend RAG service.

RAG provides numerous benefits. For example, instead of sending entire company documentation to an LLM for each question, the system can:

- Split documents into chunks

- Generate embeddings for the chunks

- Store these embeddings in a vector database

- Convert user questions into embeddings

- Retrieve the most relevant chunks

- Send those chunks to the LLM

- Return a grounded answer based on the retrieved information

The architecture of a Kotlin client for such a backend RAG service can be organized using a clean Android architecture, such as:

- Compose UI

- ViewModel

- RagRepository

- API Client

- RAG Backend

The vector database and LLM should typically remain on the backend rather than being exposed directly to the mobile application. The API model defines two data classes: AskRequest and AskResponse.

The AskRequest data class includes a question and an optional conversationId, while the AskResponse data class includes an answer and a list of Source objects. The Source data class contains a title and a chunk of text.

A Retrofit interface is used to expose the backend. This interface includes a POST method called ask, which takes an AskRequest as input and returns an AskResponse.

The RagRepository class keeps network details outside the ViewModel. This class contains a suspend function called ask, which takes a question and an optional conversationId. The function sends the request to the backend and returns the response.

The ViewModel uses a UI state model to manage different states, including Idle, Loading, Success (with an AskResponse), and Error. The ask function is used to send the user's question to the repository, which then communicates with the backend via the RagRepository instance.

Document ingestion in the backend pipeline typically involves converting PDF, Markdown, or HTML files into text, chunking the text, generating embeddings for the chunks, and storing them in a vector database. Each chunk includes the text, the original document, and optionally, a section label.

Embeddings are generated by converting text into vectors, which can then be stored in a vector database that supports similarity search. When a user asks a question, the backend generates an embedding for the question and searches for similar chunks in the database. The top results are then added to the LLM prompt for generating a grounded answer.

To provide sources alongside the generated answer, a simplified prompt is used, including the retrieved chunks and the original question. The backend then returns the answer along with the sources, allowing users to verify the information.