Don't Start With RAG: Lessons From Building an Automotive AI Pipeline

When constructing an AI product, it's tempting to begin with trendy components like vector databases, RAG, agents, and multimodal models. However, when developing the automotive AI pipeline behind Inspecly, the team took a different approach. Instead of asking, "Which LLM should we use?" the initial question was, "What information do we have, how reliable is it, and which system should process it?" This distinction altered the architecture significantly.

The input is typically messy, as drivers rarely describe vehicle problems like mechanics. A driver might say, "My car makes a strange noise when I start it," but the request could also include a written description, a voice message, vehicle photos, an OBD scan, or other vehicle metadata. These inputs have varying reliability levels. An OBD code offers structured information, while a photo provides visual evidence. Voice messages represent the driver's observations, and vehicle metadata may require precise lookup.

To ensure the LLM can effectively reason about the problem, the team normalized the available evidence. The normalized object looks like this:

```

{

vehicle: {

make: ...,

model: ...,

vin: optional

},

description: "The engine loses power when accelerating.",

voice_transcription: null,

obd_codes: [...],

images: [...]

}

```

This object doesn't include a diagnosis; it describes the actual information known. Sending everything to one multimodal LLM is attractive for a prototype, but it's difficult to control. Consider these inputs: OBD code (deterministic lookup), photo (visual analysis), voice (transcription), and vehicle information (exact lookup/API).

Retrieving semantically similar paragraphs from documents instead of performing an exact lookup against validated fields is an important distinction. Structured data excels at exact identifiers, validated fields, controlled records, and deterministic queries. RAG, on the other hand, is useful when knowledge primarily resides inside documents.

When reliable structured information already exists, it should be used directly. For example, querying an OBD database when an OBD diagnostic trouble code is available can provide precise information. Structured data and RAG serve different purposes. Structured data is great for exact identifiers, validated fields, controlled records, and deterministic queries.

RAG becomes useful when knowledge primarily lives inside documents. If structured knowledge is missing, a tool-using agent can search for additional information, but the retrieved information should not be treated as equivalent to validated information. The system should preserve provenance to indicate the source of the information.

Images are evidence, not diagnosis. While a vision model can reveal dashboard warning lights, visible body damage, tire wear, fluid traces, or damaged components, a photo rarely tells the whole story. The output should include observation, confidence, limitations, and whether physical inspection is required, rather than simply stating that the vehicle has an oil leak.

Voice input is valuable because describing a mechanical problem verbally can be challenging. The message is transcribed and then becomes another input to the evidence layer. However, the distinction between the driver's description and confirmed technical facts should be maintained throughout the pipeline.

The final system should consist of multiple processing paths that contribute information to the generation context. This normalized evidence includes reported symptoms, OBD findings, visual findings, retrieved information, missing information, conflicts, and safety flags. This approach enables provenance tracking, conflict detection, confidence handling, evaluation, safety rules, and debugging.

The architecture becomes easier to evolve as the number of sources increases. The RAG pipeline is not implemented at the beginning, as adding PDFs to a vector database is not the challenging part. The real difficulty lies in knowing whether a retrieved procedure applies to the correct manufacturer, model, year, engine, transmission, vehicle version, document revision, and more.