From 30 Minutes to 3 Seconds: Automated LLM Memory Recall Testing with pytest + FAISS

At 1:30 a.m., the user group suddenly became active with a question: Why did the AI forget my dietary restriction again? After checking my memory store, I found that the "no cilantro" restriction was still there, yet top-k recall had failed to include it. Comparing 20 memories manually took 30 minutes. As I lay in bed, the exhaustion hit me: this kind of regression testing needed automation.

The design of the LLM memory pipeline typically involves conversation->extract memory->vectorize->write to FAISS/vector store->embed query->top-k recall. However, recall inconsistencies often stem from subtle issues like unnormalized vectors, drifting embedding models, order changes during index rebuild, top-k truncation, or incorrect similarity threshold settings.

Addressing these problems required an automated verification suite that could run in seconds locally and on every commit. The chosen solution was pytest, with FAISS serving as the vector store. This combination allowed for deterministic embeddings, reproducible runs, and clear assertions.