8. Synthetic customers, part 1: can a fake shopper judge your recommender?
source link: https://4thwithme.dev/blog/synthetic-customers-theory/ Picture a shop that sells colored and flavored toilet paper. Mint. Lavender. Bacon. You build a new "customers also liked" box. Offline, it scores well. You ship it to half the traffic. Three weeks later, the A/B test (a live split test) shows no difference. That is the normal story. This post is about a faster check: fake…
The post, part 1 of 2, discusses a method to test new recommender systems faster using simulated customers. It explains two flawed options - simulators that replay old logs, and A/B tests with real users. Simulators score well offline but lose performance online due to factors like site changes, lack of consideration for novelty, missing feedback loops, and biased data.
A simulator, with five components called C1-C5, learns fake users from real data and lets them interact with a recommender. The five parts include generating fake data, running a recommender on it, and measuring the results. The post details three types of simulated users - fake data without fake people, language models acting as shoppers, and train a small model on real feedback.
It also briefly mentions UserSimCRS v2, which focuses on conversational recommender systems with LLMs for text generation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
