How We Built an LLM Review Pipeline and Why 91.67% Accuracy Wasn’t Enough

Our team developed a pipeline to review competitor reviews using large language models (LLMs). Initially, the process involved human reviewers taking screenshots of public reviews and inputting them into ChatGPT to estimate monthly review volume. However, this method was unreliable and produced non-transparent results.

The team aimed to track competitors by determining which companies, markets, and frequency of reviews should be monitored, as well as defining what would be considered a real signal. They began with five companies in two markets and expanded to fourteen companies across three markets, running the pipeline nightly.

Initially, the team used prompting techniques without fine-tuning the model. After about a week, they obtained a stable prompt. The primary challenge was ensuring the output was simple enough for another process to parse nightly. Each review was assigned a single sentiment label and up to five category labels, which were stored as pipe-separated values.

The team initially used a single review per request, but this led to timeouts, rate limits, request limits, and memory pressure. They settled on processing twenty reviews per request, which proved to be a stable point in their environment.

The team evaluated the classifier using a stratified proportional sample of 300 reviews across active domains. They set an 85% acceptance threshold for overall sentiment accuracy, and the classifier achieved 91.67% accuracy. However, further evaluation revealed that neutral reviews were where the model struggled, recalling only 19 out of 33 neutral examples and resulting in a neutral F1 score of 0.62.

The team also analyzed five most frequent categories in the sample. While some categories like DeliveryIssues (92% precision and 89% recall) performed well, others such as Other (58% precision and 71% recall) and ProductQualityComplaints (100% precision and 50% recall) had lower performance. The error file showed that seventy-five out of 300 reviews had at least one disagreement, with category misclassifications being the most common.

The team decided to postpone major taxonomy revisions and instead focused on adjusting the model's temperature from 0.5 to 0.3. This change helped improve the headline score, but the team recognized that unfinished work on the taxonomy could lead to future issues. In conclusion, while the initial 91.67% accuracy was promising, it wasn't enough to fully address the challenges posed by neutral reviews and category misclassifications.