The Two-Pass Grading System: How Context Changes the Score

The Two-Pass Grading System: How Context Changes the Score

In the previous post, we explored the hardware limitations and the choice of a local LLM for grading messages. Now, let's delve into the grading pipeline itself.

The grading process involves a two-step approach, referred to as the two-pass system. During the first pass, the messages are graded individually, regardless of whether they are part of a batch. This ensures that each message receives an individual evaluation, even if multiple messages are sent in a batch.

The first question posed is whether all messages are being discarded. The answer is a resounding no. The grading system employs heuristic-based message dropping to eliminate obvious noise in the data. By combining length and regular expression techniques, the LLM's workload is reduced, and a significant portion of redundant data, approximately 12%, is removed before reaching the grading stage.

False negatives are particularly undesirable in this context, as important messages should not be inadvertently dropped before reaching the grading pipeline.

The grading scale ranges from 1 to 5, with scores of 1 and 2 indicating noise, social, or logistical content. A score of 3 suggests that the message may be useful but lacks sufficient detail. Scores of 4 and 5 signify clear technical value, such as solutions, useful resources, practical experiences, or measured results. These valuable resources often include GitHub repositories, YouTube links, articles, or research papers.

After the first pass, messages graded as 3 are flagged for a second evaluation. The system then extracts the full metadata of the messages before and after the quality 3 message during the second pass. The LLM is instructed to use the surrounding messages as context when re-grading the initially scored 3 message. This ensures that the grading system remains effective while retaining the important messages.

Interestingly, the two-pass system was initially tested on a test dataset where the LLM's grading was compared against human verification. The system achieved a mean grading confidence of 0.89 on actual data. However, the researchers discovered that grade quality 3 was ambiguous, as it only received a 3 because it lacked enough data to support a higher grade in the grading scale.

The two-pass system was further tested on a real dataset, and it proved to be effective in grading messages on a domain-specific scale. The researchers found that individual messages in the tech domain could be accurately graded without the need to bundle them with surrounding context, as most of the data points in this domain are clearly relevant and do not require additional context for accurate grading.

This approach prevents unnecessary exploitation of the context window and reduces the likelihood of dumping irrelevant information into the grading process.