DeepSeek’s cost-saving AI may lead to inaccuracy in model performance: research

A recent study has uncovered a possible flaw in DeepSeek's memory-efficient AI models, which could compromise their accuracy. The research, authored by ByteDance and Princeton, Stanford, and UC Berkeley researchers, suggests that DeepSeek's strategy of reducing memory usage for processing lengthy inputs might come at the cost of reliability.

The study, yet to undergo peer review, discovered that DeepSeek's compressed models, including V4 and V4.1 versions, tend to miss the same information when it appears in different positions within a document. This phenomenon was observed across multiple DeepSeek models during tests involving lengthy contexts of around 128,000 tokens.

The retrieval accuracy of DeepSeek-V4 models was found to vary by up to 40 percentage points. The researchers attribute these discrepancies to the manner in which DeepSeek compresses and groups information for storage, which can make certain details harder to recover depending on their location within compressed segments. For instance, inserting irrelevant text before the same piece of code proved challenging for DeepSeek to accurately retrieve the correct answer.

While the study's findings were based on controlled experiments and a limited set of coding examples, the researchers cautioned that their results may not fully represent real-world applications. DeepSeek and ByteDance have not commented on the study. The research underscores the challenges developers face in striking a balance between efficiency and reliability, particularly for applications requiring accurate retrieval of information from extensive documents or software code.