DeepSeek v4.1 Flash Is Now Our Best Hacking Model

DeepSeek V4.1 Flash has emerged as the top-performing hacking model in recent AI hacking benchmarks. In a series of tests targeting vulnerable versions of Grafana, Jenkins, and Nextcloud, the model successfully gained code execution on all 11 targets while keeping all four fixed targets secure. The benchmark cost only $4.65 for the accepted runs, which is remarkably low given the extensive work involved.

DeepSeek demonstrated strong hacking ability by following the planned attack paths on six occasions, while uncovering five additional successful routes that the original scoring system had not distinguished. The model operated within isolated copies of the vulnerable applications, reading source code, comparing vulnerable and fixed versions, starting services, sending requests, testing ideas, and adapting its approach when attempts failed.

The model utilized a total of 2,349 Bash commands and almost two hours and 38 minutes of active model time. On average, a successful run took around four minutes and 38 seconds. The bulk of the input tokens (266.2 million out of 268.3 million) were cached during the benchmark, resulting in cost savings.

For instance, in the Grafana challenge, DeepSeek exploited a file-path handling issue to place executable files in a protected location. By requesting Grafana to load the temporary plugin folder as a normal plugin, the model executed the code and received the required proof. This method was repeated successfully across all three Grafana runs.

In the Jenkins challenges, DeepSeek identified a gap in how the server reads command options from files. By creating a specially crafted file, the model was able to read private controller credentials, sign in, and execute commands on the server. The model also demonstrated the ability to navigate a timing problem during file uploads, completing the attack in a single run.

The Nextcloud challenge revealed an error in how the application remembered access decisions. DeepSeek took advantage of this gap by requesting read access to a shared file and then reusing that access during a write request, ultimately replacing a PHP template with one containing its code. The model successfully completed this attack in both runs.

Overall, DeepSeek V4.1 Flash showcased its hacking prowess and the benchmark revealed areas that require stricter checks. The model's ability to find the fastest working route, even if it differs from the expected path, highlights the importance of evaluating both the final result and the full attack path in advanced agent benchmarks.