SentinelLABS — Can Frontier Models Tackle Autonomous Long-Horizon Malware Analysis?

The SentinelLABS research project explores the capabilities of frontier models in autonomous long-horizon malware analysis. These models, such as OpenAI's o1-preview and Google DeepMind's Big Sleep, have shown promise in vulnerability discovery but are yet to be widely applied in cybersecurity. The study focused on the complex Windows toolkit, fast16, which is designed to sabotage high-precision solvers used for nuclear-weapons behavior.

The sample was chosen due to its layered design, which makes shallow analysis difficult. The researchers built a custom harness, inspired by Rolf Rolles' methodology, to remove prompting variability across models. This harness provides the model with a sandboxed workspace, a nono shell, and a requirement to produce fully annotated IDA databases.

The model must then walk the execution flow and detail findings, with every claim anchored to specific addresses and artifacts. The study found that the models were unable to autonomously triage the fast16 sample, converging on the same plausible but inaccurate conclusions as human reverse engineers. However, when used as assistants, the models significantly reduced the time required for analysis.

The research highlights the need for further investigation into the potential of frontier models in malware analysis and reverse engineering.