Microsoft's AI Defense Research: Generating Detection Test Logs from Attack Procedures

Microsoft's AI defense research team has developed a method for generating detection test logs from attack procedures. The research focuses on using artificial intelligence to create structured logs that mimic real attack logs, without reproducing them word for word. This approach aims to improve the design and testing of detection rules while addressing challenges in collecting and sharing real attack logs.

The team used a multi-AI collaborative method, where one AI generates logs based on given attack techniques and operations, another AI evaluates the generated logs, and a third AI improves the logs based on the evaluator's feedback. This iterative process proved to be more effective than simpler methods, such as prompt-centric generation or reinforcement learning with ground truth logs.

The generated logs include key details like process names, parent process names, command lines, and the order of events. These logs are structured in a way that allows detection rules to be designed and tested more efficiently. However, the research acknowledges that synthetic logs are not a perfect substitute for real attack logs and are primarily useful for lab testing purposes.

The evaluation of the different methods focused on recall, comparing the meaning of synthetic logs against ground truth logs. While the prompt-centric method served as a baseline, the multi-AI collaborative approach showed the best results, significantly improving recall rates across various evaluation datasets. The research also demonstrated that the generated logs could accurately represent process relationships and command line meanings, even when the exact strings differed from the ground truth logs.