AI Beats Human Writers in New Benchmark

A new Creative Writing benchmark from Vulsar AI indicates that cutting-edge AI models can now match amateur human writers in long-form writing, but professional writers still lead. The benchmark examined 475 prompts, comparing 24 large language models with human writers. Evaluations were made using a reward model fine-tuned on human preference data to determine reader preference for each response.

GPT 6 Astra emerged as the top performer, with an 87.8% predicted win rate, narrowly surpassing the 86.6% average of human writers. However, when evaluating professional writers, the amateur human baseline performed significantly better than the AI overall. GPT 5.6 Sol trailed GPT 6 Astra with a 77.6% predicted win rate, while Claude Fable 5.1 scored 70.3%.

As the benchmark progressed beyond the most advanced frontier models, performance dropped dramatically. Models like Claude Opus 5, Kimi K3, and Grok 4.6 generally scored between 50% and 65%, while smaller models struggled, with Qwen3.8-27B scoring 23.2%, DeepSeek V4.1 Flash at 19.8%, and Gemma 4 26B finishing at 10.9%. The data suggests that strong creative writing abilities are still primarily found in a limited number of advanced AI systems, while human writers tend to generate longer responses and maintain coherence over extended texts.

Despite the narrowing gap between AI and casual human writing, professional creative writers still perform significantly better in this benchmark.