The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM's shouldn't follow this pattern because they're misnamed: GLM-5.3 and -flash are different architectures).
If this *is* the main factor, I'd expect Kimi K3.1 and Carbon would also do well on this test, while Haiku 6 would collapse. It's a falsifiable prediction.
DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.
I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM's shouldn't follow this pattern because they're misnamed: GLM-5.3 and -flash are different architectures).
If this *is* the main factor, I'd expect Kimi K3.1 and Carbon would also do well on this test, while Haiku 6 would collapse. It's a falsifiable prediction.
For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.