Deepseek V3 performs surprisingly bad in Misguided Attention eval, which tests for overfitting.
The Misguided Attention eval is a collection of prompts that are slight variations of commonly known thought experiments, riddles or paradoxes ("trick questions"). Most LLMs are overfit to the "normal" version of these questions from their pretraining and will provide an answer based on the unmodified problem. This is a test to show how well the LLM is able to attend to "weak" signals.
Deepseek V3 solved only 22% of the prompts in the 13 test questions. This is unexpectedly bad for a new model of this size and vintage. It appears that some of the optimizations (The compressed KV cache? MoE?) made it more sensitive to overfitting.
Edit: You can find a full evaluation with most common models here. The heatmap below only shows non-reasoning flagship models.
Edit: Some observations from the V3 evaluation.
- It failed some prompts where a single word was added that it did not detect (schroedingers cat, trolley problems). I generally observe that dense models seem to perform better here.
- It got caught in repetitive loops for problems that were not solvable (e.g. jugs4 liters, rope problems). This looks like a finetuning issue - possibly because it was trained on reasoning traces?
You can see the model responses here.