Why Most AI Benchmarks Are Misleading
benchmarks dont measure what matters for real use cases. heres why and what to look at instead
also worth noting – multimodal models are where the most interesting work is happening
am i the only one who thinks this?
6 Replies
Join the discussion.
Log In to Reply
is it really shrinking though? evals are still mostly academic setups. production has noise, bad inputs, weird edge cases. the gap feels wider to me in practice not smaller.
the benchmark problem is specifically bad for reasoning tasks. pass@1 on HumanEval tells you almost nothing about how a model handles ambiguous requirements from a non-technical stakeholder.
what actually matters for my clients is consistency across a long session. no benchmark i've seen tests for quality degradation on message 40 vs message 4. that gap destroys real workflows.
the session degradation thing is real. i noticed claude starts hedging way more after long back-and-forths. is that a context window issue or something else?
I actually wrote something about this a few weeks ago. the key insight for me was arxiv summaries are the only way i stay current anymore. changed how i think about the whole thing