Multi-Modal AI Comparison: Vision Tasks Across Models

okay so i actually did this – ran 100 real world vision tasks across gpt-4o, claude 3.5 sonnet, and gemini 1.5 pro over about three weeks. not synthetic benchmarks, not cherry picked screenshots. stuff i actually needed done: reading receipts, analyzing charts, describing product photos, pulling text from handwritten notes, identifying objects in cluttered scenes. here’s what i found.

## the results were way more task-specific than i expected

nobody dominates everything. if someone tells you one model “wins” at vision they’re either lying or only tested one category. my breakdown across the 100 tasks:

– **document/text extraction**: gemini was genuinely the best here. receipts, PDFs, dense text in images – it got more fields right with less hallucination than the others
– **chart and graph interpretation**: gpt-4o pulled ahead. it would actually explain trends and outliers, not just describe visual elements
– **creative/ambiguous images**: claude surprised me. it was more nuanced with stuff that had subtext or wasn’t literal. asked it to analyze a messy whiteboard photo and it inferred context the others missed
– **product photography**: pretty even across all three honestly, though gpt-4o had slightly better color accuracy descriptions
– **handwriting**: all three struggled, but gemini failed more gracefully – it flagged uncertainty instead of confidently hallucinating words

## the gotchas nobody mentions

a few things that tripped me up and probably matter if you’re building anything on top of these:

1. image size and compression matters a lot – i was resizing everything to under 1mb and claude in particular got noticeably worse on fine detail tasks. once i stopped compressing, its scores jumped
2. prompt framing changes everything. “what’s in this image” gives you garbage. “list every line item and its price from this receipt in a table” gives you something usable
3. gemini has the highest token context for vision which matters if you’re batching multiple images in one call – gpt-4o gets weird when you push past 4-5 images per request
4. latency varies a lot by time of day. i wasn’t logging this carefully enough early on and it skewed some of my subjective quality notes

for context on why i ran this – i’ve been doing a lot of AI-assisted content work, some of it through [walterwrites.ai](https://walterwrites.ai) which handles the heavier lifting when i’m swamped, and i kept running into situations where vision accuracy affected downstream writing quality. turns out which model you use for the “read this image and summarize” step actually matters for what the final copy looks like.

## what i’d actually recommend

if you’re doing mixed vision tasks and can only use one model: gpt-4o is the safest default. it’s not best-in-class at any single category but it fails the least badly across all of them.

if you’re doing document processing specifically: route to gemini.

if you’re doing anything that requires interpretation over description – like “what does this graph suggest about user behavior” vs “describe this graph” – claude is worth the extra consideration.

i’m going to keep adding tasks to this dataset. currently at 100 but want to get to 250 before i publish anything more formal.

anyone else tested vision tasks systematically? curious if you’re seeing the same gemini advantage on text extraction or if that’s specific to my test set.