Intel / video

Dan Shipper: generation is cheap, ranking is the human edge

Source: video.twimg.com
Author: Dan Shipper
Date: Published Nov 11, 2026

Why it matters

Argues that as default model capability spreads, value shifts to the human perspective that selects among outputs. One experiment suggests Claude can generate a target answer within 400 samples but ranks it top-10% only about 60% of the time.

Key takeaways · AI-distilled

Shipper says Every's head of , Mike Taylor, tested whether Claude could predict Shipper's Slack replies: with 400 generated answers per question his real reply was among them 93% of the time, but Claude ranked it into its top 10% only about 60% of the time.
Given the same quarterly survey and the same model, Every staff drew different conclusions: Shipper saw a team-sales opening, the go-to-market engineer found first-month churn from too much email, and the head of marketing found readers feel isolated in their AI enthusiasm.
He argues that pitting a 2026 employee against a 2027 model is rigged: the employee will also use the 2027 model, so the real comparison is a person with a perspective steering it versus the model's default prompt, whose answers everyone can get.
He cites GDPval, where blind experts compare model output with human experts' work: models won about 12% in 2024, about 48% in 2025 and about 83% in 2026 with GPT-5.4. He says aggregate benchmarks like this must freeze the human side to measure progress.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source video.twimg.com