benchmarks
Type: Concept
Standardized tests for model performance; criticized as flawed, gameable, and weakly correlated with real-world capability (e.g., models winning IMO but failing at PDFs).
Field: AI evaluation
Mentioned in 1 podcast episode
Who brought it up
- The $1B Al company training ChatGPT, Claude & Gemini on the path to responsible AGI | Edwin Chen on Lenny's Podcast — Lenny Rachitsky