Giving your AI a Job Interview

Choosing an AI for work requires testing the judgments it will actually make. Ethan Mollick proposes giving competing models representative assignments, repeating the trials, and having domain experts assess the results. Mollick's small experiment asking models to rate a guacamole-delivery business illustrates why: different models repeatedly gave different advice about the same proposal. That exercise does not establish permanent personalities, but it identifies something an organization could test systematically: whether a model's recommendations suit its actual decisions. The proposed job interview for AI evaluates both task performance and patterns of judgment.

“Every benchmark has flaws, but they are all trending the same way - up and to the right.”