Evaluating Success Of Consumer Generative Ai Products

A useful AI feature can have low repeat usage. In LinkedIn's March 2024 account of evaluating generative AI products, profile-writing assistance is an example: someone satisfied with a revised summary may have no reason to return until their career changes. Acceptance of the suggested text and the amount of editing it needs are more relevant measures.

The company combines usage measures with human review and direct user feedback. Each answers a different question: reviewers inspect output quality, feedback reveals users' reactions, and usage shows whether people complete or return to the intended activity. Choosing the activity first helps avoid rewarding engagement that has little connection to the user's goal.