78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble

In a talk published by the AI Engineer channel, Bumble product manager Supriya Subudhi describes an A/B test of an AI feature that gave people feedback on their photos in a dating app with more than 20 million monthly users. An A/B test shows one version of a product to some users and another version to the rest, then compares the results. The talk's description says 78% of surveyed users wanted photo feedback, yet only 10 to 15% used it. Asking people what they wanted and watching what they did gave very different answers.

The description lists four more ways the result was weaker than a headline win suggests. The average improvement hid odd outputs, such as telling people who were already smiling to smile more. The best model cost about 10 times more, and switching to a cheaper model also shrank the results, so the gain depended on paying for that model. Regulation meant the feature could not ship outside the US. The talk ends with five questions to ask before calling an AI test a win.

This summary comes from the publisher's description and chapter list only. No transcript was available, so it cannot say how Bumble measured these effects, how large the gains were, or which regulations applied.