Shopify Inbox Message Classification Model

Shopify's customer-message classifier for Shopify Inbox, its merchant chat app, improved after the team reworked what its labels meant. The prototype had grouped questions about whether an item could ship somewhere with questions about an existing order's status. Those messages imply different customer needs, so clearer categories mattered for a system intended to help merchants prioritize replies.

In its 2022 account, Shopify reported accuracy rising from about 70% to 90%, while the share of messages classified with high confidence rose from about 35% to 80%. The training set shrank from 40,000 examples to 20,000. Human labeling, a taxonomy growing from 20 to 45 classes and a switch to DistilBERT all accompanied the change, so the comparison measures the combined redesign rather than isolating the effect of cleaner data. Methods and results.