I used Jev to build an off-topic detector for meetings

Max Harrison built the Offtrackometer, a web app that listens to a meeting and shows on a meter how far the conversation has drifted from the meeting's stated goal (Harrison's post). The judging is done by a decision model rather than a chatbot-style language model. Instead of generating text word by word, a decision model takes some context plus questions whose answer types are defined in advance, such as a choice, a score or a probability, and returns a structured answer that code can use directly. TypeSafe AI introduced Jev as a model of this kind in September.

Despite the post's title, the app does not run Jev itself. It runs Clef-flash, which Harrison picked because he already deployed on Cloudflare and it was the cheaper of the two decision models Cloudflare offered. Cloudflare trained its Clef models itself and describes them as compatible with Jev's API (Cloudflare announcement). As speech is transcribed, using Chrome's on-device speech recognition with a small in-browser model as a fallback, the app sends the meeting goal, some earlier discussion and the latest speech, and asks whether the latest part has drifted from the goal. The model returns a number between 0 and 1, which the app smooths and displays as a percentage. A sensitive mode looks at roughly the last ten seconds of speech and waits at least three seconds between checks; a balanced mode looks at roughly thirty seconds and waits at least ten.

Harrison says an earlier version built on Gemini Flash was "pretty unreliable and not very reactive," and that the decision model was easier to integrate, more responsive and cheap enough to check every few seconds. He reports that the detector "works surprisingly well," but also that "none of this has been tested much yet" and that being off track is hard to define: a detour can still serve the meeting's goal, and a short snippet may not show it. His next step is to build evaluations on real meetings, though his public demo does not collect meeting transcripts.