← All IntelClip / EducationAppshots beat screenshots because they carry the accessibility tree
From Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex · ≈16:06
Passing structured app state instead of pixels removes the OCR-plus-lookup hop chain — the agent already has channel and user IDs, so replying is a single tool call.
What’s in it
- Passing structured app state instead of pixels removes the OCR-plus-lookup hop chain — the agent already has channel and user IDs, so replying is a single tool call.
Clip transcript
and it does it automatically with appshots. So again, I don't know how many times I'm going to say this, but appshots is one of my favorite features. How many people here have just like sent a screenshot to Slack to Codeex, right? Like almost everybody. But the issue is the screenshot does not have that much information, right? The model has to then do OCR. And if you send a screenshot of like a Slack thread, the model has to like read the Slack thread and then do a list Slack channels function and then realize like there's a guy named Charlie and then do like a list persons. It takes a lot of hops. But with appshots, it takes not only the image but the entire accessibility tree of the app. And so when I give it an appshot of a Slack channel, it knows the channel ID. So it knows exactly what function to call to post there. It has the user ids of every single person in that channel. So if I take an appshot and say do some research and reply, it's only one function call. It knows to send the send a message to channel like U12725 and then because you know it knows that Charlie is like U425, it can do that in a very fast hop. So, not only is it a very quick way of getting context into your system, it just gives so much more context that these subsequent tool calls do a really good job. I have not like filled out a form in like two weeks because I just now tell Codeex to fill out this form, right? It it knows all the fields. It then figures out that it's in Chrome and so it'll use the browser extension. If it's in Safari, it'll use computer use. The model has become really really intelligent.
Comments
Checking sign-in…
Loading comments…