Vibeleaderboard
← All Intel
Intel / article

Use screen actions to chat with the Gemini mobile app

Source
support.google.com
Author
Google Gemini Help
Date

Context

This Google help page describes two ways the Gemini mobile app takes in what is on your phone's screen, and in both what you are looking at becomes for a Gemini chat rather than staying inside the app you were looking at (Google Gemini Help). The first is screen actions: when you open Gemini over an app, video or , suggestion chips appear, and tapping one uploads the screen, the file or the web address as context for the chat. Every chip except Ask about... submits that content as soon as you tap it, although you can press and hold a chip to turn auto-submit off. The second, Intelligent Screen Context, removes the tap. When it is on, Gemini decides for itself when a question needs the screen, so asking what an image on your screen shows makes it check the screen automatically. That feature works only in English, and on Android it is controlled by two switches, Use text from screen and Use screenshot, under Screen context in Gemini's settings.

What the page does not describe is any way to limit this by app. Its switches are all or nothing: Gemini can use your screen, or it cannot. The EFF made the same point when it compared Android with Apple's new per-app exclusion list, saying Gemini's screen context cannot be set app by app (EFF), as covered in its guide. Signal, which cites this page in its announcement, says operating system makers give developers no easy way to keep such assistants out of an app, so on Android it switches on its own screen protection when it detects an assistant reading the screen, as described in Signal's post. Signal refers to Google's features as Gemini Intelligence, a name this help page does not use.

Key takeaways · AI-distilled
  • In the Gemini mobile app, tapping any screen-action chip except "Ask about…" automatically submits what is on your screen to Gemini. Long-pressing a chip offers Turn off auto-submit, and "Ask about…" can be switched to auto-submit the same way.
  • Screen actions appear when Gemini is opened by gesture or "Hey Google" over some apps, videos or files. Tapping one uploads the on-screen information or the file, including screenshots, PDFs and URLs, as context for the chat.
  • With Intelligent Screen Context on, Gemini decides for itself when to use what is visible, for example checking the screen when asked "what's the image on my screen?". Google says it works only in English and is controlled by the Use text from screen and Use screenshot toggles.
Terms in this piece · Glossary
  • Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
  • File — A named unit of stored data that keeps its contents while no program is using it.
Read the source support.google.com
Recommended reads
Comments

Checking sign-in…

Loading comments…