Gemini can now see your screen and judge your tabs

Google is rolling out new vision features for Gemini Live
The AI assistant can now “see” your phone screen or camera feed
These upgrades are powered by Project Astra, Google’s AI R&D umbrella

Google’s Gemini Live is finally getting the gift of sight. The tech giant has quietly begun rolling out features that transform your humble smartphone into an all-seeing eye for its AI assistant.

The new abilities were uncovered by a Reddit user who later shared a video of the features in action. The upgrade lets Gemini peer through your screen or camera lens and process what it sees. The rollout marks the debut of Google’s much-discussed and much-anticipated Project Astra.

Based on the video, Gemini’s ‘eyes’ can analyze your screen in real-time through a “Share screen with Live” button. Gemini has long been able to digest static screenshots, but the update maintains a continuous gaze on your screen, looking at whatever you are on your phone for better or for worse.

The other tool makes your phone’s camera Gemini’s eye. Google has demonstrated that the AI can precisely discern colors and objects. Whether the final product matches the platonic ideal of the demos isn’t clear just yet.

A short demo of Project Astra (Share screen with Live) from r/Bard

Astra eyes

The new feature is arriving first for Gemini Advanced subscribers paying $20 a month for the Google One plan with extra AI. The rollout is notably democratic in where the feature appears, though, judging from the Xiaomi phone shown by the Reddit user. Google had previously hinted that Pixel and Galaxy S25 owners would have faster or better access to Project Astra.

Other AI assistants with similar seeing tools exist, but they are mostly tied to third-party apps like Microsoft Copilot, ChatGPT, Grok, and even Hugging Face’s new HuggingSnap app. Having a real-time screen and camera-connect AI built into Android would certainly help entice users interested in an AI assistant to at least try Gemini.

And Google’s timing in releasing the feature is notable as it tries to carve out a lead among AI assistants. Though Amazon has been hyping its new “Alexa Plus” update, it has yet to arrive.

Meanwhile, Apple’s upgraded Siri has been delayed multiple times. That leaves Google with a temporary but very real lead in the AI assistant race. Gemini, for all its early hiccups and rebranding drama (RIP Bard), is now doing things that neither Alexa nor Siri can match for the moment.

Google has promised that Project Astra will be the “next-generation assistant” everyone wants to use all day. So keep your (and Gemini’s) eyes peeled for new features to arrive in the weeks ahead.

Gemini can now see your screen and judge your tabs

Don’t miss the Buzz!

Don’t miss the Buzz!

Like it? Share with your friends!

0 Comments