Gemini on Mac can now transcribe your speech and respond to what it observes.

      The Gemini app for macOS now features system-wide dictation and an optional capability that allows it to gather context from what's displayed on the screen to manage complex tasks.

      After introducing Gemini Spark on macOS earlier this year, Google is enhancing the Gemini app for Mac with two practical features that could transform your productivity, whether by replacing your existing AI dictation application or enabling Gemini to interact with whatever is on your display.

      One key press converts speech into clean text anywhere

      With this recent update, you just need to long-press the Fn key in any open window and start speaking. Gemini will transcribe your speech into clear text right at your cursor, eliminating filler words like "ums" and "ahs" as you go. This feature is enabled by default, requiring no prior setup, and is functional throughout the system, allowing you to compose an email, reply to a Slack message, or draft a document seamlessly.

      AI dictation applications like Wispr Flow and Willow already offer similar functionality, refining spoken words and displaying polished text at the cursor's location, but they necessitate a separate download and, in some instances, a subscription. With this same functionality integrated directly into the Gemini app, it may render standalone dictation tools unnecessary for Mac users.

      The dictation feature is being rolled out to all Gemini app users on macOS globally. Currently, it supports English dictation, but Google has indicated that more languages will be added later this year.

      Gemini can now interpret what's on your screen and respond accordingly

      In addition to system-wide dictation, the Gemini app for Mac has introduced screen-aware reasoning. This optional feature enables Gemini to read your active windows and use that information to manage more intricate requests. For instance, you could highlight a set of rough notes and ask Gemini to transform them into an executive summary, and it would rephrase the text and place it directly where your cursor is positioned.

      Google also provided a more complex illustration where a user organizing a team dinner requested Gemini to check highlighted files for information about the budget policy and dietary restrictions, and to draft a reminder email based on that information. Gemini also retrieved recommendations for nearby boba shops from Google Maps to add as a postscript in the email.

      With these new features, Gemini on Mac is evolving beyond simple responses. It has become a more powerful tool capable of transcribing speech and acting on screen content, positioning Google ahead of Apple, which has yet to release its own enhanced Siri with similar functionalities.

      Pranob is an experienced tech journalist with over eight years of experience covering consumer technology. His work has been…

      Google Docs has just added four new AI features to Gemini that streamline editing tasks

      Google Docs now enables Gemini to summarize comments, draft responses, and suggest edits, alleviating one of the most tedious aspects of collaborative editing.

      Anyone who has worked on a shared document knows the frustration of encountering numerous unresolved comments from multiple contributors. Google has now assigned the task of organizing that clutter to Gemini.

      So what capabilities does Gemini now offer for handling your comments?

      The music industry wants to eliminate AI-generated songs from the charts, and it has finally devised a plan.

      Universal, Sony, and Warner are advocating for global charts to exclude AI-generated music.

      I’ve found myself grooving to many new viral hits, only to discover they were produced with artificial intelligence. It seems the labels are equally tired of this trend.

      So what exactly are the labels requesting?

      Gemini is being integrated into Google Play, gaining insights into your spending habits.

      Google Pay is giving Gemini access to your transaction history.

      Reviewing your spending history on Google Pay can reveal where your money has gone, but sifting through a long list of merchants and payments can be challenging. Recognizing this, Google is incorporating Gemini into Google Pay.

      The company has launched Ask Google Pay, an opt-in conversational AI feature built directly into the payment app in India. Users can ask about their spending, seek savings advice, and learn about financial principles. It will even provide personalized offers based on user activity.