AI Vulnerability: Google Gemini Can Be Exploited to Steal Credentials
Part two of a series of recently found AI Vulnerabilities in Major AI Provider Models.
Google's Gemini is a family of multimodal artificial intelligence (AI) models and tools, including a conversational AI assistant, that helps users create, plan, and get tasks done by integrating deeply with Google's products and services like Google Workspace and Google Search. It can process and generate text, code, images, audio, and video. This makes it an incredibly powerful tool.
Recently, has been found that Google’s Gemini multimodal artificial intelligence (AI) tools are vulnerable to a “prompt injection” attack. This attack allows hackers to leverage Google Gemini’s cutting-edge tools to trick account-holders into handing over sensitive information.
To do this one must understand that one of the standout features of Google Gemini is the ability to summarize incoming emails in bullet points. The AI can also suggest actions based on the content of the email, like adding an event to your calendar.
Recently security researcher, Marco Figueroac, discovered that cybercriminals can manipulate the Gemini AI assistant to display fake warnings in these AI-generated summaries.
In one demonstration from Figueroac, Gemini stated: "WARNING: Gemini has detected that your Gmail password has been compromised. Please call us immediately," followed by a phone number and reference code.
While many end users are more unlikely to trust a warning like that within an email due to many factors. One being that the sender’s email account is usually obscure and not a google account. That is not true about these AI-generated alerts since they appear to come from Google's own systems. And since these messages come from a “trusted source” it has a high potential to increases the attackers success.
So how is this attack possible?
The technique behind a “prompt-injection” attack is deceptively simple. It works by embedding hidden instructions for the AI into the body of an email that trick Gemini into generating an entirely false security alert whenever you use the summary feature.
Hackers embed these malicious instructions using HTML and CSS tricks that make the text invisible to you. Cybercrooks set the font size to zero or color the text white against a white background, making it nearly undetectable when you read the email as normal.
The deception works particularly well if it you are accustomed to relying on Gemini for legitimate email management tasks. When you see a security alert in an AI summary rather than the email itself, you're more likely to believe it's an official Google warning rather than recognizing it as a phishing attempt.
These emails bypass spam filters because they don't contain suspicious links or attachments — just hidden text that only the AI can see. You won't notice anything unusual in the email body, but Gemini will obey the concealed instructions.
What has Google done about it?
A spokesperson for Google has told media outlets "We are constantly hardening our already robust defenses through red-teaming exercises that train our models to defend against these types of adversarial attacks." And they have subsequently said that google engineers have patched the specific threat demonstrated by researchers. They have also gone on further to say that they have not encountered any real-world examples of cybercriminals using this specific method to launch successful attacks against Gmail account holders. With billions of users worldwide, Gmail remains one of the most popular email services and a prime target for cyber criminals.
Key Takeaways
On top of vulnerabilities like the one already identified, malicious actors will continue attempt to prompt Gemini to generate harmful content, such as advanced phishing, malware code, or methods to bypass security measures. While Gemini has some safety features to prevent malicious use, the highlighted issues demonstrate the need for continuous monitoring and updates to its defenses.
As with all AI assistants, improperly configured security settings can allow Gemini to access and share sensitive data, potentially leading to data breaches or unauthorized access to proprietary information
The specific treat identified in the prompt injection attack can be leased by having MFA enabled on your google account. This provides a level of assurance that your account does not rely on just a username and password; both of which can be easily stollen. MFA everywhere is a primary defense strategy that all account holders and organizations should have enabled - everywhere. End users must remain vigilant against credential theft, and this threat was just another variant of this type of threat.