# name
Should ChatGPT Read Your Encrypted Messages?

# content
By Mallory Knodel

Until recently, if most people wanted an LLM to help them compose a witty reply or a sensitive breakup text, they had to copy their messages out of their messaging app and paste them into a chatbot. Now, OpenAI’s new iMessage plugin for ChatGPT Work and Codex users on Mac lets users search their messages and draft and send replies through the chatbot on their desktop.

As Bloomberg notes, there are several layers of permission required before the plugin works, so no one will install it by accident. But for the people on the other end of the equation, the unsuspecting users on an end-to-end encrypted platform sending messages to their contacts, there is no such protection.

This plugin bypasses the essential promise of E2EE: that only the sender and intended recipients can read a message. That is because it appears that the plugin reads the decrypted message on the computer, then sends them to OpenAI’s servers so a third party can read the plain text.

This is exactly the scenario my co-authors and I discussed in our paper “How To Think About End-To-End Encryption and AI: Training, Processing, Disclosure, and Consent,” in which we concluded that processing E2EE content outside the device is only compatible with encryption if no third party can see it, and the content is used only to fulfil the user’s request.

The OpenAI iMessage plugin violates the first three of our four recommendations from the paper, and partially violates the fourth as it does not require meaningful consent from both parties. Our recommendations were:

1. Training. Using end-to-end encrypted content to train shared AI models is not compatible with E2EE.

2. Processing. Processing E2EE content for AI features (such as inference or training) may be compatible with end-to-end encryption only if the following recommendations are upheld:

1. Prioritize endpoint-local processing where possible

2. No third party can see or use any E2EE content without breaking encryption

3. A user’s E2EE content is exclusively used to fulfill that user’s requests

3. Disclosure. Messaging providers should not make unqualified representations that they provide E2EE if the default for any conversation is that E2EE content is used (e.g., for AI inference or training) by any third party.

4. Opt-in consent. AI assistant features, if offered in E2EE systems, should generally be off by default and only activated via opt-in consent. Obtaining meaningful consent is complex, and requires careful consideration including but not limited to: scope and granularity of opt-in/out, ease and clarity of opt-in/out, group consent, and management of consent over time.

One of my co-authors, Andrés Fábrega and I, discussed the findings of the paper in a piece for Tech Policy Press in February 2025, "Can Bots Read Your Encrypted Messages? Encryption, Privacy, and the Emerging AI Dilemma," which I am republishing below. It predates this plugin by eighteen months, but it feels important to surface it again as many more people decide how much of their private communications to trust to chat bots.

Sign up for Internet Exchange

Feminist perspectives on digital justice and tech

Subscribe

Email sent! Check your inbox to complete your signup.

No spam. Unsubscribe anytime.

Can Bots Read Your Encrypted Messages? Encryption, Privacy, and the Emerging AI Dilemma

By Mallory Knodel and Andrés Fábrega. Originally published in Tech Policy Press.

It may seem like AI chatbots are taking over every digital application, whether we like it or not. You might have noticed more AI note-taking bots in online conferencing platforms, some of which offer end-to-end encryption (E2EE). Then Apple Intelligence plans were announced, promising application redesigns to offer AI features across its phone and laptop operating systems. The latest changes have come from Meta AI’s integration in WhatsApp, replete with “bots nobody wants.”

Any time new features are added to an E2EE messaging app, it raises concerns about privacy and security. So, what concerns are raised by the addition of AI bots? How can we evaluate those concerns? As AI becomes more embedded into encrypted services, is it possible to resolve the tension between the privacy users expect from E2EE and the data access needed for AI functionality? With our colleagues at Cornell and NYU, we set out to answer these questions.

We uncovered several facets of this question from both a technical and legal perspective and published a paper laying practical recommendations for E2EE messaging platforms and regulators. It’s also important that we outline the practical solutions and recommendations for the public. You can read the full preprint paper here.

Background

Online messaging systems that allow communication between users (such as iMessage, WhatsApp, and Signal) are intermediaries to every communication between users. End-to-end encryption (E2EE) is a standard secure communication system that is designed to ensure that only the sender and the intended recipients can read communications between them. Messages that are encrypted are called “ciphertexts,” and content that is not encrypted, even if it’s an image or video, are called “plaintexts.” The core requirement of private and confidential messaging is that the service provider (and any other third parties) cannot read these communications.

AI assistants are programs designed to interpret everyday language and perform computational tasks. Today’s AI assistants are able to handle a wide range of tasks, including text analysis, content creation, code generation, language translation, and more. At the core of these technologies are programs trained on data to identify patterns, e.g., large language models (LLMs) such as OpenAI’s ChatGPT, Google’s Gemini, and Meta’s Llama, which process complex inputs (queries) and provide contextually relevant responses.

Putting this together, it seems impossible that an application with no access to message content can initiate AI processing on that same message content. To introduce AI in E2EE, the strictness of E2EE is widened in two dimensions: what is an “end,” and where is the end.

It would not be a strict violation of E2EE system design if you were to copy and paste your messages into a chatbot (though you might be violating the norms of confidentiality and privacy in the context of the conversation). If an application were to facilitate that for you, however, it would need to guarantee that the processing happens in a way that preserves E2EE, such as processing on the end—your device—and not on another computer, such as the application servers. Apple Intelligence has proposed the use of “trusted execution environments” (TEE) as the way to keep private the data used for AI training and processing.

However, a key finding in our analysis is that TEEs are insufficient to achieve the strict confidentiality and privacy guarantees of E2EE. But first, let’s discuss what this looks like in practice:

How AI Interacts with Your Encrypted Messages

AI features—such as message summarization, smart replies, and chatbots—are being seamlessly integrated into a wide range of applications with the goal of enhancing user experience. However, AI models critically require access to vast amounts of plaintext user data to power these tools. There are two main ways in which AI features interact with application data. First, they receive user data as part of queries during regular feature usage. For example, a message summarization tool receives as input a list of sent and received messages and outputs a summary of these. Second, user data is generally used to continuously train and refine the AI features. For example, data could be used to fine-tune models and improve their general performance, personalize models for the usage patterns of individual users, etc.

These considerations raise significant concerns for integrating AI features in E2EE applications. Processing of user content—during regular feature usage and model training—could expose sensitive user data to the parties who own the AI models. While some lightweight features can be implemented with smaller models that live on end-user devices (and thus, all data is processed locally), other features require offloading user data to more powerful models on the application servers. This is directly in tension with the strong security promises of E2EE applications, which require that no user data is visible to third parties.

There are a number of privacy-enhancing tools that attempt to address these issues and allow cloud-based AI models to process user data while protecting it from model owners. However, these tools offer varying degrees of security, not all of which offer the same strong privacy guarantees of E2EE. It is critical that any adopted solution is compatible with E2EE, and that the models' processing of user data does not undermine users' privacy expectations. Unfortunately, none of the existing technologies that are compatible with E2EE security, such as fully-homomorphic encryption (FHE), are yet practical solutions since they currently cannot efficiently evaluate the large models used in AI applications. On the other hand, more practical approaches, such as hardware-based solutions (e.g., storing models inside TEEs), do not meet the strong confidentiality guarantees of E2EE, and raise additional security considerations. While these represent a substantial privacy improvement over plaintext processing and may be an appropriate solution for other contexts, they are not suitable for E2EE environments.

Furthermore, even if an AI feature could somehow process user content in an encrypted manner (e.g., if FHE were to become practical for this task), training AI models with E2EE data raises an additional security concern: it is well-known that AI models often inadvertently “memorize” training data, which can lead to reproduction of this data during model responses, or even deliberate extraction by actors who can query the model in the form of “adversarial attacks.” So, even if training is performed privately, E2EE data could be exposed to other application users who can query (but not observe) the model. While certain technologies address memorization and adversarial attacks (e.g., differential privacy), these do not meet the strong security of E2EE.

"Whose Bot Is It?" The Tension Between AI, E2EE, and Ownership

We need to address the key tension: users often treat AI assistants as personal tools, but these bots belong to corporations that control data access. Especially in the context of E2EE messaging, we risk treating encryption like one more feature that might contraindicate the use of AI, but it’s more than that.

Users might expect their data to remain private when communicating with these bots, especially since platforms advertise E2EE features. However, driven by business incentives, we are seeing a trend away from privacy and towards the use of novel user data—some of it from messaging platforms using E2EE—to train AI models, sometimes without explicit user consent. This practice would not only undermine the privacy protections that E2EE is meant to provide, but also puts at greater risk the massive privacy gains made over the last decade.

At the same time, AI assistants aren’t very good (yet), and many people don’t want them. Like dark patterns, this just adds to the number of repetitive failures in corporate tech that have desensitized our intuition about real measures of performance when applications are working for us and not companies.

Practical Solutions and Recommendations to Regulators and Platforms

Our technical and legal analysis is meant to inform the design and implementation of AI in E2EE platforms so as to preserve user privacy and expectations of confidentiality. Verbatim are the recommendations based on our findings:

1. Training. Using end-to-end encrypted content to train shared AI models is incompatible with E2EE.

2. Processing. Processing E2EE content for AI features (such as inference or training) may be compatible with end-to-end encryption only if the following recommendations are upheld:

1. Prioritize endpoint-local processing whenever possible.

2. If processing E2EE content for non-endpoint-local models,

1. No third party can see or use any E2EE content without breaking encryption, and

2. A user’s E2EE content is exclusively used to fulfill that user’s requests.

3. Disclosure. Messaging providers should not make unqualified representations that they provide E2EE if the default for any conversation is that E2EE content is used (e.g., for AI inference or training) by any third party.

4. Opt-in consent. If offered in E2EE systems, AI assistant features should generally be off by default and only activated via opt-in consent. Obtaining meaningful consent is complex and requires careful consideration, including but not limited to the scope and granularity of opt-in/out, ease and clarity of opt-in/out, group consent, and management of consent over time.

Similarly, regulators and platforms can make design decisions to mitigate some of the privacy challenges posed by AI features to E2EE applications, such as:

Opt-in Features: AI features should be off by default and only activated via explicit opt-in mechanisms. Each individual feature should have a separate opt-in mechanism, with unambiguous disclosure notices of what each feature entails. Once activated, users should be able to subsequently turn off AI features if they desire.

Privacy Settings and Granularity: Messaging services should provide granular privacy settings that let users control what specific data is used for AI features and how much is stored or processed.

Ease of Setting Adjustment: AI-related settings (such as turning off AI features or adjusting data usage policies) should be easy to find and navigate. There must be a low barrier to toggling off.

Clear Disclosure: Messaging services should be transparent about when and how AI interacts with encrypted messages, including the precise level of security offered by their systems. This includes specifying whether AI features process user data in ways that provide weaker privacy protections than E2EE (e.g., using trusted hardware). These disclosures should be clearly and prominently displayed. Relatedly, companies should adopt a policy of over-disclosure, ensuring that users are fully informed about data usage even when interacting with AI.

Data Ownership and Access: Services should clarify who owns and controls the data AI uses, ensuring users understand that they are interacting with corporate-owned models and not private personal assistants.

We recommend that regulators consider mandating these five practices and that platforms take proactive steps to implement them.

Practical Solutions and Recommendations for the Public

Aside from informing technologists, companies and regulators, it’s important to provide actionable steps the public can take to protect their privacy in the era of AI and encryption. Based on what we know about how companies like Apple and Meta plan to integrate AI into applications that have promised privacy, here is what anyone can do to help maintain their privacy and confidentiality:

Choose OS-level app permissions carefully: Device-wide AI capabilities like Apple Intelligence mean that you need to be aware of which of your applications interact with AI features.

Review App Settings: Regularly check the privacy settings on your applications. If you're concerned about privacy, turn off AI-based features like message summarization or smart replies.

Be Aware of What You’re Sharing: Be mindful of the data you share with AI services, especially personal or sensitive information that could be used for training or other purposes. When applications tell you they might use your data for training AI, believe them. Passwords, contact information, and a wide variety of sensitive information might end up in an AI model and out of your control.

Beware of Opt-in Conditions: If you choose to invoke MetaAI or Apple Intelligence features in a private or confidential setting, be sure you understand the limitations– is it for all chats? Is it forever?

Talk to Your Contacts: If you are having sensitive conversations over E2EE services, have a conversation with relevant contacts, and make sure they aren’t inviting bots to the conversation.

Conclusion

AI features are being developed at a rapid pace, raising significant security risks for users of E2EE applications. It is crucial that AI innovation does not come at the expense of user privacy, and that the strong protections expected from E2EE applications are maintained. Absent perfect technical solutions, service providers should inform and empower users to navigate the interplay between AI and privacy, with transparent disclosures of how data is processed, user-friendly consent mechanisms, and granular controls over how data is used.

Love IX? You can help keep us going!

If you value our work, there are lots of ways you can help support us. Including some that don't cost you anything!

Subscribe. If you're not already a paid subscriber, that's the simplest way to support independent writing about technology, human rights and the internet. Until the end of the month, we're offering £20 off annual subscriptions.

Make a tax-deductible donation. Internet Exchange is a project of Exchange Point Institute, a 501(c)(3), so you can support our public education work directly.

Get your employer to match it. If you work in tech — or anywhere with an employee giving program — search Exchange Point Institute in Benevity. Many companies match employee donations, so your gift could be worth twice as much (or more).

Fund or hire us. We also take on funded projects and fee-for-service work in technology, human rights, research, communications and public education.

Sponsor a newsletter. We accept sponsorships from values-aligned organizations working in public-interest technology, digital rights, open standards, or related social impact areas.

However you do it: help us bring the conversations shaping the internet into the mainstream.

Support the Internet Exchange

If you find our emails useful, consider becoming a paid subscriber! You'll get access to our members-only Signal community where we share ideas, discuss upcoming topics, and exchange links. Paid subscribers can also leave comments on posts and enjoy a warm, fuzzy feeling.

Not ready for a long-term commitment? You can always leave us a tip.

Become A Paid Subscriber

From the Group Chat 👥 💬

This week in our Signal community, we got talking about:

The US State Department designated Autistici/Inventati (A/I Collective) a terrorist organization. A/I is a volunteer-run collective, founded in 2001, that provides free services like email, web hosting and the Noblogs platform to a large antifascist and anti-capitalist user base. This comes just over a month after Secretary of State Marco Rubio hosted representatives from more than 65 countries at a conference to rally global opposition to ‘left-wing terrorism,’ and vowed more designations would come “soon.”

What seems to frustrate the State Department most is that A/I's services are built around data minimisation. Its fact sheet laments that A/I's users remain "anonymous, untraceable, and beyond the reach of the law." The designation has no legal force in Italy, but it could still prove perilous for A/I's members living under a far-right government.

After the announcement, group members expressed concern for collectives that also provide online communication tools for people and groups working on liberatory social change. The politics of Trump and Rubio are threatening the principles of independent internet providers, just as US Big Tech is consolidating power.

You can read A/I’s statement about the designation here.

ICYMI, in a related story Germany's state interior ministers urged the federal Interior Ministry to pursue a full ban on the activist platform indymedia.org through website seizures, takedown orders, and network blocks, plus travel limits for suspected left-wing extremists. (DE) https://de.indymedia.org/node/749687

Want to join the group chat? Become a paid subscriber.

This Week's Links

🚨

Stop press! Do you enjoy our links? Links are now available to paid subscribers only. Become a paid subscriber today.

This post is for subscribers only

Become a member to get access to all content
Subscribe now

# preview.content
OpenAI’s new iMessage plugin for ChatGPT lets users search their messages and draft and send replies through the chatbot on their desktop. But should they?

# summary
OpenAI’s new iMessage plugin for ChatGPT lets users search their messages and draft and send replies through the chatbot on their desktop. But should they?