Sign in Subscribe
privacy and security

Confidential Chats in the Age of AI

Confer is one early example of how to build a private AI chatbot. We should have many more says Mallory Knodel.

Confidential Chats in the Age of AI
Photo by Sue Winston / Unsplash

By Mallory Knodel

As conversational AI moves into messaging apps and everyday workflows, the question is not whether these systems will handle sensitive information, but whether they are built to protect it. Governments, companies and the public gathered in New Delhi for the second AI Impact Summit late last month, and not a panel went by without an expert addressing safety in AI. Indeed data is a design asset and data privacy a design liability. Some emerging projects, including tools like Confer aim to bring stronger confidentiality to AI chats.

In order to confront the paradox of privacy in a system built on data, I and my colleagues at NYU and Cornell released a paper “How to think about End-to-End Encryption and AI: Training, Processing, Disclosure, and Consent.” E2EE messaging systems achieve the greatest confidentiality and privacy. What our paper recommends on the training and processing of data in E2EE environments, and the analysis that supports those recommendations, provides a solid foundation upon which to attempt to resolve the paradoxes and challenges of privacy in an age of AI.

E2EE guarantees that only the sender and intended recipients can read message contents. AI systems, by contrast, require access to plaintext data to function during inference (processing user queries) and companies might be tempted to use novel conversation data for training. We wanted to let users know about these risks, but we also wanted to influence the way these features are being designed. Our recommendations are clear. And so far we think most platforms have taken heed.

Recommendations

  1. Training. Using E2EE content to train shared AI models is incompatible with E2EE. Models can memorize and reproduce training data, and even privacy-enhancing techniques like differential privacy do not meet the strict confidentiality guarantees users expect from encrypted messaging.
  2. Processing. AI features may be compatible with E2EE only if endpoint-local processing is prioritized. If processing must occur off device, no third party can see or use E2EE content, and a user’s encrypted data must be used exclusively to fulfill that user’s request. Not to improve global models.
  3. Disclosure. Providers should not make unqualified claims that a service is “end-to-end encrypted” if encrypted content is, by default, accessible to third parties for AI processing.
  4. Consent. AI features in E2EE systems should be off by default and activated only through meaningful, granular opt-in.

Enjoying this article? Consider becoming a paid subscriber.

If you find our work to source, research, write, and edit the IX valuable, consider becoming a paid subscriber. You’ll get access to our members-only Signal community and be able to leave comments and replies to posts.

For a limited time, we’re offering 50% off an annual subscription. Right now you can subscribe for just $25.

Not ready to commit? You can always leave us a tip!

Become A Paid Subscriber

These are not hypothetical concerns. Consider WhatsApp’s integration of Meta AI: Invoking Meta AI through a search bar or in a standalone chat may fall outside E2EE contexts, but tagging “@MetaAI” inside a private group chat might effectively bring a third party into what users think is a private, encrypted conversation. The interface still looks secure, even though the underlying privacy guarantee has changed. The details for how this is integrated matters, as our paper points out.

Apple’s approach is different but raises related issues. Apple Intelligence relies on what it calls “Private Cloud Compute,” which uses trusted execution environments (TEEs) in Apple’s data centres. As cryptographer Matt Green has noted, TEEs are a meaningful improvement over sending data to the cloud in plain text, but they are not the same as end-to-end encryption. The cloud server – however secure – still acts as an endpoint that can access decrypted data. This may be reasonable for some AI features, but it does not offer the strict confidentiality that E2EE is meant to provide.

That distinction matters because encryption is not merely a feature, it is a security guarantee. Treating a cloud server as an “endpoint” weakens that guarantee, even if the technology behind it is sophisticated.

Which brings us to something genuinely new

Last month, Moxie Marlinspike introduced Confer, an early-stage project described as “end-to-end encryption for AI chats.” Confer encrypts conversations so that only the user can access them. The service cannot read, train on, or hand over user chats because it does not possess the keys.

Confer’s framing is different from ours. Our paper is about maintaining E2EE guarantees in systems where encryption already exists by default. Confer focuses on adding stronger confidentiality to AI chats that are typically not encrypted at all. It relies in part on TEEs and treats a server-side enclave as part of its end-to-end model – a definition we would not adopt.

If Confer were integrated directly into an E2EE messaging app, it could alter the baseline security guarantees of that app. But as a standalone AI chat system, the comparison point is not E2EE messaging; it is today’s standard AI chatbot, where conversations live in corporate data lakes.

Why Confer matters

Today’s AI assistants invite something new: not just queries, but confessions. The conversational interface encourages us to elaborate, to think out loud, to reveal uncertainty and context. We are shown what looks like a private dialogue when in reality, it is a group chat with corporate operators, service providers, future advertisers, and potentially even governments.

Confer attempts to make the interface match the underlying technology. If it looks like a private conversation, it should be a private conversation.

Is it perfect? No. Is it equivalent to strict E2EE messaging? Also no. But it represents a meaningful step toward private AI systems architected from the start to minimize corporate access to user thought.

Privacy-respecting AI is not what we are seeing today. Instead, we are watching the normalization of off-device inference, broad data retention, and training on user inputs by default. If we want AI agents and assistants that work for people rather than platforms, privacy cannot be an afterthought layered onto an extraction model. It must shape the product from the beginning.

E2EE with AI, implemented according to our recommendations, remains the highest bar for confidential communications. But not every AI use case requires that bar. What we DO require are systems that are honest about their guarantees and ambitious about protecting user data.

Confer is one early example of how to build a private AI chatbot. We should have many more.

Human Rights at IETF 25

In two days, Shenzhen will host the IETF community for a week of technical debate on the protocols shaping the internet’s future. IX’s Mallory Knodel will there hosting the Human Rights Protocol Considerations (HRPC) research group, presenting new work on end-to-end encryption and AI to the Crypto Forum Research Group (CFRG), and sharing progress on E2EE interoperability with the Decentralised Internet Research Group (DINRG). See the agenda here.

Internet Society has kindly made a list of sessions and working groups at IETF 25 that may be of interest to people following internet governance, rights, and policy-relevant technical debates.

🚨
Stop press! Do you enjoy our links? In a few weeks, the links roundup will be available to paid Internet Exchange subscribers only. For a limited time, we’re offering 50% off an annual subscription. Normally $50 per year: right now you can subscribe for $25. Become a paid subscriber today.

Internet Governance

Digital Rights

Technology for Society

Privacy and Security

Upcoming Events

Careers and Funding Opportunities

Opportunities to Get Involved

  • Applications are now open for the second US cohort of Snap’s Council for Digital Well-Being (CDWB), a year-long program designed to elevate young people’s voices on online safety and digital well-being. https://values.snap.com/news/applications-us-council-for-digital-well-being 
  • Call for Nominations: Network Leadership Grants. In this call for nominations, the Green Screen Catalyst Fund is looking to support individuals, networks, and organizations who are catalysing action on environmental justice issues across the AI supply chain. Nominations due  March 15. https://greenscreen.network/en/catalyst-fund 

What did we miss? Please send us a reply or write to editor@exchangepoint.tech.

💡
Want to see some of our week's links in advance? Follow us on Mastodon, Bluesky or LinkedIn, and don't forget to forward and share!

Read next

Comments ()