IETF Work Related to Personal Data Portability
I just returned from one of the thrice-yearly IETF in-person meetings. IETF standards are, sometimes quietly, a major part of data portability. This is an overview of relevant IETF work today, especially documents in active Working Groups (WGs).
The IETF is responsible for major parts of the Web API stack with HTTP and JSON the most visible, stable and widespread components. Below HTTP, the IETF is responsible for lower layers like TCP/IP and QUIC among many others. Most portability APIs use HTTP+JSON on top of some network/transport stack, providing their own data formats for the personal data being transferred. For purposes of this piece, we won’t worry about the lower-level components and the work going on to make those components faster, more secure or more manageable.
The work that goes on at the top layer of IETF protocols is often the most relevant for data portability. Some standards are only relevant for one topic (for example, if your personal data isn’t a calendar then CalDAV is not relevant). This article lists work relevant to implementers of:
- Personal productivity servers, clients and helper apps
- Text, voice and video chat or conference platforms
- Social media feed clients or services
- Personal data services implementing their own APIs, data formats and authorization scopes
Some work is general-interest, especially on OAuth, so jump ahead if your data portability subject matter doesn’t match the above list.
Updating the old guard
There is a family of related standards that have been used for personal data portability for decades: the IETF standards for email, events, tasks and contacts. Let’s call these the personal productivity standards, the widely deployed originals of which are:
- IMAP and Internet Message Format for mailboxes and mail messages
- CalDAV and iCalendar for calendars, events and tasks
- CardDAV and vCard format for contacts
Did you think of these as solutions for data portability? I’ve changed email providers by using IMAP twice, first telling an IMAP client to copy all my email to my laptop, then configuring that client to sync all that email history to a new email provider. This relies on well-established interoperability on email formats, mailboxes, and metadata like read/unread and draft/sent. Especially because there are two transfers and three storage implementations involved, these standards have to be pretty solid for all the details and oddities to be transferred correctly.
Alternately, some folks are comfortable providing a new email service with their old email service password. After all, the decision to upload all email history to the new service requires nearly the same trust in it anyway. Then the new service uses IMAP or JMAP to fetch email from the source service until it’s all synchronized. (There’s also the variation of using a third service like Audriga and providing credentials to both source and destination, then it can use IMAP or JMAP.)
CalDAV and CardDAV are used the same way, either server-to-client-to-server, or sharing access credentials so that the destination server can act as a client. The iCalendar formats are typically used in export/import data portability flows, because the data volume is more manageable.
These personal productivity standards are worth reviewing because of work in several groups that may affect data portability.
- The JMAP WG is completing some refinements to a mature set of protocols and formats for accessing productivity data using HTTP and JSON. Since the Web kind of runs on HTTP and JSON these days, and JMAP was designed to unify and simplify a bunch of prior work, JMAP lowers the bar for supporting interoperability and portability for many kinds of personal data. More folks should look at implementing JMAP.
- In the mailmaint WG, we are working on PDPArchive, a consolidated export/import format that can be used to do data transfers including mailboxes, mail, calendars, tasks, events, contacts and address books. This saves users from exporting and importing calendars and address books one by one, and adds other user experience and fidelity improvements.
- The mailmaint WG is also working on hooking OAuth up to IMAP/SMTP/JMAP, which is useful and relevant to data portability for a number of reasons - but this should be discussed in context of all active work on OAuth which I’ll tackle below.
One more well-established personal data standard, not technically “productivity”, is Atom. Atom is described as a feed standard for clients to follow a feed like a blog or newsletter, but it’s also commonly used for the owner to move their feed content to a new system through export/import steps. Atom does not have active work but it felt incomplete not to mention it.
OAuth
Many are unsatisfied with the approaches described above for personal productivity data. Transferring data twice may be error prone, slow, and in many cases users don’t have the storage to hold all the email they’ve received for a decade or three. New providers would like to be able to make this process easier for new users by handling transfers server-to-server. But security folks are definitely not happy with sharing passwords to enable server-to-server because this gives ALL permissions to the destination (or 3rd party transfer service), not just content access. Everybody is turning to OAuth to provide scoped delegated authorization – but there are a few gaps.
OAuth is broadly deployed for Web services and their service partners, where a user logged into a Web service can authorize access to their data to the partner service. This works by pre-establishing a client_id and client_secret that the Web service and partner service know, to protect the user from rogue services pretending to be trusted services. But manually established secrets don’t scale well or work well in some situations.
- OAuth for client access
The goal here is to use OAuth with the kind of client software we run on our laptops and phones. OAuth would seamlessly allow partial authorization rather than full account access, keep passwords even more secret, and support two-factor authentication. OAuth replacing SASL would make a service’s security surface more consistent and manageable between Web access and client access.
The blocker to using OAuth with user-installed copies of software is that the client_id/ client_secret solution doesn’t work across many installs. The developer of the client software can’t distribute the same client_id to all, or it would be discovered and used by hackers. Nor is it easy to prearrange client_id and client_secret and then deliver matched pairs to every client and every service those clients need to access.
Some solutions are being explored here:
- Using a fixed client_id in clients without short-circuiting the protections that client_id offers in server-to-server cases
- Using certificates instead of secrets, like DPoP or DSig
- Dynamically configuring and exchanging client_id and client_secret
- Reducing barriers to new parties requesting API access
In an ideal rich and functional data portability ecosystem, a service provider would not have to go through manual applications to get a client_id and client_secret with every other service it may port data from. This is an NxN-scale operations and configuration challenge. At the same time, many more people are thinking of reducing barriers to AI services and AI agents requesting authorization for personal data access, for exactly the same NxN scale and overhead issues, so lots of folks are tackling this problem from SEVERAL different directions.
- DPoP or HTTPSIG using signatures instead of client_secret
- Dynamic Client Registration Protocol - still NxN shared secrets but automatically set them up
- AAuth
-
Tying OAuth to a layer with certs: mTLS
Some HTTP-based protocols like JMAP already support OAuth and will be able to benefit from these new extensions. Still, we’ll need some consolidation. It can get complicated if different services make different choices among all the options above.
- Better defined scopes and permissions
Again, AI services and AI agents are driving innovation and elaboration in this space. OAuth scopes are a black box, and although the resource server offering the scope can define any scope they like, and also walk the user through additional limits on the scope via Web UI, automation and interoperability are hard to do with that approach. For example, if a user wants to limit a booking agent to $1500 on a travel booking site, that choice of $1500 would have to be set manually each time and can’t be interoperably reviewed or set by tools such as agent harnesses.
- OAuth RAR and extensions to RAR
- Scoping and motivating new work
As you can see, work on OAuth is exploding.
Brand new work
There is some newer work that is application specific.
VCon defines formats for virtual conversation transcripts and metadata. People wishing to port their conversation history from one platform to another, or wishing to make a reference archive of conversations, can use VCon.
ATP, also known as AT Proto, used by the BlueSky network, has a WG and is laying groundwork. It’s an explicit goal of the group to allow users to move their post/follow history between servers and not lose their connections (the way a Twitter user leaving Twitter would have had to). Note that in parallel W3C is working on ActivityPub with the same goal.
XMPP is being discussed at IETF again despite having a few years of development outside the IETF. It’s too early to say if a WG will be formed but folks may be interested in helping form one.
Things you won’t find…
Since personal productivity data formats were established twenty years ago, despite their success, we don’t tend to see more IETF standards for personal data.
Why is there no access protocol for photo albums in the cloud, or playlists in the cloud?
- File formats for images, videos and music are defined in other standards organizations (like the ISO for JPEG). Defining a protocol in IETF would require collaboration with folks who are used to operating elsewhere.
- IETF hasn’t been a great place to define data formats (e.g. for playlists). Folks seem to take semantic arguments to extremes, less willing to just accept a working proposal, even when a choice such as a field’s name just doesn’t matter that much for interop.
- It’s easier for implementers to host their own API. There’s a lot of consistency about using HTTP+JSON even if the data format for each photo album site is different.
This is OK. The biggest barrier to interoperable data portability today is trust, not data formats. It’s even more true in the last two years as AI has made it easy to use JSON data, often with the help of JSON Schema defined APIs (as Juan Cruz Viotti has pointed out, AIs are very good at JSON and JSON Schema). It’s easier to write or have AI write translators for slightly different photo album or music playlist formats than it is to standardize on a single format. That’s not to say we won’t see actual or defacto standards for those data types, because real stability, performance and perfection in the edge cases comes with standardization.
Takeaways
- Since trust is the largest barrier to a more functional data portability ecosystem, the OAuth work is important to everybody in data portability.
- New data format standards are more likely to be ad-hoc than IETF standards, although doors are open for folks to bring new data format standards if there is demand.
- AI is driving a great deal of work that will be relevant to non-AI driven personal data portability too.
- Personal productivity implementers who haven’t been engaged for a few years will find a pile of new work to digest.