Data Transfer Project — What is it
A few years ago, several companies wanted to move the state of the art forward for data portability. Wanting to move further together than they could separately, they started collaborating on an Open Source project that allowed them to make use of each other’s contributions. That project became the Data Transfer Project (DTP) and is now supported by an independent non-profit organization, the Data Transfer Initiative (DTI).
Today, a data exporter can run DTP on a server inside their service, and provide high quality and scalable user-initiated data transfers to other services. The DTP repository includes common data models and a library of adapters to connect data objects to the services APIs that are providing or receiving those data objects, as well as a generic adapter for additional flexibility.
How does it affect me as a user?
If you have personal data on one of the companies running DTP, such as the members of DTI, you may be able to use DTP to transfer a copy of that data to other platforms. The ability to move data directly from one service to another is often housed alongside the features that allow data export – a download of data to you, to a personal computing device. In many cases, data exports are often used to achieve data portability by allowing users to download content that they then upload to a different platform. DTP represents a higher standard of secure and simple service-to-service delivery. These services listed below offer both exporting and direct service-to-service porting, depending on content-type. Because that availability may change over time, we list the umbrella data access/export/porting services here:
- Google Takeout
- Apple Privacy
- Facebook’s Export your information
If you do port your content from one service to another service, you may be using DTP without knowing it. The goal of the DTP contributor community is to make data porting work better (more complete transfer, fewer steps, faster results) and that’s often invisible to the user.
Who contributes? Can I contribute?
The people and organizations who run DTP and contribute adapters or core code are primarily services with large amounts of user-contributed content or data. Users of these services come to the place where their content or data is stored and processed to ask for that data to be provided in a specific format or to a specific destination, whether that be for purposes of backup or experimentation or moving to a different service.
As a result, it’s engineers at these services who have built the majority of DTP’s functionality, supported now by staff at DTI. Because DTP is open source, they can benefit from each other’s work. A new adapter that can send data to a new destination can be used by all the exporters for whom sending to that destination makes sense.
Open Source contributors are often “scratching their own itch” famously, and solving their own needs while supporting the Open Source project. DTP solves the needs of data exporters (and data destinations) involved in mass data portability. Thus, if you’re not working on data portability at a data exporting platform, it may be hard to contribute to DTP! Still, if you believe you have an angle, please elaborate on it (via github issues, most likely); the contributors are all friendly.
What is the future for DTP?
DTP already provides great value where it’s running and in use, but work continues. Some of the work currently or recently in-progress includes:
- New technology verticals and new partners for existing verticals, notably photos
- Improved service quality including error handling and managing interruptions
- Telemetry improvements
If you think that your company or service should be running DTP, we have an intro for incoming contibutors especially to help understand what service components will be affected and what kinds of engineers within the organization will need to be involved.