The sad state of personal data and infrastructure [see within blog graph]

Note: for clarity, I will use 'service' to refer to anything holding your data and manipulating it, whether it's a website, phone app or a device (i.e. not necessarily something having an online presence).

On one hand, in 2019 things are pretty great. For almost anything you wish to do on your computer or phone, you can find several apps, platforms and ecosystems that will handle your task in one way or another.

On the other hand, typically, once the service has your data it's siloed and trapped. You are completely at the mercy of service's developers and management.

• Within the same ecosystem (e.g. Google/Apple/Microsoft) you might get some integrations and interactions if the company spares them. Apart from these, integrations are virtually non-existent.

  We have so much data, yet it just sits there doing nothing.

  Now and then some startup pops up that connects together couple of APIs for a fee. I don't want to pick on startups but typically it's something trivial like displaying calories consumed from your food tracker app on the same plot as calories burnt from your fitness tracker. Trivial is okay, and I do acknowledge it's way harder to implement than it looks (I even explore why later). The sad reality is that as a user, you're lucky if you use the right kind of fitness tracker that the service supports, and you agree with their analysis methodology. Otherwise, sorry!

  There are also services like IFTTT which offer pretty primitive integrations and also require cooperation from all parties:

  • My Heroic and Lazy Stand Against IFTTT (Pinboard)
  • Google removing Gmail access from IFTTT

• Often UIs have inconveniences (or just plain suck). They are often fine for an average user (aka KPIs) but leave a number of dissatisfied users, who are often the power users.

  In essence, services fully control the way they present information to you.

  Sure, it's a free market, just switch to another/better service, right? Switching to new and unfamiliar tools is cognitively hard enough as it is, but what's even worse is that in most cases you have to leave behind all your old data. You're lucky if you can do some sort of data import/export and if it works properly.

Personal data is in a sad state these days. Let me elaborate.

To be fair, I don't understand how does it not bother you!

To start with, allow me to quote myself here:

I consume lots of digital content (books, articles, Reddit, Youtube, Twitter, etc.) and most of it I find somewhat useful and insightful. I want to use that knowledge later, act and build on it. But there's an obstacle: the human brain.

It would be cool to be capable of always remembering and instantly recalling information you've interacted with, metadata and your thoughts on it. Until we get augmented though, there are two options: the first is just to suck it up and live with it. You might have guessed this is not an option I'm comfortable with.

The second option is compensating for your sloppy meaty memory and having information you've read at hand and a quick way of searching over it.

That sounds simple enough but as with many simple things, in practice you run into obstacles. I'll give some I've personally been overcoming as examples:

• convenience of access, e.g.:
  • to access highlights and notes on my Kobo ebook I need to actually reach my reader and tap through e-ink touch screen. Not much fun!
  • if you want to search over annotations in your PDF collections… well good luck, I'm just not aware of such a tool. It's actually way worse: many PDF viewers wouldn't even let you search in highlights within the file you're currently viewing.
  • there is no easy way to quickly access all of your twitter favorites, people suggest using hacks like autoscroll extension.
• searching data, e.g.:
  • search function often just isn't available at all, e.g. on Instapaper, you can't restrict search to highlights. If it is available, it's almost never incremental.
  • builtin browser search (Ctrl-F) sucks for the most part: it's not very easy to navigate as you don't get previews and you have to look through every match
  • sometimes you vaguely recall reading about something or seeing a link, but don't remember where exactly. Was it on stackoverflow? Or in some github issue? Or in a conversation with friend?
• data ownership and liberation, e.g.

  • what happens if data disappears or service is down (temporary/permanently) or banned by your government?

    You may think you live in a civilized country and that would never affect you. Well, in 2018, Instapaper was unavailable in Europe for several months (!) due to missing the GDPR deadline.

  • 99% of services don't have support for offline mode. This may be just a small inconvenience if you're on a train or something, but there is more to it. What if some sort of apocalypse happens and you lose all access to data? That depends on your paranoia level of course, and apocalypse is bad enough as it is, but my take on it is that at least I'd have my data :)
  • if you delete a book on Kobo, not only you can't access its annotations anymore, but they seem to get wiped from the database.

As you can see, my main frustrations are around the lack of the very basic things that computers can do extremely well: data retrieval and search.

I'll carry on, just listing some examples. Let's see if any of them resonate with you:

Things I listed above are frustrating enough as they are. There is another aspect to this: your data is slipping away.

Privacy concerns are important and it's understandable when people are pissed about services keeping hold of their data instead of properly wiping it.

However, oftentimes the opposite is the case and you find that your data is gone or very hard to access:

• Google Takeout data, that is, all your browser activity, Youtube watch history, etc., are only kept by Google for few years

  If you only began exporting it today, chances are you've already lost some of your history.

• Chrome browser deletes history older than 90 days
• Firefox browser expires history based on some magic algorithm
• Reddit API limits your requests to 1000 results only

• Twitter API would only give you 3200 latest tweets

  You can get the rest of your tweets via manual export, but then you'll have to integrate two different ways of accessing data.

• Monzo API only allows fetching all of your transactions within 5 minutes of authentication.

  I understand that it's a security measure, but my frustration still stands.

The problems above are sort of technical and in theory, can be solved by some engineering. There is another side to vanishing data:

• information is generally rotting away from the Internet

• comments/posts/tweets you've interacted with get deleted by their authors

  While people have the right to delete their data from the Internet, arguably it doesn't extend to derived content like comments or thoughts that you had on it.

And a bit more:

• Jawbone UP has gone bust

  In July 2017 Jawbone announced[73] it would liquidate its assets. Since the app is still available for at least some phones (Android) and the servers seem to be running, it is unclear who has access to collected personal data.

  sweet. In addition, the API doesn't work anymore either, so if you haven't been exporting data, it's basically gone.

• 'My GitHub account has been restricted due to US sanctions as I live in Crimea'

• HN: Github blocked me and all my libraries

  Ok, the author apparently violated Github's code of conduct, and Github has the right to ban him. Does it really require shadowbanning the code and other people's comments on issues though? Would you be happy if your contributions were gone without any trace?

• Verizon/Yahoo Blocking Attempts to Archive Yahoo Groups

  This one is particularly bad.

If you consider your digital trace part of yourself, this is completely unacceptable. But sadly it's happening all the time. You can't rely on third parties to keep it safe.

I want all these inconveniences somehow solved, but I live in the real world and it's not gonna magically happen. So let me be more specific: I argue that one major reason these tools and integration I want don't exist is that people don't have easy uniform access to their data in the first place.

"Easy" is used here in two senses:

• easy for humans to look at and browse through

  This bit is hard in practice as (typically) the more machine friendly something is, the less human friendly it's.

• easy for programmers to manipulate, analyze and interact with

  Let's concentrate on this part for now. If this is solved, it automatically enables programmers to develop human-friendly tools. In addition, playing with data and building tools on it feels fun as it is (unlike getting the data).

So how would 'easy access to data' look in an ideal world? Let me present you my speculations on it, and I would be happy to hear your opinions on it!

I want an API that I can query and get any of my personal data. Ideally, it wouldn't really matter where the data is and it could be a web API.

Realistically, as of today, the easiest way to quickly access your data and more importantly, play with it, is when it's already on your filesystem.

As you've probably noticed, it's almost never the case that you have your personal data locally at hand. You need to spend extra effort to achieve this.

Hopefully we can agree that the current situation isn't so great. But I am a software engineer. And chances that if you're reading it, you're very likely a programmer as well. Surely we can deal with that and implement, right?

Kind of, but it's really hard to retrieve data created by you.

Recommended soundtrack for rest of the section: The World's Smallest Violin, playing for us software engineers.

At first glance it doesn't look like a big deal. It's just data, right? Every programmer should be capable of getting it from the API, right?

This is until you realize you're probably using at least ten different services, and they all have different purposes, with various kinds of data, endpoints and restrictions.

Even if you have the capacity and are willing to do it, it's still damn hard.

You're gonna have to deal with the following problems:

The way I see it, ideally the service you're using provides you with:

• a data mirror app

  Best case scenario is if the service is local-first in the first place. However, this may be a long way ahead and there are certain technical difficulties associated with such designs.

  I'm suggesting a data mirror app which merely runs in background on the client side and continuously/regularly sucks in and synchronizes backend data to the latest state.

  Ideally this would be exactly the same state the backend uses, although in practice it would be hard from efficiency considerations (e.g. it's faster for the backend to keep data in the same database instead of separate databases for each user).

  It shouldn't be too resource demanding for the backend, e.g. data sync via push notifications basically already does that, but in an even less efficient way.

  Data mirror app should dump data into an open machine-friendly format like json/sqlite database/etc.

  This solves:

  • authorization: however tedious it's to implement, can be handled by the service's developers.

    They can make it as secure as necessary (e.g. 2FA/etc), and it's okay as long as you have to log onto it only once.

  • pagination/consistency/rate limiting: non-problems, considering it's easier for the service's developers to correctly implement incremental data fetching
  • error handling: also the developers' responsibility. They would be better aware of which situations are programming bugs and which have to be handled carefully
  • documentation and discovery: hopefully developers are better suited to keep their internal representations and exports consistent (even incentivised as it allows for less code to be written)
  • backups: will still have to be done by external means, but the task is massively simplified: you just need to point your backup tool at your data storage

• minimalistic data bindings in some reasonable programming language that represent all of this data.

  Hopefully, specific language doesn't matter, it's a simple task to map data from one programming language to another.

  This solves:

  • parsing: developers know better how to get it right; in addition the code can potentially be shared with the backend
  • abstract representation: would massively lower the barrier for integrating and interacting with data
  • offline: if you have all data locally you've got efficient access without latency and need for extra error handling

That's perhaps a naive and oversimplified view. But to be honest, we're so far away from that that even some small steps towards would be quite a progress.

These suggestions would decouple data from the UI and let the community develop better tools for consuming and working with it.

Of course, I'm not expecting someone to come and implement all of this for me. I could start some sort of movement to demand it from services and platforms, but I hardly see myself as a good fit for that role.

Instead I've put effort into exporting, integrating and utilizing my data on my own according to the suggestions I formulated. Putting this in writing helped me motivate and summarize many technical and infrastructural decisions.

I'll be describing my setup in more detail in future posts, however here are some bits and pieces:

• jborichevskiy: Digital Tools I Wish Existed (HN)
• IndieWeb wiki on silos
• Ink & Switch: Local-first software: You own your data, in spite of the cloud
• The Upspin manifesto: On the ownership and sharing of data
• The old Internet died and we watched and did nothing (HN)
• NetOpWibby: A Personal API (HN)

Attempts of resolving this sad state:

• Solid, web decentralization project by Tim Berners-Lee, WWW pioneer and the director of W3C

  The project "aims to radically change the way Web applications work today, resulting in true data ownership as well as improved privacy" by developing a platform for linked-data applications that are completely decentralized and fully under users' control rather than controlled by other entities.

I'd be interested to know your opinion or questions, whether on my motivation, or particularities of my suggestions or implementation.

Let me know if you can think of any other data integrations you are missing and perhaps we can think of something together!