May 21, 2017

Putting the Memex into a container

Into Docker you go

My main objective right now is packaging what I’m working on into something that I can easily get to beta testers. Last month, I spent a bit of time helping a few friends install the app on their private servers but it’s a fragile setup. Having a single OSX app would be way easier and I’ve been poking around strategies to get the app running in a single installable file. But rewriting a huge part of the API and importers is a huge amount of technical risk.

This week, I experimented with Docker as a simpler, shorter-term solution. I put together a working system using docker-compose and I’m extremely excited about how well it runs. After sorting out a few kinks and initialization issues, I might be able to get a beta out to technical-ish people very soon. I’m pretty sure I could even get the installation process to be driven by an Electron app as long people have Docker installed already.


Some of the importers need a lot of work before they can be set up smoothly for new users but lots will work well right from the start (for example, Chrome history, bash history).

Overall, I’m spending time trying to streamline some of the onboarding slow. Instead of manipulating localStorage to “log in” as a few of you had to do, here’s a slightly slicker login screen:


Fuzzy date ranges

Although the current query system desperately needs a better interface, my hope is that advanced users will always be able to use typed out queries. That said, no matter how many times I have to format a date such as “before:’2017-01-01’”, it never gets easier to type so I added fuzzy date parsing to the querying engine. The before and after filters now support everything that the Chronic gem handles .



I’ve been using Shazam for almost a decade. I love using a tagged song to remember where I was, for instance — human memory is funny thing. Unfortunately, all previous attempts to export this data have resulted in tears. For years, the timestamps on the web interface were hopelessly incorrect (despite many support emails alerting them); more recently, their login system didn’t work because of the way Facebook was connected to my account. I even joined Shazam’s beta group in the hopes that I could barter feedback for customer support but no luck. Alas, after many years of waiting, Shazam launched a new web interface with properly organized data which means I was finally able to complete an importer and imported all 657 of my song tags.

The coffeeshop I was working in while testing the importer used a time machine and stole a mixtape from my high school: 3

The final word about Shazam goes to Peter who gave an amazing talk at Papers We Love last year about how their algorithm works. It’s worth watching


Fifteen years ago, a researcher named Gordon Bell put together a Memex as part of a Microsoft Research project called MyLifeBits. He got an assistant to scan his decades of paper archives, wrote importers for his digital records, and covered himself with wearable cameras, audio recorders, and GPS devices. His team hacked together a windows app as an interface and published lots about the project but it never got commercialized.

What a Memex might look like in 2004: 0

The next time I’m frustrated from working on an API for an importer, I should remember that Bell managed to build a similarly comprehensive project that had a large number of analog sources like the audio of his phone calls and full archives of his tv watching.


He wrote a book called Total Recall about his experiences and I’ve been reading it this week. The first section of the book describes what his team built. The rest explores the implications of a future where everyone has the ability to access a complete history of their lives, a superpower he calls “Total Recall”. Gordon Bell predicts this future will be fully implemented by 2020 (tick tock, tick tock) and is pretty excited about it and how it will empower individuals — it’s a shining example of California Ideology .


In contrast to how people like Bell or Wired circa 2005 predicted things would go, the internet has led to incredible centralization of power, corporate influence, and a surveillance system like none other. But I believe in the promise of technology and hope that my project is a small, decentralizing pushback that can help us take control of our digital lives on our own terms.

Yet another Memex

Finally, if you’re sick of me taking my sweet time releasing a beta, you can use this version of the Memex that my Google Alerts brought to my attention:

© 2021