Building a Memex by Andrew Louis

The dangers of Memex'ing

After a relative of mine advanced past the the Awkward Teenage Phase of her life, she poured bleach on her diaries to destroy them so nobody (including herself) would ever have to endure reading them. I told a friend this story and she said she had the same problem but solved it differently: she sealed her high school diaries in a clear lacquer so she could keep them as physical mementos without being able to read them.

I’m reading Margaret Atwood’s post-apocalyptic The Year of the Flood which features an enviro-religious cult called God’s Gardeners. Their members are instructed to live simple lives in tune with nature, not using any technology or even writing anything down. The main character leaves the cult and starts keeping a journal but laters burns it, concluding to herself that the Gardeners were right: “reading someone else’s secret words does give you power over them.” The cult’s leader had warned his followers: “Beware of words. Be careful what you write. Leave no trails.”

This isn’t an unreasonable strategy. Filling a Memex with years of exhaustive personal records isn’t the safest decision. Beyond just being awkward to experience older versions of yourself, having a comprehensive, consolidated record of your life is a huge liability if people other than you gain access to it. But I’ve decided that the ability to have my personal data around for search or introspection is the worth the risks.

Someone else who made a same decision is my friend Michael. This week, he showed me a personal dashboard he built for himself to keep himself accountable to some personal goals and habits. Here’s a screenshot:

The cards regenerate each day and the goal is to fill all of them out by the end of the day. I love this combination of qualitative data (sleep records, exercise logs) combined with qualitative, journal-like notes. A hope for the Memex project is people like Michael can build their own custom frontends for things like this without having to build a database and API system from scratch.

OSX App update

Assembling so much personal data into a single app is a huge risk and means the “move fast and break things” attitude is the wrong approach. In particular, I don’t want to host unencrypted user data in a shared cloud service; hence my current focus on packaging the Memex into an installable OSX app with data stored locally.

I’ve made good progress on packaging the frontend into an Electron app with the API and importers running in the background with Docker. The whole system can now run from scratch through a single app file with only a few things that have to be configured manually (but working on interfaces for these). This week, I finished a backup/restore system for the database that can be invoked from the app’s main menu:

The Electron app (a combination of Ember and Node under the hood) controls the API through a bunch of shell scripts that I created to interface with Docker. I’ve been doing a lot of work to harden my proof-of-concept scripts. If you find yourself with similar work to do, I found this guide about defensive Bash programming to be quite helpful.

Mapzen path snapping addendum

In last week’s newsletter, I mentioned the new Mapzen map matching API to snap GPS trails to streets. In working with it a bit more this week, I came across their explanation of how the algorithm works . It’s a pretty neat use of a Hidden Markov Model, used to figure out a likely sequence that gets us from one point to another. A graph is constructed (nodes are points on a road segments; edges (weighted) are possible transitions from one point to another) and the Viterbi algorithm is usedto figure out the most likely sequence.