Building a Memex by Andrew Louis

Being on nodding terms with the past

“What’s the most useful thing you’ve learned from your data,” I often get asked after telling someone about my Memex and the amount of personal history I’ve collected. I suspect the question is more an assessment of my mental health so it should be simple to respond to but I’m always unsatisfied by the answers I give. Maybe this project isn’t actually that useful?

This week, Joan Didion made me feel a bit better about myself. I just finished reading her excellent collection of essays Slouching Towards Bethlehem.

One essay, On Keeping a Notebook is about her relationship with her private notebooks and the chaotic scraps of information she collects. She observes that there isn’t universal consensus on the merits of journal-keeping:

The impulse to write things down is a peculiarly compulsive one, inexplicable to those who do not share it, useful only accidentally, only secondarily, in the way that any compulsion tries to justify itself.

The fragments of information she collects often seem trivial or sometimes barely factual but they always answer the important question of “how it felt to me”, making the habit worthwhile, especially with the passage of time:

It all comes back. Perhaps it is difficult to see the value in having one’s self back in that kind of mood, but I do see it; I think we are well advised to keep on nodding terms with the people we used to be, whether we find them attractive company or not. Otherwise they turn up unannounced and surprise us, come hammering on the mind’s door at 4 a.m. of a bad night and demand to know who deserted them, who betrayed them, who is going to make amends. We forget all too soon the things we thought we could never forget. We forget the loves and the betrayals alike, forget what we whispered and what we screamed, forget who we were.

I think I can actually justify the value of this Memex in a lot of ways but a big personal motivation are these casual encounters with previous versions of myself as awkward as that can sometimes be. Or as Virginia Woolf experienced:

I have just re-read my year’s diary and am much struck by the rapid haphazard gallop at which it swings along, sometimes indeed jerking almost intolerably over the cobbles.

More practically, jumping through old messages is a great way to be reminded of someone you should catch up with. Scanning a list of coffee shop visits is a great way to figure out where to go next. Re-reading a project’s work log is a great way to reprioritize future tasks or be reminded of old ideas. I don’t know if any of these represent a “killer feature” but I can’t imagine living without these habits that my Memex facilitates.

Mapbox

The embeddable Google Maps API is showing its age and it hasn’t received all the shiny new parts of the full Google Maps product. For example, embeddable maps still don’t use the webgl technology that the regular Google Maps uses which means we can’t do smooth zooming or sophisticated overlays.

This week, I took a bit of time to switch the Memex’s mapping display system to use an alternative mapping service called Mapbox which is based on OpenStreetMaps data. Mapbox allows for pretty nifty customizations of how map layers are displayed.

A while ago, I spent a lot of time manually setting up this chart showing bike route frequency and distribution of visits. I’m pretty sure I’ll be able to set it up pretty easily with Mapbox.

I’m mostly excited to use Mapbox’s soon-to-be-released heatmap API. This sort of visualization is definitely not possible to do with Google Maps:

Docker

If there’s one rule of thumb with Docker, it’s this: one process per container. My Memex’s importer system uses cron to trigger jobs to run at particular times and this sort of background daemon process doesn’t fit naturally into the Docker world.

I worked on this area of the OSX Electron/Docker app this week and seem to have something that works. I added another container to the docker-compose configuration with the same setup as the main importer app except the container’s command is running a shell script that writes the crontab from a schedule configuration file and runs cron in the foreground with cron -f. If a user wants to add an importer job or change the schedule, under the hood, it’s just a matter of taking the changes, updating the config file, and restarting the cron container. Seems to work pretty well!

One big problem with using cron for this sort of stuff is that if (when) an importer stalls, cron ends up starting new runs of the same job over and over again and I have to kill jobs manually. But I’m not worrying about this too much right now and am just trying to get to a [buggy] first release.

Git importer

One of the first importers I completed was for commits to Github — the records became a meta stream documenting my work on the Memex. Sometimes the repos we’ve worked on are no longer accessible on Github. This week, I whipped up a quick importer for git activity that reads commits from the filesystem instead of Github. For me, it was specifically so I could import a log of my work from my startup days before our incredible journey at ShopLocket came to an end.

Here’s the final commit from when we turned off the service for the last time.