June 25, 2017

A Memex for dreams

Dreams have always been of great interest to anthropologists and colonizers alike. The Jesuits collected Iroquois dreams in the 17th century. In Tanzania, an anthropologist would go around his village each morning before breakfast to collect dreams before people forgot them. Dreams are of great significance in the Ojibwe culture, acting as protectors to the dreamer or charters for a group; an anthropologist named Alfred Hallowell spent years in northern Manitoba collecting as many as he could record.

Half a century ago, a researcher named Bert Kaplan began an ambitious project to build the largest database of sociological data ever assembled. He collected records of dreams from different anthropologists as well as data from cutting-edge assessment techniques like the Rorschach test. All this data would be packaged and distributed to researchers using a microfilm device called the Readex.

A Readex in action
A Readex in action

The history of this project is told in an excellent book I read this week, Database of Dreams by Rebecca Lemov. It’s an interesting story of the rise and fall of this project, but also on the history of anthropology, culture, and even the idea of objectivity.

Database of Dreams, by Rebecca Lemov
Database of Dreams, by Rebecca Lemov

Technologically, the database project was of the same era as the Memex. Both Vannevar Bush and Bert Kaplan had high hopes for how microfilm could bring libraries-worth of information directly into the desks of researchers.

Lemov also positions the Memex within academic thinking and political theory. The concept of an “assemblage” refers to a cluster of ideas that can end up acting as a system with agency. In this view, the Memex is not just the particular idea of a single inventor working in 1945; instead, it is a group of ideas, goals, and technologies that continues to be potent today. Lemov summarizes:

The Memex was a fantasy machine that was never build and thus never actually existed. Yet history has proceeded as if had and did.

And here we are today! That’s this week’s edition of the Memex Book Club.

A Flickr importer

My life got a lot more interesting when I realized I could meet strangers on the internet with similar interests (note: this is a huge privilege of presenting as a white man on the internet). Flickr was one of the first online communities where I got to experience this.

This week, I wrote an importer for my Flickr activity and archived more than a decade of photos, comments, messages, and favourites. Though this was nowhere near the top of the priority list and few people still use the service today, adding this history to my Memex was emotionally fulfilling.

A screenshot of my Flickr data in my Memex
A screenshot of my Flickr data in my Memex

So many of my early photos were public journal entries, capturing memorable experiences and social interactions that led to close friendships today. Here’s a fun discussion about the electric grid (with a cameo by Wolever , one of our first interactions ever!), now safely archived in my Memex.

The Flickr API is showing its technical age but it was still a breath of fresh air to work with — almost all aspects of the service are accessible through its endpoints, some of which don’t even require authentication. It’s hard to imagine a new photo service launching today and, for instance, freely giving away all its data/metadata of photos tagged in a particular location. I’m reminded of Anil Dash’s great essay The Web We Lost .

Youtube subscriptions / likes

On the other hand, YouTube has retrenched, removing API access to your watching history, the watch later list, and other data.

I’ve had an importer that uses Takeout data but this week, I improved it to use the API to periodically retrieve liked videos and subscription activity.

Youtube data in my Memex
Youtube data in my Memex

Updates on goal of an installable OSX app

I made more progress on getting the Memex running in an installable Electron/Docker app. This week, I got the importer codebase (a separate repo, set up as an ETL system) integrated into the Docker system using docker-compose. The Electron frontend app now knows how to install the importer app and interact with the configuration files for the jobs and their schedules. Next, I’ll be working on getting the frontend to be able to initialize jobs (for example, authenticating with Twitter and setting a schedule for when tweets get imported).


Also, as I went about figuring out how my javascript app could read/write yaml configuration files for the importers, I was reminded of what happens when a programming language has an inadequate standard library: dozens of packages that do almost exactly the same thing:


Consuming the Memex through a calendar

Andrey and I were talking about how dreadful calendar software is (Google Calendar’s interface has barely been updated in a decade) and Andrey shed a tear for Sunrise Calendar, a delightful-to-use exception which was unfortunately gobbled up by Microsoft last year.

As we talked about calendars, he put forward an idea of a calendar integration for the Memex where journal data could be aggregated and put into an ical feed which could be added into most calendar apps. For instance, travel/visit history could be aggregated into a feed so a user could always open up their calendar and look back and see where they were on a given day. Similarly, users could retroactively add events to log what happened on a particular moment of a day. All this can be done in the Memex interface already but its inclusion into a space that people use all the time could make a great entry point for new users.

The one challenge is that I couldn’t provide these calendar feeds in an encrypted way but users might be ok with this for less sensitive data. I will stew on this idea!

© 2021