June 11, 2017

The Memex becomes installable

After last week’s depressing experience with javascript, I’m pleased to say I only had to rm -rf node_modules twice this week. I even feel I might have learned a thing or two about javascript and Electron.

A recap of the short-term goal: I want an installable OSX app that I can distribute to people to start beta testing the Memex. It’ll package both the frontend and the API and importer system (running in the background through Docker).

I mainly focused on making Electron talk to Docker this week. Below is a clip of the installation process running. Here’s what it’s doing:

  • check that Docker is installed and running
  • clone the API repo into the “~/Library/Application Support/Chronobase/“ directory that’s already been auto-generated by electron
  • run docker-compose build in this directory; this creates four containers (app, background worker, postgres, redis) and runs the app and database initialization
  • creates an access token that the Electron app saves and uses for authentication

The design is rough but it’s getting pretty close!

What I have to do now:

  • get rid of all the hardcoded strings in the scripts and add some basic error detection / handling
  • add the importer system to the Docker images and add a few scripts for configuring everything

In general, I’m happy with how this Electron+Docker system will allow me to get the beta out to people without me having any access to their data. However, an outstanding hurdle is figuring out a good way to authorize third-party services like Twitter, Gmail, Github, etc. It’s bad security practice to include the OAuth secret keys inside the app because people who abuse their access tokens will be tied back to my app. I could set up a single hosted OAuth service that the electron app can connect to facilitate authorizations but I could theoretically steal user access tokens this way. What I think I’ll do is do this but open source the OAuth app for people to verify that I’m not caching credentials and also give the paranoid the option to configure OAuth on their own. If anyone has any clever ideas, I’m all ears but I think this is just a fundamental limitation of OAuth which was designed for the era of hosted web apps.

I also worked on the tab system in Electron that lets people run different queries and screens at the same time. Some improvements this week:

  • implementation of back/forward behaviour for tabs (doesn’t come for free in <webview>’s)
  • tabs now survive restarts of the app (state persisted to localStorage)
  • external links pop out of the Electron app and open with default browser

Speaking of tabs

Earlier this month, I was talking to Dann about the Memex’s API and how it’s powered by a graph schema implemented on top of Postgres. Dann wrote a great article about implementing a graph database in Javascript and I joked that I could reimplement my API this way.

But first, I told him, I’d have to actually finish reading the article which has been open in a Chrome tab for months. How many months? I decided to actually check and it’s been more than a year (this is basically a graph of browser restarts):


A fun demonstration of using the Memex to quantify shame.

Anticipating the beta

If you’re craving for the ability to do the above (and hopefully the other usecases), the beta should be ready very soon!

Here are a few things you can do now that will ensure you have a lot of data when you get started:

  • install Moves on your iOS device . It does a great job of passively tracking your movements/visits and it’s a great contextual anchor for any other data you decide to import. If you have an Android, make sure Google is passively tracking your location history.
  • if you use bash and want to import your commandline history, increase the history limit and change the format so timestamps are saved alongside the command. Instructions here.
  • If you use Safari on iOS (default), make sure history is syncing to your OSX’s Safari . Then in Preferences on OSX, make sure to set “Remove history items” to “Manually” so all history remains available.
  • Spotify doesn’t have an API for listening history but last.fm does. Create an account if you don’t have one already, open Spotify’s preferences, and connect it.
  • If you want your podcast history synced, you have to use the Podcast iOS app and occasionally sync your phone to iTunes. Unfortunately Overcast doesn’t have an API.
  • Install Timing or RescueTime to start tracking your activity on your desktop computer.

Other weirdos

I love when people send me links of people working on similar projects because it makes me feel less of a weirdo. Here are a few links that have come my way recently:

  • Andrey came across Aaron Parecki who’s been doing lifelogging for years. I’m building a private database for my records and Aaron seems to be doing the opposite, putting it all out in the open . Here’s how much he’s been sleeping lately .
  • Kate sent me a podcast about someone who created an app to randomize his life experiences . Though not quite the same, I often use my Memex to check how many new people I talked to or new places I visited in a given week. It would be fun for someone to build an app on top of the Memex dataset that quantifies and incentivizes new experiences.
  • Tom sent me a link about the launch of Monica , a private “CRM” for tracking friends and family. It’s centrally hosted but also open source so you can install it on your own.
  • A few different people sent me an article by a local startup CEO Kunal Gupta about how he tracks his time. I was at an event with him this week but he had to leave before I got a chance to talk to him about journalling; I’ll try to catch him another time.

Please send me your finds!

© 2021