August 20, 2017

Getting the Memex ready for beta

If you haven’t enjoyed my meandering rambles about literature and Memex-related history, this newsletter edition is for you! This update is all business.

I spent most of this week polishing the Memex desktop app so it’s ready for beta testing. In particular, the onboarding flow is ready to use. Here’s what it looks like now:

A recap of the Memex architecture:

  • a Rails app for the API: it processes new personal data and responds to queries for existing data
  • another Rails app for extracting data from files or third-party APIs, transforming them into the right schema, and loading them into the API
  • a Docker Compose script that runs these apps in isolation inside Docker containers
  • an EmberJS frontend for viewing the data
  • a second EmberJS app that runs in Electron that gives users a desktop app for viewing their data and interacting with the importers and API

This solves the a primary design goal of ensuring that user data is only accessible by the user alone. There are many downsides: no cloud API for mobile devices to connect to, Docker is resource-intensive, and deploying updates and fixing bugs is way harder than pushing a fix to a single server. But this lets me get a working version out to people so I can get feedback and direction.

If you don’t want Docker running your computer and want to run the API/importers on your own server, it’s possible to configure the frontend to make external calls.

If you’re interested in having your own Memex and want to be a guinea pig let me know! Ideally, we would do the installation together in person so I can get a high fidelity view of all the things that might will go wrong.

A few other specific things that got done this week:

  • got rid of all hardcoded paths in the installation and API scripts
  • switched the API/importers to using uncommon ports (5701/5711) so that they can live alongside other Rails apps that a user might be running
  • better messaging when initially using the interface without data from importers
  • ability to handle credential changes
  • better naming for the docker containers and volumes

Things to code still (but not essential for beta users with high tolerance):

  • Adding a system for error collection. Using a normal exception notifier like Airbrake would send too much personal data to me; I need to adapt something similar but that lets users send an anonymized error report on their terms.
  • simple interface for adding API credentials for optional addons like AWS S3 (for image storage) and Google Cloud (OCR functionality)
  • interface for adding Chrome extension (for logging browser history and saving highlights from web page)
© 2021