Prince, force diagrams, and New YorkApr 30, 2017
Joy in Introspection
Spotify finally secured the streaming rights to Prince’s music in February. When he was alive, Prince was an artist who always attempted to own the means of production/distribution and is likely rolling in his grave. Personally however, it means I was able to easily listen to his music again after a long break. I like being able to check my graphs to quickly confirm these sorts of things:
Phill also used his graphs to confirm a spike in StackOverflow visits after agreeing to do a quick “two hour job” for his partner involving Visual Basic scripting.
Force Diagram continued
I continued to work on the d3 force diagram this week. I refactored the component to be able to display arbitrary graph segments from a query result. The query endpoint returns a set of nested JSON objects, they get converted by Ember into models/relationships, and the d3 component traverses through these objects using the
eachRelationship() method. I’m pretty proud of this component, both as a neat use of Ember Data and because it’s a good way to visualize the structure and breadth of my dataset. One issue I still have to work on is avoiding cycles in the graph; I had to hardcode a set of edge types so inverse relationships would get ignored, thus avoiding infinite loops (a post is
contained in a thread which
contains a post which is
contained in a thread, ad infinitum).
Here’s recent podcast activity (blue episodes, red podcasts, grey publishers):
Other interface updates
I continue to thrash on a direction for improving the querying interface. In particular, I need users to be able to build search queries and traversals and configure how the results are displayed (as a timeline, table, graph, etc). Much time has been spent trawling Dribbble and sketching on paper but I don’t feel I have a clear path forward on this so not much to report.
The path to an OSX app
I played around with sqlite this week to figure out if it might work well as the master database for an OSX-based version of the app (as opposed to rails API in the cloud). I set up a mix of sample data (including some very clever posts from the ‘tweeter’ service generated by the Faker gem) and some slices of real data.
- for basic querying and browsing, the performance was almost equivalent to postgres on my local machine
- if the ETL scripts are going to be running at the same time as user queries, work will have to be done to get around sqlite’s locking system
- the SQL for traversing the nodes/relationships tables (a sequence of INNER JOINs) works as well as on postgres
- fulltext search might be roughly equivalent but because the indexes are set up as virtual tables, work will have to be done to mix search terms and regular WHERE clauses on columns in the traversals.
I also nerdsniped Max who figured out how to package postgres into an electron app. But even without an API server and the ETL scripts, the app clocks in at 500mb. The query endpoint isn’t too complicated and I feel I could rewrite it in Node and the ETL scripts could probably written in golang but I don’t have much (err, any) experience in either so I might try to package the current rails app before I venture to riskier technical ventures.
New York, New York
Isaac set up a special edition of the Ember meetup group ( https://www.meetup.com/EmberJS-NYC/events/239541566/ ) and we’re both going to do half hour talks.
Things I want to include in my demo:
- the history of the Memex
- a few query examples (github activity, search ‘engelbart’, finding a particular book quote via a burrito consumption) to give an idea of the breadth of the data
- force diagram to show the data structure and go over how it works as an Ember component
- timeline interface
- the query language and Ember parsing engine
Things I need to do before the demo: get hyfen.net updated and publish a subpage for this project, making sure it has an email collector
- find and massage a query that demonstrates social data
- make the force graph visually understandable
- invite more friends