January 21, 2018

Three origin stories about the internet

This newsletter edition tells two stories you may already know and one you might not.

It’s based on a talk I did a StarCon a few weeks ago (slides here (13mb))

Story 1: Vannevar Bush and the Memex

Vannevar Bush had a problem. He was drowning in paper. When America entered World War Two, it caused a bureaucratic explosion. More paperwork, reports, and correspondence were being produced than ever before.

alt text
NSA employees in WW2, part of the explosion in paperwork and bureaucracy

Bush was a successful engineer and scientific administrator in the 20s and 30s. During the war, he headed up the National Defense Research Committee which coordinated all civilian research. TIME magazine called him the “General of Physics”. His organization produced 35,000 reports in just a few years. “We are being buried in our own product,” he wrote — new tools were being invented to create information but few were being created to make sense of it.

And the problem was about to become more urgent because nuclear bombs were just around the corner. One of his secret roles was overseeing the Manhattan Project which was building the first atomic bomb. Our species was losing the ability to understand complexity while at the same time gaining the ability to blow ourselves up.

As the war came to an end, he thought about solutions. Wouldn’t it be amazing if people could have a device that would store all their information and let them search and organize it easily? He thought up with a device to do this, called it “The Memex” and published his thoughts in an essay in The Atlantic.

The Memex was to be a desk-sized device that contained a microfilm, a screen to display information on, and control buttons.

An illustration of the Memex from 1945
An illustration of the Memex from 1945

The most powerful feature of the Memex was Bush’s idea that users should be able to link arbitrary pieces of data together and be able to navigate through this graph. Just like how the brain works by association.

If you’re thinking “what’s the big deal?”, here’s what a computer was in that era:

A room-sized computer
A room-sized computer

Here’s how Doug Engelbart described it:

I’ll tell you what a computer was in those days. It was an underpaid woman sitting there with a hand calculator, and they'd have rooms full of them, that's how they got their computing done. So you'd say, "What's your job?" "I'm a computer."

At most, there were a few specialized mechanical computers; the first digital computer, the ENIAC, was still a few years away.

The Memex essay caused quite a stir when it came out but it was never built. But as I explored in previous issues, the idea didn’t die — just the opposite. One of the many people it impacted:

Story 2: Doug Engelbart and his demo

Doug Engelbart has just finished his training as a radar technician in the navy but as soon as he got deployed, WW2 ended and he found himself in the Philippines with a lot of free time on his hands. One day, he stumbles into a library run by the Red Cross in a hut like the one below. By chance, he comes across Vannevar Bush’s Memex article.

26BF56EF 4077 407B A1A2 E194F7E58ACB

It was like a bolt of lightning and he knew he had to make the Memex vision a reality. It took almost 15 years but Engelbart finally got funding to pursue his own research at Stanford Research Institute. He got to work building a computer system that would help users organize and navigate through information.

Engelbart quickly put together a quirky, ambitious team. Groups of people would casually hang out around computer monitors and collaborate. People wore sandals and tshirts to the office. Hair grew longer. People occasionally snuck joints into the office. Everyone (including Engelbart) was trying LSD. They had standing desks. They even had yoga desk. Even by 1960s California standards, people thought his group was weird. Engelbart would smile and say “You will all be doing this one day”

62A9466F 89A9 47BB 8BB3 E01BEF7909DF

They made good progress on their interactive computer system but Engelbart was having trouble explaining what exactly he had built to the outside world. In 1968, he made the risky decision to do a live demo in front of 2000 computing professionals in San Francisco.

683C0339 35EA 4AE6 847F C2A03A1B241C Screenshot from the demo’s recording. It’s amazing.

Computer professionals in 1968 worked almost exclusively with punchcards and printers — for everyone in the audience, the 90 minute demo was like a religious awakening. For the first time, the world saw the computer mouse (unfortunately, what Engelbart is most remembered for), interactive text editing, hyperlinks. His system also included a form of wikis, email, and source control, all firsts. As a final wow, he video conferenced other team member and collaborated with them remotely.

Story 2.5: an interlude about ARPANET

ARPA was a research and development arm of the US Department of Defence. The majority of ARPA’s attention in the 60s was devoted to coming up with horrible ways to win the war in Vietnam (great history of ARPA in this book). But they also decided to fund a small experiment in networking computers together.

288E8D78 E49F 43D5 8AD5 1DF575A386C1 The folks at UCLA, one half of the internet in 1969

When Doug Engelbart heard about the idea of connecting computers together, he immediately recognized the potential and volunteered his system to be part of the experiment. By 1969, the first routers were delivered and installed, connecting Engelbart’s system at Stanford to UCLA. The first data transferred was an “l” and then an “o” and then nothing — the programmer was running the login command but it crashed on the ‘g’. The buffer size was increased, the operating system was recompiled, and 15 minutes later, the first message was exchanged on the internet! (Story recounted here.)

ARPANET added more hosts but there was a problem: people weren’t that into it. For one thing, universities didn’t want pesky grad students from other schools clogging up their valuable computer resources. Another problem: ARPA had spent a lot of time on engineering the packet switching hardware but software, documentation? ¯_(ツ)_/¯ . Users were left to their own to figure this out. At the end of 1972, there was going to be a big ARPANET showcase in Washington DC and the project leaders hoped this would help drum up enthusiasm.

Which brings us to our final story:

Story 3: Elizabeth Feinler and the NIC

EE9EE66A B967 43AB 996C 144F7C8DD069 More photos of her here

Elizabeth Feinler worked in the same building as Doug Engelbart, running a team that did literature searches for chemists. As Engelbart scrambled to get ready for ARPANET’s big demo, he came to her:

“I have a job for you. We need a resource handbook for the internet.”

“What’s a resource handbook?” asked Feinler.

Engelbart replied: “I don’t have a clue but we need one in six weeks”

This is what had happened: the ARPANET network engineers had started writing this handbook but then they got distracted by shinier technical problems. She helped put together almost 1000 pages of documentation that would help new users get started on the system. The ARPANET demo was a success and she began working full-time on the network.

(Not the Resource Handbook but a similar piece of documentation from a bit later)
(Not the Resource Handbook but a similar piece of documentation from a bit later)

Over the next twenty years, Feinler, played a huge part in shaping the internet. Here are some of the ways:

When new users would connect to the ARPANET, there was no Google to help figure things out. Instead, there was the NIC — the Network Information Center. This is the group that Elizabeth Feinler began managing in 1972. At the beginning, they focused on things like mailing out huge handbooks and running a toll-free hotline. Then they began adding interactive services. There was the WHATIS service that let users query servers. There was also the WHOIS service that let someone enter a person’s last name and get back information about a person and how to reach them.

One day, the military came to her and asked her to add titles and ranks to the directory. She recounts what happened next:

First of all, I refused to do it, because it was an administrative nightmare; there was no way we could have everybody’s title up to date, because in the military they’re always getting promoted. So I kind of refused to do it, and my pitch was on the grounds that it would just be impossible to do—which was not untrue, but I really didn’t want to do it because I didn’t want anybody to know who they were talking to!

This egalitarianism remains part of the internet, at least in spirit. She says:

I’d have a fifteen-year-old hacker talking to a Nobel Prize-winner, and neither of them knew who the other one was!

ARPA’s attention was mostly on the hardware and engineering aspects of the network and users were left on their own to figure out protocols. Feinler and the NIC made sure a wide range of people participated in this development. They handled documentation and backups when the programmers inevitably forgot to do it:

Well, I sat in on a lot of the protocol meetings, because it was my job to gather up the information, and I would pick up anything that wasn’t nailed down; because often the NIC got questions about it, or we got people wanting copies of things, and the people that were building it were not thinking about that, or the administrators. They never thought about the paper, the back-up information and so forth. So I would just go around like this little squirrel, grabbing everything that wasn’t nailed down, because after the fact people would want it.

46A15471 36AC 4526 97CD CA0F3FF9CAC1 The NIC’s archive vault, shelves full of print documents and magnetic backup tapes.

Not many users were too excited about ARPANET but that all changed when email was created. It was the first killer app and it quickly took up the majority of bandwidth on the network. Most importantly for users, you could reach other users by using an email address instead of remembering network address. Feinler’s team took on the responsibility of maintaining the table that translated hostnames into network addresses. At first, this was done by sending out a paper version to each administrator. Then, they automated this process by storing the host table as an ASCII flat file named HOSTS.TXT that administrators would have to copy over FTP a few times per week. A few lazy admins who didn’t update their table could make the whole network break down and Feinler spent hours convincing sys admins that they needed to keep up to date.


As the ARPANET grew in importance, everyone wanted their own top level hostnames. But nobody could agree on the rules for assigning them. Feinler looked into the future and saw that this problem would just keep getting worse and worse (or “an avalanche of problems” as she put it). Feinler and the NIC decided to split up the network into various top level domains that could be administrated separately. There would be .mil, .gov, .edu, .org, .bus. At the last minute, someone realized .bus was being used for some hardware controllers so they switched .com. (Story from here.)

Every time there was a protocol to update, her team would stay up all night working on the migrations and then remain at work the next day to handle all the user support. She drank a lot of Mountain Dew during this time. The biggest of these migrations was the switch to TCP/IP. It was a new and improved protocol developed in the 70s but people were taking their sweet time to upgrade. Finally, the military imposed a hard deadline of January 1, 1983. It was a huge scramble and some hosts didn’t make it but the switchover was a success.

D5D9CA91 83B3 496E 805B BAC77906DDE0

Around this time, the ARPANET went from being considered a research network to an operational network. The NIC became more important than ever and Feinler’s role grew. By the end of her time with the NIC, she was managing a team of 45 and had a budget of $11M/year.

ARPANET wasn’t the only computer network out there. France had CYCLADES and then later, a TV and phone-based system called Minitel. Canada had a similar called Telidon. A popular academic system called Plato had email, forums, and online games in the 70s. And there were lots of corporate and institutional networks like IBM’s SNA, DEC’s DECNET, and IPSANET in Canada. And in the 80s, most people expected the technically-superior OSI standards to win.

Why did the internet win? It’s a complicated question to answer, of course. Having the backing of the American government certainly helped. But letting users shape the way things ran turned out to be a pretty good idea. Elizabeth Feinler and the NIC were crucial in facilitating this.

In doing research for the talk, I came across a usenet post by Feinler long after her work at the NIC had wrapped up. It’s a heated debate where people are complaining and flaming each other about the domain name system. She drops in to spread some wisdom, concluding:

If you have good ideas get them in there...global feedback and consensus are what have made the Internet great...it's a chaotic democracy not a cartel conspiracy.

I think it nicely summarizes the way she was a leader and shaper of the internet.

These three stories are only a fraction of what got us to the internet. But I think they’re representative of the different types of work required to make a new technology successful.

Conceptualizers like Vannevar Bush open up space and let us imagine new world. Doug Engelbart took a wild vision and made it work. In academia, people say “publish or perish”. In tech, we sometimes say: “Demo or die”

Elizabeth Feinler never learned how to program and never got involved in nitty gritty technical debates about the internet. But her thirty years of writing, talking on the phone, and exchanging emails had an incredible impact on the version of the internet we have now. She convinced a whole bunch of people with big egos and different goals to work together and to do it in an open and collaborative way. And most impressively, at the height of the Cold War, with a project sponsored by the military, she fought to create an open network where planning and control was in the hands of users.

9CAACFD6 4C60 465B A2B4 C786D98FA667

The stories of Vannevar Bush’s Memex and Doug Engelbart’s demo are often celebrated but the story of Elizabeth Feinler is probably a new one.

Researching the history of Feinler and the NIC was a reminder to me about how easy it is to overlook certain categories of work when creating narratives and origin stories.

Project updates

This is the first newsletter in a month — I’ve been distracted by conferences for the last few weeks and holidays the weeks before. But work on my Memex has been marching on!

Installer progress

A few weeks ago, I installed the Electron+Docker version of the app for a friend and it went smoother than previous times! I’ve been working through some of the bugs and onboarding flows and should have confidence to do a few more installs soon.

Importer updates

  • Spotify recently added a listening history endpoint to their API (thanks Peter!) and I put together a quick proof of concept importer. I’m pretty excited about this one because before this, last.fm, stagnant for almost a decade, was the only way to do this.
  • I put together a sneaky importer for the great podcasting app Breaker. It takes advantage of their undocumented internal API. This is good news for me because most other podcasting apps on iOS are awful (especially the official Apple one).
  • A few newsletters ago, I was happy to talk about a new Dribbble importer. This month, they announced they’re phasing out the API that this depended on. #thewebwelost

Preparing for demos

I have a few big demos coming up starting in March and I’ve been working on a list of flows and scenarios for the dream version of these demos. I need to be able to show the breadth of this project. I’ll probably end up building a few new visualizations and query types to make this work.

© 2021