Individual and Community: A Tale of Two Unix Programs and Three Generations of Code

The Internet is incredibly old, if you count it in Internet years. These days Web two-point-oh is the big deal, but essentially all it does is recreate old Unix commands using HTML and JavaScript. Right across the Internet, clever coders, wise writers and righteous readers continue to reinvent the oldest ideas.

These days it is common to speak about the blogosphere: the ill-defined cloud of websites organised in a similar fashion (chronologically, then by category), with similar features (commenting) and natures (stories built from linking). The blogosphere is a kind of community, and has within it many subcommunities. It is loosely connected via the use of hyperlinks; though for some clever coders that will never be enough.

But the thing is that it’s not new. The blogosphere existed before it was called the blogosphere. In a previous generation it was known for particular sites such as Slashdot or Kuro5hin. And before that were the dim and distant days of true nerdery.

Let There Be Code …

In the beginning there was Unix. As an operating system it has continued to dismay purists and dominate the computer industry in one form or another. The family tree of Unix stretches back three decades and it was intimately related to the growth of the Internet.

In the early days, everything was done at terminals by keyboard alone. There were no mice and no icons, though beards were (and still are) quite common. The openness of the system meant that programmers were forever tinkering and adding to the features of Unix; by itself Unix was a corpus of latin and the code that followed a body of great literature.

There came the emergence of the finger command. This cheekily named command allowed one to query information about anyone belonging to a conforming system. It could tell you when they last logged in, if they were online currently, perhaps some details about which office they worked in. It also allowed you to read a “plan file”, which the finger‘d user could update themselves with details about what they were up to.

Plan files are the first blogs. They held currency even into the 90s. Graphics programming god John Carmark’s plan file was eagerly followed by legions of fans. It was frequently republished on game news websites and dissected by experts.

There also came the emergence of uucp. This program allowed two Unix systems to copy files to one another over a phone line. It led to the creation of the Network News Transport Protocol — NNTP — which underlay the Usenet.

While plan files have mostly died off, Usenet is still with us today. Its genius is that it works by individual sites synchronising with each other at various intervals. No one master server handles all traffic, instead messages are copied from server to server, fanning out across the entire Usenet network. The propagation of messages was based on cooperation on a site-to-site basis, and not every site carried every Usenet group.

Today Usenet messages propagate in minutes. It used to be that days could pass between propagations. Apocryphal legends refer to tapes being sent via airfreight to distant sites as it was quicker and cheaper than propagation via long distance modem connections, leading to the quip that “you should never underestimate the bandwidth of a station wagon full of tapes”.

Usenet was the first community of its kind: a vast, loosely connected cloud of subject-specific groups. Over time these groups gelled into subcommunities with different norms, rules, injokes and histories. Among other things, we owe the acronym FAQ to Usenet groups.

The Swiss Connection

The Hypertext Transport Protocol and Hypertext Markup Language were created by Tim Berners-Lee at the CERN physics lab in the early 1990s. As a protocol and language they were mostly static. However it did not take long for clever coders to start serving websites out of databases and hacked up Perl code, which led to the second generation of community code.

This generation was marked most of all by the emergence of Slashdot, which in its heyday acted as a central clearing house for an emergent worldwide Internet-connected geek culture. Even today Slashdot has millions of users and the “Slashdot effect”, where tens of thousands of Slashdot readers click on a link within the space of a few minutes, is the benchmark phrase by which crippling surges in traffic is measured.

Slashdot pioneered the weblog as a movable feast of links to interesting webpages, a function which it continues to this day. Slashdot also added a comments mechanism on a per-story basis, which led to the formation of a Slashdot community of sorts. Slashdot’s ornate system of two-tiered moderation, ‘karma’, comments ratings and threading show the intensity of troll activity and the resultant evolution of defence mechanisms. Even today, the community-management code of second generation platforms like Slash (the code driving Slashdot) and Scoop (driving Kuro5hin, Daily Kos and others) is far more sophisticated and capable than third generation platforms like WordPress.

Slashdot and other second generation sites like Kuro5hin — which was initially populated by Slashdot users dissatisfied with aspects of how Slashdot worked — mostly inherited from Usenet. The details vary, but the spirit is the same. In Slashdot users submit stories to the site editors, who pick which ones to publish, similar to moderated Usenet groups. At Kuro5hin anyone can write and publish stories, some of which can be voted onto the front page, a bit like unmoderated Usenet groups. In both cases the website is a community, a group. It is a multiuser system designed to be driven by users from the bottom up, not top down.

The third generation platforms — best typified by Movable Type and WordPress — went a different way. They originally recaptured the simplicity and individuality of plan files. The earliest versions of these programs did not have a multi-poster capability. And none of them is really driven by users in the same way as the second generation are.

After a fashion this is being reversed. Platforms like WordPress MU and its flagship site WordPress.com are aimed at creating a cohesive “ecosystem” of blogs: a sort of Usenet of plan files. Whether this works remains to be seen.

Still to Come

Will there be a fourth generation? I’m not sure, and of course such a fuzzily defined subject defies objective decision. I do expect that either third generation platforms will eventually co-opt the second generation focus on community, or that some fourth generation platform will simply overtake the third generation by returning to a focus on community. It may be that social networking sites are the fourth generation, except that they violate the own-installation many-sites model which the first three generations of community code have followed. It might also be that the Semantically Interlinked Online Communities project linked to above will create a more Usenet-like experience by more tightly linking conversations together across multiple blogs. Or it might prove to be too complicated to use and thus fall by the wayside.

From the Bible comes a saying that seems apropos: “there is nothing new under the Sun”. This is probably more a reflection on the ways of human nature than the cycles of technological change, and indeed the various authors of the Bible would probably goggle at the strides technology has made since they decided that nothing really changes. So perhaps a better image is the wheel of dharma, of death and rebirth; that what dies is reincarnated as a ‘new’ idea, freshly minted. That we all live, all of us, in the fascinating glow of the Once And Future Internet.

This entry was posted in Geeky Musings, History, IT and Internet. Bookmark the permalink.

22 Responses to Individual and Community: A Tale of Two Unix Programs and Three Generations of Code

  1. gilmae says:

    Perhaps fourth generation will be posts+comment threads on multiple blogs to be seamlessly weaved together into one big thread.

  2. Jacques Chester says:

    I think that’s what the SIOC project is trying to achieve, though mainly through the brute force application of buzzwords.

    Cam and I have kicked the idea around a few times. It’d probably wind up being a reinvention of NNTP as a WordPress plugin. But it’s problematic: what happens when your cat blog gets its thread union’d with Slashdot because you posted about Cats 2.0?

  3. Cam says:

    Jacques, gilmae had a go with something similar with greasemonkey after Blair and Lambert had their dust-up. That also led to newcopia.com (gone now) which wrapped rss feeds with commenting ability for the likes of news/etc which didnt have them.

  4. Jacques Chester says:

    Yeah, I think I saw that thread before — you keep referring me to it and I keep pretending I had the idea first.

    For simplicity I think the best way to do it is mutually exchanged RSS feeds of comments. Trackbacks could be used to identify where else the conversation is taking place. Or we could throw our hat into the SIOC all-encompassing vision ring, though experience shows that the good beats the perfect.

  5. gilmae says:

    That’s stunning. I could have sworn I read every single word of that post, and yet I didn’t even see that you were talking about exactly what I said. I must be on autopilot.

    Anyway, it’s an interesting idea. This isn’t the first time I’ve seen it tried. I remember a few years ago when I would spend hours tracking the movements of the meta-blog crowd there was an attempt to do something like this, Threadneedle. Never went anywhere though, mostly because all the blog APIs at the time sucked. Given that the blog APIs now are basically the same as then – Atom is yet to be implemented – the SIOC boys are at this stage trying to solve step 3 without solving step 2.

    Speaking of ideas kicked around with Cam, he has been bored stupid on a number of occasions on my fear of a spam planet. Spam always wins. Too much money in it, so it is like herpes; just when you think you’ve beaten it down, it comes back itchier than ever. I imagine imaginative spammers secretly dream of this kind of thing as it solves one of their only remaining problems. It is trivially easy to fill up the comment threads of a d-list blog, less so on a heavily-read blog. Weaving together the comment threads of A- and D- list blogs means spammers can target the D-Listers and still have the ads read by readers of the A-listers.

    Akismet? Bah! Give me a million dollars and I’ll hire a thousand Chinese girls tired of making bra strap clips and they’ll overload Akismet with hand-posted spam comments :- )

  6. gilmae says:

    Cam: Yeah, the problem was I didn’t anticipate Rupert realising that putting blogs on his websites was a license to print ad-generated money :- )

  7. Cam says:

    gilmae, Yeh more comments = more page views = more ad impression. It had limited life cycle with news media as comments were inevitable on their stories. I think it still has value on things like the Parliamentary Library rss feed and other areas such as Hansards, etc.

  8. Jacques Chester says:

    Atom is yet to be implemented

    I’ve never seen a clear exposition about why Atom is/was/will be such a big deal. RSS as it exists today seems to do what needs to be done. What am I missing?

    As for spam-by-thread-union, that’s a toughie. A web of trust model, perhaps? Then you’re getting back to the SIOC problem: a grand vision with no implementation and no way to easily fold it into the writing experience of Nick Gruens and Ken Parishes.

  9. Graham says:

    It’s funny to remember the olden days when I first used the Internet. The top of the range modem was 14.4 run over a phone line. Netscape was king. That was about 15 years ago…The really cool modern movies before that had teenage hackers with phone handsets they planted into some sort of special cradle. (Remember “War Games”?) That was around 20 years ago. Fifty years ago Harry Messel installed the fastest, most powerful computer in Australia at Sydney Uni. It took up a couple of floors (I think) and was about as powerful as a $5 calculator. As you know, computers in those days had valves and mechanical relays and the first “bug ” is said to have been a moth that inserted itself between the contact points of one of those relays in a naval computer in Bethesda, Maryland.

    I thought the first internety/bloggy things were Bulletin Boards, weren’t they? which have morphed into today’s open source BBs and forums which are still fairly popular.

    Something I want in the future is, rather than linking off to another blog (or whatever), to be able to display an external post or comment in my own blog – “placed” but not imported (so without infringing copyright) – as part of a thread of several people’s ideas which I might be constructing. So it would be a community of debate rather than just my own opinion.
    Group hug?

  10. Jacques Chester says:

    I thought the first internety/bloggy things were Bulletin Boards, werent they?

    In practice Usenet came first, though some BBs like WELL are of similar vintage. There was also FIDOnet, which was a Usenet-like site-to-site system between bulletin boards running the Fido software. It was later folded into Usenet.

  11. Christopher says:

    Ive never seen a clear exposition about why Atom is/was/will be such a big deal.

    And you never will. It was always about big egos clashing.

    cheers,
    Christopher

  12. gilmae says:

    Atom is a big deal, for the purposes of this at least, because the publishing API is hands down superior. The syndication format, however, is probably six of one, half-dozen of the others except for one aspect, one of the original driving forces – it is defined. Implementing a reader/parser doesn’t require steeping oneself in a knowledge captured only in the heads of other implementors.

  13. David Rubie says:

    Graham said:

    The really cool modern movies before that had teenage hackers with phone handsets they planted into some sort of special cradle.

    An acoustic coupler. The early modems occasionally had to work with telephones you couldn’t unplug and you needed the telephone itself to do the dialling. Modems are one thing I do not miss. Usenet might still be around, but it has never been the same since the majority of the audience was overtaken by the great unwashed – not that I resent the presence of non-geeks but the threads were often so geek-biased it was fascinating just what a distributed mono-culture existed amongst them.

  14. dr faustus says:

    I like these occasional geek reminiscing threads!

    I never quite made the connect between .plan files and blogs myself. Although a Linux user for many years, I never actually used it for work. Unless you were on an always-on network (most of us Linux users were on modems), there wasn’t much point.

    For me, the online social networks really developed from local, multi-line BBSs (many years ago I spent most evenings logged onto a 6-line Victorian BBS called Empire), through to UseNet to places like Slashdot (I still have a low five-figure User ID on Slashdot) and eventually to blogs.

    I haven’t been on K5 now for many years, but they had a ‘journal’ section (I’m pretty sure they had it before Slashdot), where you could post your own story and allow comments. That was back before RSS feeds, so it encouraged you to log on regularly to see if anyone had commented. That was my first blog-type experience (complete with flames and all).

    I’m not even sure if Kuro5hin is still active, but what killed it for me was that all of the interesting discussion would occur in the story voting phase, and by the time it actually hit the front page, it was old news. I think MetaFilter has probably filled the hole left by K5. Unlike other social blogs like digg, the links are interesting, but comments are the best part of MeFi.

    Interestingly, in the last few days Slashdot has announced ‘firehose’, which sounds a lot like K5’s story voting system. It will be interesting to see if they manage to maintain a balance between those who get in early to vote on stories, and those who just want to read the front page.

  15. Yobbo says:

    On a tangential but somewhat related note. Anyone know how to turn off F***ING “SnapShots”? Most annoying malware since quicktime.

  16. gilmae says:

    Yobbo: There is some cookie you can set to disable them, http://www.snap.com/about/shots_faq.php#3 Thats the quick and dirty way. You could probably also investigate adding snap.com to your hosts file and redirect it to localhost

  17. ds says:

    dr faustus: re k5. k5 is still kicking, sort of. Many people complain that it is dying, but I think it will limp along for another couple of years at least. After all, I am pretty sure it is still making money through ads for Rusty, and the maintenance costs appear to be minimal. So why not let it live(limp).

    I actually go there daily, but mostly to read the diary section. It has been a long time since I last read a story that made its way out of the queue, but they do exist. They are just too long for me to bother reading.

    As for .plan files, they died because of the security risk of finger. When many people have weak passwords it is very beneficial to know what usernames to try!

  18. Rusty says:

    K5 is still alive. It’s much lower volume than it was in the old days (circa 2000-2002ish), mainly (I think) because there are a lot more topic-specific options these days. If you’re interested in politics, you’ve got dKos and a dozen other big political communities, there’s lots of tech communities now, linkdump and short-item places like MetaFilter (which is older than K5, incidentally — in terms of “generations,” K5 and MeFi would be siblings). Back in the day, you had Slashdot for nerd stuff, and K5 for everything else, and that was about it. I’m glad there’s more diversity out there now. Dealing with K5’s traffic in those days was a pain. :-)

    My two cents on what the “Fourth Generation” might be is community edited sites + personal member blogs + social networking site features. We’re going to see (and are already seeing) the standalone social networking stuff merge with the blog community stuff. The blogging community tools encourage ongoing communication, and the social networking tools help groups form and keep connected, online and off. It’s a natural (please avert your eyes while I use this loathsome word:) “convergence” for both kinds of sites.

  19. Jacques Chester says:

    Do you see K5 moving in that direction, Rusty?

  20. Pingback: Club Troppo » Friday’s Missing Link - on Friday!

  21. Dave Bath says:

    Graham said:
    * The top of the range modem was 14.4 run over a phone line….
    * Netscape was king

    Giggle… how about warbled connections over a normal audio phone line (and some people could warble up a handshake with their throat. Tranmission speed was probably slower than a fast typist, and took between five and ten seconds to type a line of text. It was almost faster to walk across town carrying your card desk or paper tape. (As for later magnetic media, and 1600 bits per inch you could “cut and paste”, literally, with not much data lost over the splice).

    Netscape was King?? Naaaa, Mosaic was the original king of GUI browsers, and had features STILL not found in some modern browsers (buttons to navigate according to metadata tags, GUI settings of fonts and sizes to different markup instructions (e.g. set H2 to Helvetica 14pt underlined). Of course, there was no Windows version, as Microsoft didn’t even include the TCP/IP stack on their distribution media, so there wasn’t much point unless you went through the hassle of going to Waterloo to get one.

    Jacques said:
    * In the beginning was Unix
    Hmmmmm. What about MULTICS??? All the decent operating systems (e.g. VMS and Unix, and of course, the cute PDP series) came from lessons learned from the MULTICS initiative. Indeed, the word “UNIX” is a pun implied a cut-down version of MULTICS…. EUNUCHS!

    Mind you, I agree (as is so often the case) with the thrust of Jacques’ positions.

  22. Jacques Chester says:

    Does it date me that I used Mosaic on a 14.4k modem when I was … um … 15? 16? I don’t exactly recall.

    As for MULTICS, isn’t that the OS developed by St Peter?

Leave a Reply to Graham Cancel reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.