BloggerCon II Weblog - Celebrating the art and science of weblogs, April 17 at Harvard Law School.

Web Directories in the 21st Century

Posted by dwiner@cyber.law.harvard.edu, 10/13/03 at 6:32:32 AM.

In the aggregators session on Day 2 at BloggerCon I promised Jenny Levine that I would write up my plan to empower librarians and scholars to build distributed Yahoo-like directories, with the key ingredient that makes the Web work so well -- competition.

Today's Web directories 

Think about Yahoo and DMOZ, the two leading Web directories.

Pick a subject you know something about and visit the sub-directory for that subject.

Is it in the right place in the hierarchy? Often not. For example, I created a piece of software for the Mac, and then ported to Windows. But it's listed under Mac software.

Is the sub-directory complete? Probably nowhere near complete. For years, Scripting News wasn't listed as a weblog on Yahoo. RSS 2.0 isn't listed under RSS, which is like omitting Vail from the list of American ski resorts. Or on DMOZ, RSS is listed under RDF, reflecting a bias among the editors of the directory.

Does the organization make sense? Even if it does, is it the only way the information could be organized?

Does it go deep enough? It wouldn't make sense for Yahoo to have a directory of first semester English courses at your community college, but it might make sense for your school to have such a directory.

Is it useful? Directories are lists, and lists are useful. But today's directories have such serious problems, most people rarely use them.

Could they work if we changed the way they work? I believe the answer is yes.

Competition is the way of the Web 

We accept that there are two main directories, but why do we accept that?

Suppose someone said there were two home pages for the Web. How good could they be?

What if they said that in order to write for one of these home pages you either had to be an employee of a relatively small company, or be part of an open source project. What if the subject you're an expert in already has an editor in DMOZ? What if you already have a job you like and don't want to go work for Yahoo? And how many people can Yahoo afford to employ to work on their directory? Wouldn't it make more sense to open the process up, as the Web is open, and let the cream rise to the top?

First, we need to break the monopolies, to open up the idea of web directories to competition, so there can be as many sub-directories for a category as there are people who have an interest and have expertise.

It's all just a bootstrap away. People have to start creating and maintaining sub-directories, and then link to each other through their directories, and away we go. This is something that is entirely in the user's hands. The technology is already fully invented.

The format for directories 

OPML or Outline Processor Markup Language, is an XML dialect that's designed to model outlines, or hierarchies of information that can be edited with an outliner. There are several OPML-capable outliners, including Radio UserLand, which is what I use to edit directory outlines.

There's a directory on this site, which is edited in OPML. The content management system for this site, Manila, knows how to display outlines in Web-directory-style. Any content system could do this, the format is open and very simple and easy to implement.

Here's the OPML source as a text file. Some browsers will have trouble directly rendering an OPML file.

Key point -- any text editor could be used to create the OPML source for a directory.

And any dynamic content system, from PHP to Zope to WebObjects, could be used to browse them.

Another key idea -- inclusion 

When I link to another directory from my directory, it is included in my directory, and the inclusion is seamless.

The only difference that a user might notice is that the author at the bottom of the page changes, and that suggested links go to that author instead of the person doing the linking.

By including one directory in mine, I'm delegating authority on that subject to the person I'm linking to. If they should stop updating their directory, or take it in a direction I don't support, or if a better one comes along, I can change the link in my directory. So we compete at every level for authority, the same way Web pages compete. If our goal is to be the authority, we are encouraged to be inclusive. In other words, this design for directories follows the grain of the Web, where Yahoo and DMOZ, emphatically, do not.

More on inclusion here.

Pointers 

Paul Boutin explains 21st Century Web Directories.

Discuss this message.