Keith Seabourn

Some friends are in a lively discussion about a knowledge system we’re building. How do you help people find the “best” information? One idea is a group of experts who review posted knowledge and sort out the best. Another is to trust the “wisdom of the crowds.”

I wrote the following in that discussion. My friend Rob promised to blog about it if I’d post to my blog. Rob, I’m watching!

Concerning the idea of an “expert editorial group”, remember the example of internet search engines/directories. We can debate the similarity, but I see this as a good parallel to our dream of an explosion of info in knowledge system and the need to get people connected to the best stuff.

Keith’s unofficial history finding good information on the internet:

1. One of the first search/directories was Yahoo. There were others, but Yahoo is indicative of the directory approach. Yahoo’s “search” really was a search across their human-approved directory listings.

Yahoo was started by a couple of guys who would look at websites and list them in categories they found best. This is equivalent to an expert editorial committee, finding the “best” by some definition, assigning it to categories, etc.

Soon Yahoo was the most successful thing out there and the two guys began to enlist others. That is, the expert editor committee had to grow. They changed the process so that you listed yourself, in a category but an expert would look at your site and approve the listing before it became public. Eventually, success overtook their ability to keep up and they instituted a priority fee-for-listing service that would get you listed within 48 hours if you paid. Otherwise, they would eventually look at your site but it might be 2-4 months.

Yahoo’s success overwhelmed their ability to manage using an expert editor approach.

2. The next round of search engines were artificial intelligence geniuses. AltaVista was an early one. There were many others with varying AI analysis routines. The idea was to replace the expert editor committee with expert AI systems. They were the best thing going. You could submit your website URL. In the early days, within a few days your site could be found in the search engine. Later it began to take a few weeks, with a pay-for-priority service instituted if you wanted earlier consideration.

This spawned an amazing tug-of-war. AI algorithms are pretty smart, but humans are smarter. Humans would reverse-engineer the AI algorithm of the top search engines (AltaVista, Excite, Northern Light, etc.) They would then configure their web pages to get the top listings, not because they were the best pages, but because their creators understood how to fool the AI-based search engine results. Then the search engine guys would change their AI algorithms, and the dance would begin again. Break the algorithm, improve the algorithm, break the improved algorithm, improve the broken improved algorithm, etc.

The masses of users began to lose faith in AI searches because the masses want the best information to be listed at the top of the search results.

3. The current round of search engines, particularly Google, used a more populist approach — let the people vote on the best pages, with one link equalling one vote. Google developed the PageRank algorithm that basically counted the number of other webpages linking to a certain webpage. The pages with the greatest number of hyperlinks TO them were the “best” and listed at the top of the search results.

This was a great example of trusting the “wisdom of crowds”, or more modernly called “swarm theory.”

The Google PageRank algorithm was more difficult to spoof. There was nothing you could do to your page/site that would improve the ranking. You had to get others to link to your page/site to improve the ranking. But then human intelligence won over even this populist algorithm and people learned a year or two ago to force their sites to rank high in Google even though they didn’t have the most links from the general population. Several months ago, Google changed their algorithm to reduce the false high placements. The new dance has begun, this time between Google engineers and human search engine optimizers.

So, it seems to me that initially expert editor committees work well because human intelligence is better than the best AI at present, but over time, we need to implement peer-review or swarm-theory or wisdom-of-the-crowd approaches. They have proven superior in the search engine wars.

There are two separate issues: editorial comments and best ideas. Editorial comments are what Amazon does, improving knowledge by allowing others to add to it. Best ideas is what Google’s algorithm does, allowing people to identify the best solutions so that we are offering the best ideas “above the fold”, at the top of a search or browse listing.

Human intelligence is superior, but capacity is limited. So let’s blend a solution that maximizes human intelligence where possible, to both enhance knowledge through comments and find knowledge through the “wisdom of the crowds”.

The experience of my relationship with Bill Bright has taught me much about the promise and power of reconciliation. I will never again deny the prospect of coming together with those with whom I disagree. It is indeed the power of the gospel of Jesus Christ to break down the walls between us. Sojourners Magazine, 11/2004

The U.S. News and World Report cover story for this week is The Deep Divide: Why voters for both sides are so angry. This touches something that concerns me.

Having lived for 15 years in an African country that went through periods of democratic elections and military coups, I was always amazed at how few countries can go through a polarizing election then everyone works together following the election. The African country I resided in could not do that. After an election, people were left polarized. To the point of killing one another. Or plotting military coups because their party did not win.

I felt that it had something to do with the strong Christian heritage that still had residual influence in the U.S.

Today, I read an amazing article that spread hope. The November issue of Sojourners Magazine, which I rarely read, had an article by Jim Wallis on The Power of Reconciliation.

Jim Wallis, founder of Sojourner’s Magazine, and Bill Bright, founder of Campus Crusade for Christ, found themselves on opposite sides of the political process many years ago. This is the story of their reconciliation. Amazing.

I love it when something I wanted to read or see is published on the Internet!

A few weeks ago, I was travelling in Ohio and unable to watch a PBS program that I really wanted to see. Then I received Christian Leadership Ministries’ excellent The Real Issue ezine. I followed the URL there to order the PBS program DVD. I really wanted to see this program.

Probing around the PBS site, I found all the video segments online! So I was able to watch this program.

The recent PBS special was based on a popular course at Harvard Medical School. Dr. Armand Nicholi has taught a course comparing the worldviews of Sigmund Freud and C.S. Lewis. The Real Issue, a publication by Christian Leadership Ministries for university professors, published a two-part article by Dr. Nicholi.

I recommend this to anyone. It follows two of the most influential men of recent history through their lives. It documents two different worldviews, and shows how each man lived out his worldview, with dramatically different results.

Good stuff.

I don’t really intend to turn this into a tech-geek discussion, but I have found another tool that is pretty cool. It’s designed for those of us using LCD screens (that’s what is in your notebook computer). ClearType Tuner is the name. It’s part of Microsoft’s semi-official Power Toys.

I read about it in Woody’s Office Watch.

Note: partial results only. One-time indexing in progress. Indexing is only done when your computer is idle. Outlook email is only indexed when Outlook is open.

This message greeted me this morning. I excitedly opened up Google’s desktop search to test some searches. Having left my computer running all night, setting the power options to never shut it down, I expected Google’s desktop search to have completed indexing my notebook computer.

“Partial results” means it didn’t finish. Then I read the entire message and realized that I would have to leave Outlook open AND my notebook running in order to index everything. Oh well! I’ll try again tonight!

I’m still trying out Google’s desktop search, and comparing it with Copernic’s desktop search. I blogged about this yesterday. Copernic’s solution will index Outlook files even if Outlook is closed. This is nice, but is a security risk, I guess. Google must take a safer approach. If I reboot my notebook and do not open Outlook, Copernic will try to open Outlook mail files. This causes Outlook’s username/password login box to appear just like when I normally open Outlook (I’m on a network and my Outlook is set to synchronize with an Exchange server.)

Some differences I’ve noticed:

1. Copernic will index Outlook mail stores even when Outlook is not open. Google will only index Outlook mail stores while Outlook is open.

2. Both Copernic and Google will index all my Outlook mail stores. I have 3 — the normal Outlook offline mail store, a personal store, and an archive store. This is great! I like being able to search all my mail store files at once since the purpose of search is to not know where something is stored! However, Copernic tells me which mail store a message is found in. Google doesn’t seem to tell me where the message is stored. Maybe this doesn’t matter, since the message is found. But I like knowing where it found the message!

Time to quit typing and let Google continue indexing my stuff.