'Searching the deep web' - quo medicine?

Posted on October 10, 2008 - 03:36

Alex Wright - he of the 'Web time forgot' in the NYTimes and the monograph 'Glut:
mastering information through the ages - has written a very readable piece entitled Searching the Deep Web in the Communications of the ACM (October 2008). Wright has written about the deep Web before, in fact, back in 2004 in Salon magazine.

Wright continues to characterize search techniques that get at the deep web in a way that resonates with much of my thinking.  However, I tend to conflate the deep web with the idea in scholarly circles and medicine of seaching for the 'grey literature'. That may be an inaccurate way of looking at the two concepts.

Grey literature has some specific connotations in academia in that some documents are merely hard to find because they are published unconventionally and unindexed in traditional abstracting services.

Conversely, the deep web does seem to conjure up a specific kind of searching for documents, especially for those hidden within the deepest parts of the web. (Wright uses the trawling the ocean's floor metaphor.)

It'll comes as little surprise to web watchers that Google has been working on deep web searching. Wright uses that to springboard into a discussion of the semantic web. What I find intriguing about this whole area of the web's future is how it moves into using machine-based reasoning to link to similar documents in new and meaningful ways.

Intelligent, meaningful searching - something we need in medicine rather badly.