Nutch

For the past five weeks I've been working with the Nutch open source search engine and web crawler. Nutch is at the point that Drupal was a year and a half ago. For Drupal, that "meant lots of potential," "poised for explosive growth," and "lacking in documentation." We had the first Drupal conference in February of 2005, and though we expected only a handful of people, many more showed up! I hear that a similar thing happened at the Nutch meetup recently.

Doug Cutting, currently at Yahoo!, is the lead developer. The Nutch project also resulted in an interesting distributed system called Hadoop.

Here's a site running Drupal as the front end and Nutch as the back end. There is currently work being done to integrate Drupal vocabularies and terms with Nutch's search.