News

The indexing API in 0.3 will change once again to allow multiple design documents and "views" into Lucene. It will also move much of the Lucene-specific stuff into an options object. Please read the TODO for details.

The indexing API in 0.2 has completely changed, please re-read this document and report any surprises/bugs to the bug tracker;

Issue tracking now available at lighthouseapp.

System Requirements

Sun JDK 5 or higher is necessary. Couchdb-lucene is known to be incompatible with OpenJDK as it includes an earlier, and incompatible, version of the Rhino Javascript library.

Build couchdb-lucene

1. Install Maven 2.
2. checkout repository
3. type 'mvn'
4. configure couchdb (see below)

Configure CouchDB

[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.
[external]
fti=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -search
[update_notification]
indexer=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -index
[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

Indexing Strategy

Document Indexing

You must supply a transform function in order to enable couchdb-lucene .

Add a design document called _design/lucene in your database with an attribute called "transform". The value of this attribute is a Javascript function.

The transform function can return null, to prevent indexing, and either a single Document or an array of Documents.

The transform function is called for each document in the database. To pass information to Lucene, you must populate Document instances with data from the original CouchDB document.

The Document class

You may construct a new Document instance with;

var doc = new Document();

Several functions are available that populate a Document.

// Indexed, analyzed but not stored.
doc.field("name", "value"); 
// Indexed, analyzed and stored.
doc.field("name", "value", "yes"); 
// Indexed, stored but not analyzed.
doc.field("name", "value", "yes", "not_analyzed");
// Extract text from the named attachment and index it (but not store it).
doc.attachment("name", "attachment name"); 
// Interpret "value" as a date using the default date formats.
doc.date("name", "value"); 
// intrepret "value" as a date using the supplied format string
// (see Java's SimpleDateFormat class for the syntax).
doc.date("name", "value", "format"); 

Example Transforms

Index Everything

function(doc) {
  var ret = new Document();
  function idx(obj) {
    for (var key in obj) {
      switch (typeof obj[key]) {
        case 'object':
          idx(obj[key]);
          break;
        case 'function':
          break;
        default:
          ret.field(key, obj[key]);
      /* Uncomment next line to include
       * all attributes into a single field.
           */
      // ret.field("all", obj[key]);
          break;
      }
    }
  }
  
  // Index all attributes
  idx(doc);
  // Index all attachments
  for(var a in doc._attachments) {
    ret.attachment("attachment", a);
  }
  return ret;
}

Index Nothing

function(doc) {
  return null;
}

Index Select Fields

function(doc) {
  var result = new Document();
  result.field("subject", doc.subject, "yes");
  result.field("content", doc.content);
  result.date("indexed_at", new Date());
  return result;
}

Index Attachments

function(doc) {
  var result = new Document();
  for(var a in doc._attachments) {
    result.attachment("attachment", a);
  }
  return result;
}

A More Complex Example

function(doc) {
    var mk = function(name, value, group) {
        var ret = new Document(name, value, "yes");
        ret.field("group", group, "yes");
        return ret;
    };
    var ret = [];
    if(doc.type != "reference") return null;
    for(var g in doc.groups) {
        ret.push(mk("library", doc.groups[g].library, g));
        ret.push(mk("method", doc.groups[g].method, g));
        ret.push(mk("target", doc.groups[g].target, g));
    }
    return ret;
}

Attachment Indexing

Couchdb-lucene uses Apache Tika to index attachments of the following types, assuming the correct content_type is set in couchdb;

Supported Formats

• Excel spreadsheets (application/vnd.ms-excel)
• Word documents (application/msword)
• Powerpoint presentations (application/vnd.ms-powerpoint)
• Visio (application/vnd.visio)
• Outlook (application/vnd.ms-outlook)
• XML (application/xml)
• HTML (text/html)
• Images (image/*)
• Java class files
• Java jar archives
• MP3 (audio/mp3)
• OpenDocument (application/vnd.oasis.opendocument.*)
• Plain text (text/plain)
• PDF (application/pdf)
• RTF (application/rtf)

Searching with couchdb-lucene

You can perform all types of queries using Lucene's default query syntax. The _body field is searched by default which will include the extracted text from all attachments. The following parameters can be passed for more sophisticated searches;

All parameters except 'q' are optional.

Special Fields

Dublin Core

All Dublin Core attributes are indexed and stored if detected in the attachment. Descriptions of the fields come from the Tika javadocs.

Examples

https://localhost:5984/dbname/_fti?q=field_name:value
https://localhost:5984/dbname/_fti?q=field_name:value&sort;=other_field
https://localhost:5984/dbname/_fti?debug=true&sort;=billing_size&q;=body:document AND customer:[A TO C]

Search Results Format

Here's an example of a JSON response without sorting;

{
  "q": "+_db:enron +content:enron",
  "skip": 0,
  "limit": 2,
  "total_rows": 176852,
  "search_duration": 518,
  "fetch_duration": 4,
  "rows":   [
        {
      "_id": "hain-m-all_documents-257.",
      "score": 1.601625680923462
    },
        {
      "_id": "hain-m-notes_inbox-257.",
      "score": 1.601625680923462
    }
  ]
}

And the same with sorting;

{
  "q": "+_db:enron +content:enron",
  "skip": 0,
  "limit": 3,
  "total_rows": 176852,
  "search_duration": 660,
  "fetch_duration": 4,
  "sort_order":   [
        {
      "field": "source",
      "reverse": false,
      "type": "string"
    },
        {
      "reverse": false,
      "type": "doc"
    }
  ],
  "rows":   [
        {
      "_id": "shankman-j-inbox-105.",
      "score": 0.6131107211112976,
      "sort_order":       [
        "enron",
        6
      ]
    },
        {
      "_id": "shankman-j-inbox-8.",
      "score": 0.7492915391921997,
      "sort_order":       [
        "enron",
        7
      ]
    },
        {
      "_id": "shankman-j-inbox-30.",
      "score": 0.507369875907898,
      "sort_order":       [
        "enron",
        8
      ]
    }
  ]
}

Fetching information about the index

Calling couchdb-lucene without arguments returns a JSON object with information about the index.

https://127.0.0.1:5984/enron/_fti

returns;

{"doc_count":517350,"doc_del_count":1,"disk_size":318543045}

Working With The Source

To develop "live", type "mvn dependency:unpack-dependencies" and change the external line to something like this;

fti=/usr/bin/java -cp /path/to/couchdb-lucene/target/classes:\
/path/to/couchdb-lucene/target/dependency com.github.rnewson.couchdb.lucene.Main

You will need to restart CouchDB if you change couchdb-lucene source code but this is very fast.

Configuration

couchdb-lucene respects several system properties;

You can override these properties like this;

fti=/usr/bin/java -Dcouchdb.lucene.dir=/tmp \
-cp /home/rnewson/Source/couchdb-lucene/target/classes:\
/home/rnewson/Source/couchdb-lucene/target/dependency\
com.github.rnewson.couchdb.lucene.Main

Basic Authentication

If you put couchdb behind an authenticating proxy you can still configure couchdb-lucene to pull from it by specifying additional system properties. Currently only Basic authentication is supported.

IPv6

The default for couchdb.url is problematic on an IPv6 system. Specify -Dcouchdb.url=https://[::1]:5984 to resolve it.