WARNING

This branch (master) contains the developmenet version of the next release (0.5). It is not yet stable or feature complete and should not be used for anything but testing.

I strongly recommend using the 0.4 release (available at GitHub, please see the README for unpacking details) as the 0.5 work will occur on trunk and will definitely have extended periods of instability and broken-ness.

Version Compatibility

Issue Tracking

Issue tracking at github.

Minimum System Requirements

JDK6 is required; the Sun version is recommended as it's regularly tested against.

10.5 is the minimum version of OSX for the above reason.

Build couchdb-lucene

1. Install Maven 2.
2. checkout repository
3. type 'mvn'
4. configure couchdb (see below)

Configure CouchDB

[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.
[external]
fti=/path/to/couchdb-lucene/couchdb-external-hook
[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

Configure couchdb-lucene

From 0.5 onwards, couchdb-lucene runs in a single, standalone JVM. As such, you can choose to locate your couchdb-lucene server on a different machine to couchdb if you wish, or keep it on the same machine, it's your call. In either case, you need to tell couchdb-lucene where couch is. Edit the couchdb-lucene.properties file and fill in the host and port details;

couchdb.url=https://localhost:5984

Start couchdb-lucene

To start couchdb-lucene, run;

couchdb-lucene start

To stop couchdb-lucene, run;

couchdb-lucene stop

Indexing Strategy

Document Indexing

You must supply a index function in order to enable couchdb-lucene as, by default, nothing will be indexed. To suppress a document from the index, return null. It's more typical to return a single Document object which contains everything you'd like to query and retrieve. You may also return an array of Document objects if you wish.

You may add any number of index views in any number of design documents. All searches will be constrained to documents emitted by the index functions.

Here's an complete example of a design document with couchdb-lucene features:

{
    "_id":"_design/a_design_document_with_any_name_you_like",
    "fulltext": {
        "by_subject": {
            "defaults": { "store":"yes" },
            "index":"function(doc) { var ret=new Document(); ret.add(doc.subject); return ret }"
        },
        "by_content": {
            "defaults": { "store":"no" },
            "index":"function(doc) { var ret=new Document(); ret.add(doc.content); return ret }"
        }
    }
}

Here are some example URL's for the given design document;

https://localhost:5984/database/_fti/lucene/by_subject?q=hello
https://localhost:5984/database/_fti/lucene/by_content?q=hello

A fulltext object contains multiple index view declarations. An index view consists of;

The Defaults Object

The following indexing options can be defaulted;

The Analyzer Option

Lucene has numerous ways of converting free-form text into tokens, these classes are called Analyzer's. By default, the StandardAnalyzer is used which lower-cases all text, drops common English words ("the", "and", and so on), among other things. This processing might not always suit you, so you can choose from several others by setting the "analyzer" field to one of the following values;

• brazilian
• chinese
• cjk
• czech
• dutch
• english
• french
• german
• keyword
• porter
• russian
• simple
• standard
• thai

Note: You must also supply analyzer=<analyzer_name> as a query parameter to ensure that queries are processed correctly.

The Document class

You may construct a new Document instance with;

var doc = new Document();

Data may be added to this document with the add method which takes an optional second object argument that can override any of the above default values.

// Add with all the defaults.
doc.add("value");
// Add a numeric field.
doc.add(35, {"type":"integer"});
// Add a date field.
doc.add(new Date("2009-01-01"), {"type":"date"});
// Add a date field (object must be a Date object
// Add a subject field.
doc.add("this is the subject line.", {"field":"subject"});
// Add but ensure it's stored.
doc.add("value", {"store":"yes"});
// Add but don't analyze.
doc.add("don't analyze me", {"index":"not_analyzed"});
// Extract text from the named attachment and index it (but not store it).
doc.attachment("attachment name", {"field":"attachments"});

Example Transforms

Index Everything

function(doc) {
    var ret = new Document();
    function idx(obj) {
    for (var key in obj) {
        switch (typeof obj[key]) {
        case 'object':
        idx(obj[key]);
        break;
        case 'function':
        break;
        default:
        ret.add(obj[key]);
        break;
        }
    }
    };
    idx(doc);
    if (doc._attachments) {
    for (var i in doc._attachments) {
        ret.attachment("attachment", i);
    }
    }
    return ret;
}

Index Nothing

function(doc) {
  return null;
}

Index Select Fields

function(doc) {
  var result = new Document();
  result.add(doc.subject, {"field":"subject", "store":"yes"});
  result.add(doc.content, {"field":"subject"});
  result.add({"field":"indexed_at"});
  return result;
}

Index Attachments

function(doc) {
  var result = new Document();
  for(var a in doc._attachments) {
    result.add_attachment(a, {"field":"attachment"});
  }
  return result;
}

A More Complex Example

function(doc) {
    var mk = function(name, value, group) {
        var ret = new Document();
        ret.add(value, {"field": group, "store":"yes"});
        ret.add(group, {"field":"group", "store":"yes"});
        return ret;
    };
    var ret = [];
    if(doc.type != "reference") return null;
    for(var g in doc.groups) {
        ret.add(mk("library", doc.groups[g].library, g));
        ret.add(mk("method", doc.groups[g].method, g));
        ret.add(mk("target", doc.groups[g].target, g));
    }
    return ret;
}

Attachment Indexing

Couchdb-lucene uses Apache Tika to index attachments of the following types, assuming the correct content_type is set in couchdb;

Supported Formats

• Excel spreadsheets (application/vnd.ms-excel)
• HTML (text/html)
• Images (image/*)
• Java class files
• Java jar archives
• MP3 (audio/mp3)
• OpenDocument (application/vnd.oasis.opendocument.*)
• Outlook (application/vnd.ms-outlook)
• PDF (application/pdf)
• Plain text (text/plain)
• Powerpoint presentations (application/vnd.ms-powerpoint)
• RTF (application/rtf)
• Visio (application/vnd.visio)
• Word documents (application/msword)
• XML (application/xml)

Searching with couchdb-lucene

You can perform all types of queries using Lucene's default query syntax. The _body field is searched by default which will include the extracted text from all attachments. The following parameters can be passed for more sophisticated searches;

All parameters except 'q' are optional.

Special Fields

Dublin Core

All Dublin Core attributes are indexed and stored if detected in the attachment. Descriptions of the fields come from the Tika javadocs.

Examples

https://localhost:5984/dbname/_fti/design_doc/view_name?q=field_name:value
https://localhost:5984/dbname/_fti/design_doc/view_name?q=field_name:value&sort;=other_field
https://localhost:5984/dbname/_fti/design_doc/view_name?debug=true&sort;=billing_size&q;=body:document AND customer:[A TO C]

Search Results Format

The search result contains a number of fields at the top level, in addition to your search results.

The search results array

The search results arrays consists of zero, one or more objects with the following fields;

Here's an example of a JSON response without sorting;

{
  "q": "+content:enron",
  "skip": 0,
  "limit": 2,
  "total_rows": 176852,
  "search_duration": 518,
  "fetch_duration": 4,
  "rows":   [
        {
      "id": "hain-m-all_documents-257.",
      "score": 1.601625680923462
    },
        {
      "id": "hain-m-notes_inbox-257.",
      "score": 1.601625680923462
    }
  ]
}

And the same with sorting;

{
  "q": "+content:enron",
  "skip": 0,
  "limit": 3,
  "total_rows": 176852,
  "search_duration": 660,
  "fetch_duration": 4,
  "sort_order":   [
        {
      "field": "source",
      "reverse": false,
      "type": "string"
    },
        {
      "reverse": false,
      "type": "doc"
    }
  ],
  "rows":   [
        {
      "id": "shankman-j-inbox-105.",
      "score": 0.6131107211112976,
      "sort_order":       [
        "enron",
        6
      ]
    },
        {
      "id": "shankman-j-inbox-8.",
      "score": 0.7492915391921997,
      "sort_order":       [
        "enron",
        7
      ]
    },
        {
      "id": "shankman-j-inbox-30.",
      "score": 0.507369875907898,
      "sort_order":       [
        "enron",
        8
      ]
    }
  ]
}

Content-Type of response

The Content-Type of the response is negotiated via the Accept request header like CouchDB itself. If the Accept header includes "application/json" then that is also the Content-Type of the response. If not, "text/plain;charset=utf-8" is used.

Fetching information about the index

Calling couchdb-lucene without arguments returns a JSON object with information about the whole index.

https://127.0.0.1:5984/enron/_fti

returns;

{"doc_count":517350,"doc_del_count":1,"disk_size":318543045}

Working With The Source

To develop "live", type "mvn dependency:unpack-dependencies" and change the external line to something like this;

fti=/usr/bin/java -server -cp /path/to/couchdb-lucene/target/classes:\
/path/to/couchdb-lucene/target/dependency com.github.rnewson.couchdb.lucene.Main

You will need to restart CouchDB if you change couchdb-lucene source code but this is very fast.

Configuration

couchdb-lucene is configured from the couchdb ini files

[lucene]
; Send lucene indexes elsewhere, defaults to hidden subdir under database_dir
;lucene_dir=
; Send logs elsewhere, defaults to same dir as couchdb log file.
;log_dir=
; alter (soft) maximum ram for indexing.
;max_ram=16
; Specify the default operator for boolean queries. If not specified, or if the setting is not recognized, the defaul is "OR". The other choice is "AND".
; default_boolean_operator="OR"

Basic Authentication

If you put couchdb behind an authenticating proxy you can still configure couchdb-lucene to pull from it by specifying additional system properties. Currently only Basic authentication is supported.