News

The indexing API in 0.3 has changed since 0.2 to allow multiple design documents and "views" into Lucene. It also moves the Lucene-specific stuff into an options object.

Issue Tracking

Issue tracking at github.

System Requirements

Sun JDK 5 or higher is recommended.

Couchdb-lucene is known to be incompatible with some versions of OpenJDK as it includes an earlier, and incompatible, version of the Rhino Javascript library. The version in Ubuntu 8.10 (6b12-0ubuntu6.4) is known to work and it uses Rhino 1.7R1.

Build couchdb-lucene

1. Install Maven 2.
2. checkout repository
3. type 'mvn'
4. configure couchdb (see below)

Configure CouchDB

[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.
[external]
fti=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -search
[update_notification]
indexer=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -index
[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

Indexing Strategy

Document Indexing

You must supply a index function in order to enable couchdb-lucene as by default, nothing will be indexed.

You may add any number of index views in any number of design documents. All searches will be constrained to documents emitted by the index functions.

Declare your functions as follows;

{
  "fulltext": {
    "by_subject": {
      "defaults": { "store":"yes" },
      "index":"function(doc) { var ret=new Document(); ret.add(doc.subject); return ret }"
    },
    "by_content": {
      "defaults": { "store":"no" },
      "index":"function(doc) { var ret=new Document(); ret.add(doc.content); return ret }"
    }
  }
}

A fulltext object contains multiple index view declarations. An index view consists of;

defaults

The default for numerous indexing options can be overridden here. A full list of options follows.

index

The indexing function itself, documented below.

The Defaults Object

The following indexing options can be defaulted;

name	description	available options	default
field	the field name to index under	user-defined	default
store	whether the data is stored. The value will be returned in the search result.	yes, no	no
index	whether (and how) the data is indexed	analyzed, analyzed_no_norms, no, not_analyzed, not_analyzed_no_norms	analyzed

The Document class

You may construct a new Document instance with;

var doc = new Document();

Data may be added to this document with the add method which takes an optional second object argument that can override any of the above default values. The data is usually interpreted as a String but couchdb-lucene provides special handling if a Javascript Date object is passed. Specifically, the date is indexed as a numeric value, which allows correct sorting, and stored (if requested) in ISO 8601 format (with a timezone marker).

// Add with all the defaults.
doc.add("value");
// Add a subject field.
doc.add("this is the subject line.", {"field":"subject"});
// Add but ensure it's stored.
doc.add("value", {"store":"yes"});
// Add but don't analyze.
doc.add("don't analyze me", {"index":"not_analyzed"});
// Extract text from the named attachment and index it (but not store it).
doc.attachment("attachment name", {"field":"attachments"});

Example Transforms

Index Everything

function(doc) {
    var ret = new Document();
    function idx(obj) {
    for (var key in obj) {
        switch (typeof obj[key]) {
        case 'object':
        idx(obj[key]);
        break;
        case 'function':
        break;
        default:
        ret.add(obj[key]);
        break;
        }
    }
    };
    idx(doc);
    if (doc._attachments) {
    for (var i in doc._attachments) {
        ret.attachment("attachment", i);
    }
    }
    
    return ret;
}

Index Nothing

function(doc) {
  return null;
}

Index Select Fields

function(doc) {
  var result = new Document();
  result.add(doc.subject, {"field":"subject", "store":"yes"});
  result.add(doc.content, {"field":"subject"});
  result.add({"field":"indexed_at"});
  return result;
}

Index Attachments

function(doc) {
  var result = new Document();
  for(var a in doc._attachments) {
    result.add_attachment(a, {"field":"attachment"});
  }
  return result;
}

A More Complex Example

function(doc) {
    var mk = function(name, value, group) {
        var ret = new Document();
        ret.add(value, {"field": group, "store":"yes"});
        ret.add(group, {"field":"group", "store":"yes"});
        return ret;
    };
    var ret = [];
    if(doc.type != "reference") return null;
    for(var g in doc.groups) {
        ret.add(mk("library", doc.groups[g].library, g));
        ret.add(mk("method", doc.groups[g].method, g));
        ret.add(mk("target", doc.groups[g].target, g));
    }
    return ret;
}

Attachment Indexing

Couchdb-lucene uses Apache Tika to index attachments of the following types, assuming the correct content_type is set in couchdb;

Supported Formats

• Excel spreadsheets (application/vnd.ms-excel)
• Word documents (application/msword)
• Powerpoint presentations (application/vnd.ms-powerpoint)
• Visio (application/vnd.visio)
• Outlook (application/vnd.ms-outlook)
• XML (application/xml)
• HTML (text/html)
• Images (image/*)
• Java class files
• Java jar archives
• MP3 (audio/mp3)
• OpenDocument (application/vnd.oasis.opendocument.*)
• Plain text (text/plain)
• PDF (application/pdf)
• RTF (application/rtf)

Searching with couchdb-lucene

You can perform all types of queries using Lucene's default query syntax. The _body field is searched by default which will include the extracted text from all attachments. The following parameters can be passed for more sophisticated searches;

q

the query to run (e.g, subject:hello). If not specified, the default field is searched.

sort

the comma-separated fields to sort on. Prefix with / for ascending order and \ for descending order (ascending is the default if not specified).

limit

the maximum number of results to return

skip

the number of results to skip

include_docs

whether to include the source docs

stale=ok

If you set the stale option ok, couchdb-lucene may not perform any refreshing on the index. Searches may be faster as Lucene caches important data (especially for sorting). A query without stale=ok will use the latest data committed to the index.

debug

if false, a normal application/json response with results appears. if true, an pretty-printed HTML blob is returned instead.

rewrite

(EXPERT) if true, returns a json response with a rewritten query and term frequencies. This allows correct distributed scoring when combining the results from multiple nodes.

All parameters except 'q' are optional.

Special Fields

_db

The source database of the document.

_id

The _id of the document.

Dublin Core

All Dublin Core attributes are indexed and stored if detected in the attachment. Descriptions of the fields come from the Tika javadocs.

_dc.contributor

An entity responsible for making contributions to the content of the resource.

_dc.coverage

The extent or scope of the content of the resource.

_dc.creator

An entity primarily responsible for making the content of the resource.

_dc.date

A date associated with an event in the life cycle of the resource.

_dc.description

An account of the content of the resource.

_dc.format

Typically, Format may include the media-type or dimensions of the resource.

_dc.identifier

Recommended best practice is to identify the resource by means of a string or number conforming to a formal identification system.

_dc.language

A language of the intellectual content of the resource.

_dc.modified

Date on which the resource was changed.

_dc.publisher

An entity responsible for making the resource available.

_dc.relation

A reference to a related resource.

_dc.rights

Information about rights held in and over the resource.

_dc.source

A reference to a resource from which the present resource is derived.

_dc.subject

The topic of the content of the resource.

_dc.title

A name given to the resource.

_dc.type

The nature or genre of the content of the resource.

Examples

https://localhost:5984/dbname/_fti/design_doc/view_name?q=field_name:value
https://localhost:5984/dbname/_fti/design_doc/view_name?q=field_name:value&sort;=other_field
https://localhost:5984/dbname/_fti/design_doc/view_name?debug=true&sort;=billing_size&q;=body:document AND customer:[A TO C]

Search Results Format

The search result contains a number of fields at the top level, in addition to your search results.

q

The query that was executed.

etag

An opaque token that reflects the current version of the index. This value is also returned in an ETag header to facilitate HTTP caching.

skip

The number of initial matches that was skipped.

limit

The maximum number of results that can appear.

total_rows

The total number of matches for this query.

search_duration

The number of milliseconds spent performing the search.

fetch_duration

The number of milliseconds spent retrieving the documents.

rows

The search results array, described below.

The search results array

The search results arrays consists of zero, one or more objects with the following fields;

id

The unique identifier for this match.

score

The normalized score (0.0-1.0, inclusive) for this match

fields

All the fields that were stored with this match

doc

The original document from couch, if requested with include_docs=true

Here's an example of a JSON response without sorting;

{
  "q": "+content:enron",
  "skip": 0,
  "limit": 2,
  "total_rows": 176852,
  "search_duration": 518,
  "fetch_duration": 4,
  "rows":   [
        {
      "id": "hain-m-all_documents-257.",
      "score": 1.601625680923462
    },
        {
      "id": "hain-m-notes_inbox-257.",
      "score": 1.601625680923462
    }
  ]
}

And the same with sorting;

{
  "q": "+content:enron",
  "skip": 0,
  "limit": 3,
  "total_rows": 176852,
  "search_duration": 660,
  "fetch_duration": 4,
  "sort_order":   [
        {
      "field": "source",
      "reverse": false,
      "type": "string"
    },
        {
      "reverse": false,
      "type": "doc"
    }
  ],
  "rows":   [
        {
      "id": "shankman-j-inbox-105.",
      "score": 0.6131107211112976,
      "sort_order":       [
        "enron",
        6
      ]
    },
        {
      "id": "shankman-j-inbox-8.",
      "score": 0.7492915391921997,
      "sort_order":       [
        "enron",
        7
      ]
    },
        {
      "id": "shankman-j-inbox-30.",
      "score": 0.507369875907898,
      "sort_order":       [
        "enron",
        8
      ]
    }
  ]
}

Fetching information about the index

Calling couchdb-lucene without arguments returns a JSON object with information about the whole index.

https://127.0.0.1:5984/enron/_fti

returns;

{"doc_count":517350,"doc_del_count":1,"disk_size":318543045}

Working With The Source

To develop "live", type "mvn dependency:unpack-dependencies" and change the external line to something like this;

fti=/usr/bin/java -cp /path/to/couchdb-lucene/target/classes:\
/path/to/couchdb-lucene/target/dependency com.github.rnewson.couchdb.lucene.Main

You will need to restart CouchDB if you change couchdb-lucene source code but this is very fast.

Configuration

couchdb-lucene respects several system properties;

couchdb.url

the url to contact CouchDB with (default is "https://localhost:5984")

couchdb.lucene.dir

specify the path to the lucene indexes (the default is to make a directory called 'lucene' relative to couchdb's current working directory.

couchdb.log.dir

specify the directory of the log file (which is called couchdb-lucene.log), defaults to the platform-specific temp directory.

You can override these properties like this;

fti=/usr/bin/java -Dcouchdb.lucene.dir=/tmp \
-cp /home/rnewson/Source/couchdb-lucene/target/classes:\
/home/rnewson/Source/couchdb-lucene/target/dependency\
com.github.rnewson.couchdb.lucene.Main

Basic Authentication

If you put couchdb behind an authenticating proxy you can still configure couchdb-lucene to pull from it by specifying additional system properties. Currently only Basic authentication is supported.

couchdb.user

the user to authenticate as.

couchdb.password

the password to authenticate with.

IPv6

The default for couchdb.url is problematic on an IPv6 system. Specify -Dcouchdb.url=https://[::1]:5984 to resolve it.