CARVIEW |
rnewson / couchdb-lucene
- Source
- Commits
- Network (15)
- Issues (6)
- Downloads (7)
- Graphs
-
Branch:
carrot2
name | age | message | |
---|---|---|---|
![]() |
LICENSE | Fri Mar 13 17:19:09 -0700 2009 | fix license file [rnewson] |
![]() |
README.md | Mon Mar 16 15:57:49 -0700 2009 | update README for basic auth. [rnewson] |
![]() |
TODO | Mon Mar 16 17:10:32 -0700 2009 | Merge branch 'master' into carrot2 Conflicts: ... [rnewson] |
![]() |
federation.rb | Wed Mar 11 17:40:30 -0700 2009 | highest score first, obviously... [rnewson] |
![]() |
pom.xml | Mon Mar 09 16:34:43 -0700 2009 | Merge branch 'master' into carrot2 Conflicts: ... [rnewson] |
![]() |
src/ | Mon Mar 16 17:10:32 -0700 2009 | Merge branch 'master' into carrot2 Conflicts: ... [rnewson] |
News
Issue tracking now available at lighthouseapp.
Build couchdb-lucene
- Install Maven 2.
- checkout repository
- type 'mvn'
- configure couchdb (see below)
Configure CouchDB
[couchdb] os_process_timeout=60000 ; increase the timeout from 5 seconds. [external] fti=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -search [update_notification] indexer=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -index [httpd_db_handlers] _fti = {couch_httpd_external, handle_external_req, <<"fti">>}
Indexing Strategy
Document Indexing
By default all attributes are indexed. You can customize this process by adding a design document at _design/lucene. You must supply an attribute called "transform" which takes and returns a document. For example;
{ "transform":"function(doc) { return doc; }" }
The function is evaluated by Rhino. You may add, modify and remove any attributes. Additionally, returning null will exclude the document from indexing entirely.
Attachment Indexing
Couchdb-lucene uses Apache Tika to index attachments of the following types, assuming the correct content_type is set in couchdb;
Supported Formats
- Excel spreadsheets (application/vnd.ms-excel)
- Word documents (application/msword)
- Powerpoint presentations (application/vnd.ms-powerpoint)
- Visio (application/vnd.visio)
- Outlook (application/vnd.ms-outlook)
- XML (application/xml)
- HTML (text/html)
- Images (image/*)
- Java class files
- Java jar archives
- MP3 (audio/mp3)
- OpenDocument (application/vnd.oasis.opendocument.*)
- Plain text (text/plain)
- PDF (application/pdf)
- RTF (application/rtf)
Searching with couchdb-lucene
You can perform all types of queries using Lucene's default query syntax. The following parameters can be passed for more sophisticated searches;
All parameters except 'q' are optional.
Special Fields
Dublin Core
All Dublin Core attributes are indexed and stored if detected in the attachment. Descriptions of the fields come from the Tika javadocs.
Examples
https://localhost:5984/dbname/_fti?q=field_name:value https://localhost:5984/dbname/_fti?q=field_name:value&sort;=other_field https://localhost:5984/dbname/_fti?debug=true&sort;=billing_size&q;=body:document AND customer:[A TO C]
Search Results Format
Here's an example of a JSON response without sorting;
{ "q": "+_db:enron +content:enron", "skip": 0, "limit": 2, "total_rows": 176852, "search_duration": 518, "fetch_duration": 4, "rows": [ { "_id": "hain-m-all_documents-257.", "score": 1.601625680923462 }, { "_id": "hain-m-notes_inbox-257.", "score": 1.601625680923462 } ] }
And the same with sorting;
{ "q": "+_db:enron +content:enron", "skip": 0, "limit": 3, "total_rows": 176852, "search_duration": 660, "fetch_duration": 4, "sort_order": [ { "field": "source", "reverse": false, "type": "string" }, { "reverse": false, "type": "doc" } ], "rows": [ { "_id": "shankman-j-inbox-105.", "score": 0.6131107211112976, "sort_order": [ "enron", 6 ] }, { "_id": "shankman-j-inbox-8.", "score": 0.7492915391921997, "sort_order": [ "enron", 7 ] }, { "_id": "shankman-j-inbox-30.", "score": 0.507369875907898, "sort_order": [ "enron", 8 ] } ] }
Fetching information about the index
Calling couchdb-lucene without arguments returns a JSON object with information about the index.
https://127.0.0.1:5984/enron/_fti
returns;
{"doc_count":517350,"doc_del_count":1,"disk_size":318543045}
Working With The Source
To develop "live", type "mvn dependency:unpack-dependencies" and change the external line to something like this;
fti=/usr/bin/java -cp /path/to/couchdb-lucene/target/classes:\ /path/to/couchdb-lucene/target/dependency org.apache.couchdb.lucene.Main
You will need to restart CouchDB if you change couchdb-lucene source code but this is very fast.
Configuration
couchdb-lucene respects several system properties;
You can override these properties like this;
fti=/usr/bin/java -D couchdb.lucene.dir=/tmp \ -cp /home/rnewson/Source/couchdb-lucene/target/classes:\ /home/rnewson/Source/couchdb-lucene/target/dependency\ org.apache.couchdb.lucene.Main
Basic Authentication
If you put couchdb behind an authenticating proxy you can still configure couchdb-lucene to pull from it by specifying additional system properties. Currently only Basic authentication is supported.