XQuery and XPath Full Text 1.0

2.2.1 Description

A full-text contains expression returns a Boolean value. It returns true if there is some item returned by the RangeExpr that, after tokenization, matches the full-text selection FTSelection. Since tokenization includes tokens derived only from the string values of items, a full-text contains expression searches the text of element nodes and of their descendant elements. The string value of other kinds of nodes, such as attributes and comments, will not be included unless the attribute or comment node itself is the target (RangeExpr) of the full-text contains expression. See Section 3 Full-Text Selections for more details. For the purpose of determining a match, certain descendants of nodes (identified by FTIgnoreOption) in the RangeExpr may be ignored, as specified in Section 3.7 Ignore Option.

An XQuery and XPath Full Text 1.0 processor SHOULD try to use the information available in xml:lang for processing of collations, as well as the various match options defined in Section 3.4 Match Options.

2.2.2 Examples

The following example in XQuery 1.0 Full Text returns the author of each book with a title containing a token with the same root as dog and the token cat.

for $b in /books/book
where $b/title contains text ("dog" using stemming) ftand "cat" 
return $b/author

The same example in XPath 2.0 Full Text is written as:

/books/book[title contains text ("dog" using stemming) ftand "cat"]/author

In the next example a ComparisonExpr is combined with an FTContainsExpr using the logical XQuery operator and. The query selects books that have a price of less than 50 and a title which contains a token with the same root as train:

/books/book[price < 50 and title contains text ("train" using stemming)]

The following example shows the combination of two contains text expressions the results of which are compared using the not-equals operator. The query selects books where either the title contains the token dog and the token cat and the content does not contain a token with the same root as train, or where the title fails to have one of the matching tokens but the content does:

/books/book[title contains text "dog" ftand "cat" ne
            content contains text ("train" using stemming)]

2.3.1 Using Weights Within a Scored FTContainsExpr

[Definition: Scoring may be influenced by adding weight declarations to search tokens, phrases, and expressions.] Weight declarations are introduced syntactically in the FTPrimaryWithOptions production, described in Section 3.1.1 Weights.

The weights assigned are not related to any absolute standard, but typically have a relationship to other weights within the same FTContains expression.

The effect of weights on the resulting score is implementation-dependent. However, scoring algorithms MUST conform to the constraints: that when no explicit weight is specified, the default weight is 1.0.

The following example illustrates how different weights can be used for different search terms.

for $b in /books/book
let score $s := $b/content contains text ("web site" weight {0.5})
                                ftand ("usability" weight {2})
return <result score="{$s}">{$b}</result>

3.1.1 Weights

As shown in the grammar, a full-text primary selection may be optionally followed by match options (which are discussed in 3.4 Match Options) and by a "weight" value that is specified using an expression enclosed in braces. The Expr is evaluated as if it were an argument to a function with an expected type xs:double. The weight MUST have an absolute value between 0.0 and 1000.0 inclusive. If the absolute value of the weight is greater than 1000.0, an error is raised: [err:FTDY0016].

3.4.1 Language Option

[Definition: A language option modifies token matching by specifying the language of search tokens and phrases.]

The StringLiteral following the keyword language designates one language. It must be castable to xs:language; otherwise, an error is raised: [err:XPTY0004]^XP.

The "language" option influences tokenization, stemming, and stop words in an implementation-defined way. The "language" option MAY influence the behavior of other match options in an implementation-defined way.

The set of standardized language identifiers is defined in [BCP 47]. The set of valid language identifiers among the standardized set is implementation-defined. An implementation MAY choose to use private extensions introduced by a singleton 'x' for additional language identifiers, or other singletons for registered extensions as described in sec. 2.2.6 of [BCP 47]. It is implementation-defined what additional language identifiers, if any, are valid. If an invalid language identifier is specified, then the behavior is implementation-defined. If the implementation chooses to raise an error in that case, it must raise [err:FTST0009].

The default language is specified in the static context.

When an XQuery and XPath Full Text processor evaluates text in a document that is governed by an xml:lang attribute and the portion of the full-text query doing that evaluation contains an FTLanguageOption that specifies a different language from the language specified by the governing xml:lang attribute, the language-related behavior of that full-text query is implementation-defined.

This is an example where the language option is used to select the appropriate stop word list:

//book[@number="1"]/content//p contains text "salon de thé"
using stop words default using language "fr"

3.4.2 Wildcard Option

[Definition: A wildcard option modifies token and phrase matching by specifying whether or not wildcards are recognized in query strings.]

When the "wildcards" option is used, wildcard syntax may be included within query strings. A wildcard consists of an indicator (a period or full stop, "."), optionally followed by a qualifier. Each wildcard in a query token will match zero or more characters within a token in the text being searched, as described below. The number of characters that can be matched depends on the qualifier. The forms of wildcard syntax specified by this document are:

1. A single period, without any qualifiers: Matches a single arbitrary character.

2. A period immediately followed by a single question mark, "?": Matches either no characters or one character.

3. A period immediately followed by a single asterisk, "*": Matches zero or more characters.

4. A period immediately followed by a single plus sign, "+": Matches one or more characters.

5. A period immediately followed by a sequence of characters that matches the regular expression {[0-9]+,[0-9]+}: Matches a number of characters, where the number is no less than the number represented by the series of digits before the comma, and no greater than the number represented by the series of digits following the comma.

   If a period in the query string is immediately followed by a left curly brace, but the subsequent characters do not conform to the given regular expression, then an error is raised: [err:FTDY0020].

A question mark, asterisk, plus sign, or left curly brace that is not immediately preceded by a period is not treated as a qualifier. For example, using the sample tokenization and "wildcards", the query string "wil+" does not match the search text "will" or "willlllll", but only matches the search text "wil". (The sample tokenization treats the plus sign as punctuation.)

When "wildcards" is used, any character in a query string can be "escaped" by immediately preceding it with a backslash, "\". That is, a backslash immediately followed by any character represents that character literally, preventing any special interpretation that the "wildcards" option might otherwise attach to it. In particular:

1. Escaping a period prevents its interpretation as a wildcard.

2. Escaping a question mark, asterisk, plus sign, or left curly brace ensures that it is not interpreted as a qualifier.

3. An escaped backslash ("\\") represents a literal backslash.

4. If a query string is terminated by an unescaped backslash, an error is raised: [err:FTDY0020].

When the "no wildcards" option is used, no wildcards are recognized in query strings. Periods, question marks, asterisks, plus signs, left curly braces, and backslashes are always recognized as ordinary text characters.

The default is "no wildcards".

The following expression returns true, because the p element contains "well":

//book[@number="1"]/p contains text "w.ll" using wildcards

The following expression returns true, because the title element contains "site":

//book[@number="1"]/title contains text ".?site" using wildcards

The following expression returns true, because the title element contains "improving":

//book[@number="1"]/title contains text "improv.*" using wildcards

The following expression raises error [err:FTDY0020], because the query string uses incorrect syntax:

//book[@number="1"]/p contains text "wi.{5,7]" using wildcards

The following expression returns true, because the title contains "site":

//book[@number="1"]/title contains text "\s\i\t\e" using wildcards

The following expression returns true, because the title contains "Usability":

//book[@number="1"]/title contains text "Usab.+\\" using wildcards

(Note that "\\" represents a literal backslash, which the sample tokenization treats as punctuation.)

The following expression raises error [err:FTDY0020], because the query string ends with an unescaped backslash:

//book[@number="1"]/p contains text "will\" using wildcards

The following expression returns false, because the p element does not contain the phrase "w ll":

//book[@number="1"]/p contains text "w.ll" using no wildcards

(Note that, without wildcards, the sample tokenization will treat the period in "w.ll" as punctuation, thus producing "w" and "ll" as separate tokens.)

3.4.3 Thesaurus Option

[Definition: A thesaurus option modifies token and phrase matching by specifying whether a thesaurus is used or not.] If thesauri are used, the thesaurus option specifies information to locate the thesauri either by default or through a URI reference. It also states the relationship to be applied and how many levels within the thesaurus to be traversed.

If the URI specifies a thesaurus that is not found in the statically known thesauri, an error is raised [err:FTST0018].

Thesauri add related tokens and phrases to the query or change query tokens. Thus, the user may narrow, broaden, or otherwise modify the query using synonyms, hypernyms (more generic terms), etc. The search is performed as though the user has specified all related query tokens and phrases in a disjunction (FTOr).

FTThesaurusID specifies the relationship sought between tokens and phrases written in the query and terms in the thesaurus and the number of levels to be queried in hierarchical relationships by including an FTRange "levels". If no levels are specified, the default is to query all levels in hierarchical relationships or to query an implementation-defined number of levels in hierarchical relationships.

Relationships include, but are not limited to, the relationships and their abbreviations presented in [ISO 2788] and their equivalents in other languages. The set of relationships supported by an implementation is implementation-defined, but implementations SHOULD support the relationships defined in [ISO 2788]. The following list of terms have the meanings defined in [ISO 2788]. If a query specifies thesaurus relationships or levels not supported by the thesaurus, or does not specify a relationship, the behavior is implementation-defined.

1. equivalence relationships (synonyms): PREFERRED TERM (USE), NONPREFERRED USED FOR TERM (UF);

2. hierarchical relationships: BROADER TERM (BT), NARROWER TERM (NT), BROADER TERM GENERIC (BTG), NARROWER TERM GENERIC (NTG), BROADER TERM PARTITIVE (BTP), NARROWER TERM PARTITIVE (NTP), TOP Terms (TT); and

3. associative relationships: RELATED TERM (RT).

The "thesaurus" option specifies that string matches include tokens that can be found in one of the specified thesauri. When "default" is used in place of a FTThesaurusID, the thesauri specified in the static context are used, which are either given by the prolog declaration for the thesaurus option, or, if no such declaration exists a system-defined default thesaurus with a system-defined relationship. The default thesaurus may be used in combination with other explicitly specified thesauri.

The "no thesaurus" option specifies that no thesaurus will be used.

The default is "no thesaurus".

The following expression returns true, because it finds a content element containing "task" which the thesaurus identified as a synonym for "duty":

.//book/content contains text "duty" using
thesaurus at "https://bstore1.example.com/UsabilityThesaurus.xml"
relationship "UF"

The following expression returns a book element, because it finds a content element containing "users", which is a narrower term of "people":

doc("https://bstore1.example.com/full-text.xml")
/books/book[./content contains text "people" using
thesaurus at "https://bstore1.example.com/UsabilityThesaurus.xml"
relationship "NT" at most 2 levels]

Assuming the thesaurus available at URL "https://bstore1.example.com/UsabilitySoundex.xml" contains soundex capabilities, the following query returns a book element containing "Marigold" which sounds like "Merrygould":

doc("https://bstore1.example.com/full-text.xml")
/books/book[. contains text "Merrygould" using thesaurus at
"https://bstore1.example.com/UsabilitySoundex.xml" relationship
"sounds like"]

3.4.4 Stemming Option

[Definition: A stemming option modifies token and phrase matching by specifying whether stemming is applied or not. ]

The "stemming" option specifies that matches may contain tokens that have the same stem as the tokens and phrases written in the query. It is implementation-defined what a stem of a token is.

The "no stemming" option specifies that the tokens and phrases are not stemmed.

It is implementation-defined whether the stemming is based on an algorithm, dictionary, or mixed approach.

The default is "no stemming".

The following expression returns true, because the title of the specified book contains "improving" which has the same stem as "improve":

/books/book[@number="1"]/title contains text "improve" using stemming 

3.4.5 Case Option

[Definition: A case option modifies the matching of tokens and phrases by specifying how uppercase and lowercase characters are considered.]

There are four possible character case options:

1. Using the option "case insensitive", tokens and phrases are matched, regardless of the case of characters of the query tokens and phrases.

2. Using the option "case sensitive", tokens and phrases are matched, if and only if the case of their characters is the same as written in the query.

3. Using the option "lowercase", tokens and phrases are matched, if and only if they match the query without regard to character case, but contain only lowercase characters.

4. Using the option "uppercase", tokens and phrases are matched, if and only if they match the query without regard to character case, but contain only uppercase characters.

The default is "case insensitive".

The effect of the case options is also influenced by the query's default collation (see Section 2.1.1 Static Context^XQ and Section 4.4 Default Collation Declaration^XQ). The following table summarizes how these interact.

The following expression returns false, because the title element doesn't contain "usability" in lower-case characters:

//book[@number="1"]/title contains text "Usability" using lowercase 

The following expression returns true, because the character case is not considered:

//book[@number="1"]/title contains text "usability" using case insensitive

3.4.6 Diacritics Option

[Definition: A diacritics option modifies token and phrase matching by specifying how diacritics are considered. ]

There are two possible diacritics options:

1. The option "diacritics" "insensitive" matches tokens and phrases with and without diacritics. Whether diacritics are written in the query or not is not considered.

2. The option "diacritics" "sensitive" matches tokens and phrases only if they contain the diacritics as they are written in the query.

The default is "diacritics insensitive".

The effect of the diacritics options is also influenced by the query's default collation (see Section 2.1.1 Static Context^XQ and Section 4.4 Default Collation Declaration^XQ). The following table summarizes how these interact.

The following expression returns true, because the token "Véra" in the editor element is matched, as the acute accent is not considered in the comparison:

//book[@number="1"]//editor contains text "Vera" using diacritics insensitive

This returns false, because the editor element does not contain the token "Vera" in this exact form, i.e. without any diacritics:

//book[@number="1"]/editors contains text "Vera" using diacritics sensitive

3.4.7 Stop Word Option

[Definition: A stop word option controls matching of tokens by specifying whether stop words are used or not. Stop words are tokens in the query that match any token in the text being searched. ] More precisely, a stop word option defines a collection of stop words according to the rules below. Then, in every FTWords to which the stop word option applies, each query token is checked: if it appears (using an implementation-defined comparison) in the specified collection of stop words, it is considered a stop word.

Normally a stop word matches exactly one token, but there may be implementation-defined conditions, under which a stop word may match a different number of tokens.

FTStopWords specifies the list of stop words either explicitly as a comma-separated list of string literals, or by the keyword at followed by a literal URI. If the URI specifies a list of stop words that is not found in the statically known stop word lists, an error is raised [err:FTST0008]. Whether the stop word list is resolved from the statically known stop word lists or given explicitly, no tokenization is performed on the stop words: they are used as they occur in the list.

Multiple stop word lists may be combined using "union" or "except". The keywords "union" and "except" are applied from left to right. If "union" is specified, every string occurring in the lists specified by the left-hand side or the right-hand side is a stop word. If "except" is specified, only strings occurring in the list specified by the left-hand side but not in the list specified by the right-hand side are stop words.

The "stop words default" option specifies that an implementation-defined collection of stop words is used.

The "no stop words" option specifies that no stop words are used. This is equivalent to specifying an empty list of stop words.

The default is "no stop words".

The following expression returns true, because the document contains the phrase "propagating few errors":

/books/book[@number="1"]//p contains text "propagating of errors"
using stop words ("a", "the", "of") 

Note the asymmetry in the stop word semantics: the property of being a stop word is only relevant to query terms, not to document terms. Hence, it is irrelevant for the above-mentioned match whether "few" is a stop word or not, and on the other hand we do not want the query above to match "propagating" followed by 2 stop words, or even a sequence of 3 stop words in the document.

The following expression returns false. In this case specifying "few" as a stop word has no effect, since "few" does not appear in the query. Although the words "propagating" and "errors" appear in the text being searched, the phrase "propagating errors" cannot be matched, since that phrase does not occur.

/books/book[@number="1"]//p contains text "propagating errors" 
using stop words ("few")

The following expression returns false, because "of" is not in the p element between "propagating" and "errors":

/books/book[@number="1"]//p contains text "propagating of errors" 
using no stop words

The following expression uses the stop words list specified at the URL. Assuming that the specified stop word list contains the word "then", this query is reduced to a query on the phrase "planning X conducting", allowing any token as a substitute for X. It returns a book element, because its content element contains "planning then conducting". It would also return the book if the phrases "planning and conducting" and "planning before conducting" had been in its content:

doc("https://bstore1.example.com/full-text.xml")
/books/book[.//content contains text "planning then 
conducting" using stop words at 
"https://bstore1.example.com/StopWordList.xml"]

The following expression returns books containing "planning then conducting", but not does not return books containing "planning and conducting", since it is exempting "then" from being a stop word:

doc("https://bstore1.example.com/full-text.xml")
/books/book[.//content contains text "planning then conducting"
using stop words at "https://bstore1.example.com/StopWordList.xml"
except ("the", "then")]

3.4.8 Extension Option

[Definition: An extension option is a match option that acts in an implementation-defined way. ]

An extension option consists of an identifying QName and a StringLiteral. Typically, a particular option will be recognized by some implementations and not by others. The syntax is designed so that option declarations can be successfully parsed by all implementations.

The QName of an extension option must resolve to a namespace URI and local name, using the statically known namespaces.

Each implementation recognizes an implementation-defined set of namespace URIs used to denote extension options.

If the namespace part of the QName is not a namespace recognized by the implementation as one used to denote extension option, then the extension option is ignored.

Otherwise, the effect of the extension option, including its error behavior, is implementation-defined. For example, if the local part of the QName is not recognized, or if the StringLiteral does not conform to the rules defined by the implementation for the particular extension option, the implementation may choose whether to report an error, ignore the extension option, or take some other action.

Implementations may impose rules on where particular extension options may appear relative to other match options, and the interpretation of an option declaration may depend on its position.

An extension option must not be used to change the syntax accepted by the processor, or to suppress the detection of static errors. However, it may be used without restriction to modify the set of tokens in the query or how they are matched against tokens in the text being searched. An extension option has the same scope as other match options.

The following examples illustrate several possible uses for extension options:

This extension option is set as part of the static context of all full-text expressions in the module and might be used to ensure that queries are insensitive to Arabic short-vowels.

declare namespace exq = "https://example.org/XQueryImplementation";
declare ft-option using option exq:diacritics "short-vowel insensitive";

This extension option applies only to the matching in the full-text selection in which it is found and might be used to specify how compound words should be matched.

declare namespace exq = "https://example.org/XQueryImplementation";
//para[. contains text
         ("Kinder" ftand "Platz" distance exactly 1 words)
         using stemming
	 using option exq:compounds "distance=1" ]

3.5.1 Or-Selection

[Definition: An or-selection combines two full-text selections using the ftor operator.]

An or-selection finds all matches that satisfy at least one of the operand full-text selections.

The following expression returns the book element written by "Millicent":

//book[.//author contains text "Millicent" ftor "Voltaire"]

3.5.2 And-Selection

[Definition: An and-selection combines two full-text selections using the ftand operator.]

An and-selection finds matches that satisfy all of the operand full-text selections simultaneously. A match of an and-selection is formed by combining matches for each of the operand full-text selections as described in 4.2.6.2 FTAnd.

For example, "usability" ftand "testing" will find two matches in //book[@number="1"]/title: each of the two matches for the FTWords selection "usability" (the two occurrences of "usability" in the string value of the title element) is combined with the single match for the FTWords "testing" (only one occurrence of "testing" in the title). Since the above and-selection has at least one match, the following expression will return "true".

//book[@number="1"]/title contains text ("usability" ftand "testing")

The following expression returns false, because "Millicent" and "Montana" are not contained by the same author element in any book element:

//book/author contains text "Millicent" ftand "Montana"

No author element in any book element contains both "Millicent" and "Montana". Therefore, for any such author element, there are either one match for the FTWords "Millicent" and zero matches for the FTWords "Montana", or vice versa, or no matches for both of them. In any of these cases, the and-selection will have zero matches.

3.5.3 Mild-Not Selection

[Definition: A mild-not selection combines two full-text selections using the not in operator.]

The not in operator is a milder form of the operator combination ftand ftnot. The selection A not in B matches a token sequence that matches A, but not when it is a part of a match of B. In contrast, A ftand ftnot B only finds matches when the token sequence contains A and does not contain B.

As an example, consider a search for "Mexico" not in "New Mexico". This may return, among others, a document which is all about "Mexico" but mentions at the end that "New Mexico was named after Mexico". The occurrence of "Mexico" in "New Mexico" is not considered, but other occurrences of "Mexico" are matched. Note that this document would not be matched by the full-text selection "Mexico" ftand ftnot "New Mexico".

A match to a mild-not selection must contain at least one token that satisfies the first condition and does not satisfy the second condition. If it contains a token that satisfies both the first and the second condition, the token is not considered as a match.

The following expression returns true, because "usability" appears in the title and the p elements and the token within the phrase "Usability Testing" in the title element is not considered:

/books/book contains text "usability" not in "usability testing"

If either operand of a mild-not selection yields an AllMatches that contains a Match that contains a StringExclude, then a dynamic error [err:FTDY0017] is raised.

3.5.4 Not-Selection

[Definition: A not-selection is a full-text selection starting with the prefix operator ftnot.]

A not-selection selects matches that do not satisfy the operand full-text selection. Details about how such matches are constructed are given in 4.2.6.3 FTUnaryNot.

The following expression returns the empty sequence, because all book elements contain "usability":

//book[. contains text ftnot "usability"]

The following expression returns true, because book elements contain "improving" and "usability" but not "improving usability":

//book contains text "improving" ftand
"usability" ftand ftnot "improving usability"

The following expression returns book elements containing "web site usability" but not "usability testing":

//book[title/@shortTitle contains text "web site usability" ftand 
ftnot "usability testing"]

3.6.1 Ordered Selection

[Definition: An ordered selection consists of a full-text selection followed by the postfix operator "ordered".] An ordered selection constrains the order of tokens and phrases to be the same as the order in which they are written in the operand selection.

The default is unordered. Unordered is in effect when ordered is not specified in the query. Unordered cannot be written explicitly in the query.

An ordered selection selects matches which satisfy the operand full-text selection and which also satisfy the following constraint: the order that the matching tokens or phrases have in the text being searched is the same order that the corresponding query tokens or phrases have in the operand selection. In both cases, the ordering is determined from the minimum start positions of the constituent tokens.

The following expression returns true, because titles of book elements contain "web site" and "usability" in the order in which they are written in the query, i.e., "web site" must precede "usability":

//book/title contains text ("web site" ftand "usability") ordered

The following expression returns false, because although "Montana" and "Millicent" both appear in the book element, they do not appear in the order they are written in the query:

//book[@number="1"] contains text ("Montana" ftand "Millicent") ordered

3.6.2 Window Selection

[Definition: A window selection consists of a full-text selection followed by one of the (complex) postfix operators derived from FTWindow.] A window selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases, more precisely the individual StringIncludes of that match, are found within a number of FTUnits (words, sentences, and paragraphs). The number of FTUnits is specified by an AdditiveExpr that is converted as though it were an argument to a function with the expected type of xs:integer.

A window selection may cross element boundaries. The size of the window is not affected by the presence or absence of element boundaries. Stop words are included in the computation of the window size whether they are ignored by the query or not.

A window selection examines the matches generated by the preceding portion of the FTSelection, and selects those for which the matched tokens and phrases (more precisely, the individual StringIncludes of that match) are all found within a window whose size is a specified number of FTUnits (words, sentences, or paragraphs); for each such window, the window selection then generates a match containing the merge of those StringIncludes, plus any StringExcludes that fall within the window.

The following expression returns true, because "web", "site", and "usability" are within a window of 5 tokens in the title element:

/books/book/title contains text "web" ftand "site"
ftand "usability" window 5 words

The following expression returns true, because "web" and "site" in the order they are written in the query and either "usability" or "testing" are within a window of at most 10 tokens:

/books/book contains text ("web" ftand "site" ordered)
ftand ("usability" ftor "testing") window 10 words

The following expression returns false, because the instances of "web site" and "usability" in the title element are not within a window of 3. The phrase "Web Site Usability" in the attribute does not apply because the attribute is not part of the string value of the node. A similar query with a window of 5 would return true.

/books/book//title contains text "web site" ftand
"usability" window 3 words

The following expression returns the sample book element, because its number attribute is 1 and it contains a window of 2 words which contains an occurrence of "efficient" but not an occurrence of "and". There is just one such matching window in the sample text and it contains "enable efficient".

/books/book[@number="1" and . contains text "efficient" 
ftand ftnot "and" window 2 words]

The following expression returns the empty sequence, because in the selected book element, there is no occurrence of "efficient" within a window of 3 tokens which would not also contain an occurrence of "and":

/books/book[@number="1" and . contains text "efficient" 
ftand ftnot "and" window 3 words]

In order to allow meaningful results for nested positional filters, e.g., a window selection embedded inside a distance selection, the resulting matches for window selections are formed from the input matches that satisfy the window constraint as follows. All StringIncludes of such a match are coerced into a single StringInclude that spans all token positions from the smallest to the largest position of any input StringIncludes. This is explained in more detail in Section 3.6.3 Distance Selection.

3.6.3 Distance Selection

[Definition: A distance selection consists of a full-text selection followed by one of the (complex) postfix operators derived from FTDistance.]

A distance selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases satisfy the specified distance conditions.

Distances in the search context are measured in units of tokens, sentences, or paragraphs. Roughly speaking, the distance between two matches is the number of intervening units, so a distance of zero tokens (sentences, paragraphs) means no intervening tokens (sentences, paragraphs). More precisely, given two matches, we first determine their order by sorting on starting position and if necessary on ending position. Let M1 be the "earlier" and M2 be the "later". (If there are overlapping tokens involved, the designations "earlier" and "later" may not be intuitively obvious.) Then the distance between the two is M2's starting position minus M1's ending position, minus 1.

When computing distances in the search context, a distance selection may cross element boundaries; they affect the distance computed only to the extent that they affect the tokenization of the search context. Stop words are counted in those computations whether they are ignored or not.

When a distance selection applies a distance condition to more than two matches, the distance condition is required to hold on each successive pair of matches.

An FTDistance expresses a distance condition in terms of an FTUnit and an FTRange. An FTUnit can be words, sentences, or paragraphs, where words refers to a distance measured in tokens.

An FTRange specifies a range of integer values by providing a minimum and/or maximum value for some integer quantity. (Here, where the FTRange appears in an FTDistance, that quantity is a distance. When it appears in an FTTimes, the quantity is a number of occurrences.) Each one of the AdditiveExpr specified in an FTRange is converted as though it were an argument to a function with the expected parameter type of xs:integer.

Let the value of the first (or only) operand be M. If "from" is specified, let the value of the second operand be N.

If "exactly" is specified, then the range is the closed interval [M, M]. If "at least" is specified, then the range is the half-closed interval [M, unbounded). If "at most" is specified, then the range is the half-closed interval (unbounded, M]. If "from-to" is specified, then the range is the closed interval [M, N]. Note: If M is greater than N, the range is empty.

Here are some examples of FTRanges:

1. 'exactly 0' specifies the range [0, 0].

2. 'at least 1' specifies the range [1,unbounded).

3. 'at most 1' specifies the range (unbounded, 1].

4. 'from 5 to 10' specifies the range [5, 10].

The following expression returns false, because "completion" and "errors" are less than 11 tokens apart:

/books/book contains text ("completion" ftand "errors" 
distance at least 11 words)

The following expression returns true:

/books/book contains text "web" ftand "site" ftand
"usability" distance at most 2 words

The search context contains two occurrences of the phrase "the usability of a web site" (once in the <title> and once in the <content>). In this phrase, the tokens "usability" and "web" have a distance of 2 words, and the tokens "web" and "site" have a distance of 0 words, both of which satisfy the constraint distance at most 2 words. (The tokens "usability" and "site" have a distance of 3 words, but this does not cause the distance filter to fail, because these are not successive matches.) Thus, the full-text selection yields two matches, and the whole expression yields true. (The phrase "Improving Web Site Usability" would also satisfy the given full-text selection, but in the sample document it occurs in an attribute value, and so does not contribute to the string value or the tokenization of the book element.)

The following expression returns the empty sequence, because between any token "usability" and the token in any occurrence of the phrase "web site" that is the nearest to the token "usability" there is always more than one intervening token:

/books/book[.//p contains text "web site"
ftand "usability" distance at most 1 words] 

The following expression returns the book title, because for the occurrences of the tokens "web" and "users" in the note element only one intervening token appears:

/books/book[. contains text "web"
ftand "users" distance at most 1 words]/title 

In order to allow meaningful results for nested positional filters, e.g., a distance selection embedded inside another distance selection, the resulting matches for distance selections are formed from the input matches that satisfy the distance constraint as follows. All StringIncludes of such a match are coerced into a single StringInclude that spans all token positions from the smallest to the largest position of any input StringIncludes. Thus, a distance selection that embeds a window or a distance selection takes the result of the embedded selection as a single unit.

The following gives an example of nested distance selections:

/books/book contains text ((("richard" ftand "nixon") distance at most 2 words) 
                   ftand 
                   (("george" ftand "bush") distance at most 2 words) 
                  distance at least 20 words)

This expression allows to find book elements that contain, for instance, "Richard M. Nixon" and "George W. Bush" at least 20 words apart. The matches for the inner distance selections are treated as single units (represented by StringIncludes) by the outer distance selection. Suppose such phrases are present in the search context, then the outer distance selection enforces a constraint on the number of intervening tokens ("at least 20") between the last token of "Richard M. Nixon" and the first token of "George W. Bush".

3.6.4 Scope Selection

[Definition: A scope selection consists of a full-text selection followed by one of the (complex) postfix operators derived from FTScope.]

A scope selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases are contained in the same scope or in different scopes.

Possible scopes are sentences and paragraphs.

By default, there are no restrictions on the scope of the matches.

The following expression returns false, because the tokens "usability" and "Marigold" are not contained within the same sentence:

//book contains text "usability" ftand "Marigold" same sentence

The following expression returns true, because the tokens "usability" and "Marigold" are contained within different sentences:

//book contains text "usability" ftand "Marigold" different sentence

The following expression returns a book element, because it contains "usability" and "testing" in the same paragraph:

//book[. contains text "usability" ftand "testing" same paragraph] 

The following expression returns a book element, because "site" and "errors" appear in the same sentence:

//book[. contains text "site" ftand "errors" same sentence] 

It is possible that both "same sentence" and "different sentence" conditions are simultaneously safisfied for several tokens and/or phrases within the same document fragment. This can be observed if there are occurrences of the tokens and/or phrases both within the same sentence and within difference sentences. For example, consider the following document fragment.

<introduction>
... The usability of a Web site is how well the site supports the user in
achieving specified goals. ... Expert reviews and usability testing are methods of
identifying problems in layout, terminology, and navigation. ...
</introduction>

This sample will satisfy both conditions ("usability" ftand "reviews") different sentence and ("usability" ftand "reviews") same sentence. The tokens "usability" and "reviews" occur both in different sentences (the first and second shown sentences) and in the same sentence (the second shown sentences.)

The above observation also holds for the "same paragraph" and "different paragraph" conditions.

3.6.5 Anchoring Selection

[Definition: An anchoring selection consists of a full-text selection followed by one of the postfix operators "at start", "at end", or "entire content".]

An anchoring selection selects matches which satisfy the operand full-text selection and for which the matched tokens and phrases are the first, last, or all tokens in the tokenized form of the items being searched.

• Using the "at start" operator, tokens or phrases are matched, if they cover the first token position in the tokenized string value of the item being searched.

• Using the "at end" operator, tokens or phrases are matched, if they cover the last token position in the tokenized string value of the item being searched.

• Using the "entire content" operator, tokens or phrases are matched, if they cover all token positions of the tokenized string value of the item being searched.

The following expression returns each title element starting with the phrase "improving the usability of a web site":

/books//title[. contains text "improving the usability
of a web site" at start]

The following expression returns the p element of the sample, because it ends with the phrase "propagating few errors":

/books//p[. contains text "propagat.*" using wildcards ftand "few
errors" distance at most 2 words at end]

Since the distance operator doesn't imply an ordering, the last example would also yield a match if the p element ended with, say, "few errors are propagated".

The following expression returns each note element whose entire content is "this book has been approved by the web site users association":

/books//note[. contains text "this book has been
approved by the web site users association" entire content]

The following example returns true because both the content and the note elements match:

/books//* contains text "Association" at end

5.2.1 FTMildNot Operator

It is optional whether the implementation supports the FTMildNot. If it does not support FTMildNot and encounters one in a full-text query, then it MUST raise an error [err:FTST0001].

5.2.2 FTUnaryNot Operator

The unrestricted form of negation in FTUnaryNot, that can negate every kind of FTSelection, is optional. Implementations may choose to support the negation operation in a restricted form, enforcing one or both of the following restrictions.

• [Definition: Negation Restriction 1. An FTUnaryNot expression may only appear as a direct right operand of an "ftand" (FTAnd) operation.]

• [Definition: Negation Restriction 2. An FTUnaryNot expression may not appear as a descendant of an FTOr that is modified by an FTPosFilter. (An FTOr is modified by an FTPosFilter, if it is derived using the production for FTSelection together with that FTPosFilter.)]

Consider the following example FTSelections.

1. ftnot "web"
2. "web" ftand ( ftnot "information" ftor "retrieval" )
3. "web" ftand ftnot("information" ftand "retrieval")
4. "web" ftand ftnot("information" ftand "retrieval" window 5 words)
5. "web" ftand ("information" ftand ftnot "retrieval" window 5 words)

The first two FTSelections both violate restriction 1, while the third and the fourth are conform with both restrictions. The fifth one violates restriction 2, while obeying restriction 1. Note that in the last example the FTSelection to which the window operation is applied is "information" ftand ftnot "retrieval", which contains an FTUnaryNot expression.

If the implementation does enforce one or both of these restrictions on FTUnaryNot and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0002].

5.2.3 FTUnit and FTBigUnit

Support for the "sentences" alternative of FTUnit and the "sentence" alternative of FTBigUnit is optional. Similarly, support for the "paragraphs" alternative of FTUnit and the "paragraph" alternative of FTBigUnit is optional. If an implementation does not support one or more choices of FTUnit or FTBigUnit and encounters an unsupported FTUnit or FTBigUnit in a full-text query, then it MUST raise an error [err:FTST0003].

5.2.4 FTOrder Operator

The unrestricted form of the FTOrder postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTOrder.

[Definition: Order Operator Restriction. FTOrder may only appear directly succeeding an FTWindow or an FTDistance operator.]

If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0010].

5.2.5 FTScope Operator

It is optional whether the implementation supports the FTScope operator. If it does not support FTScope and encounters one in a full-text query, then it MUST raise an error [err:FTST0004].

5.2.6 FTWindow Operator

The unrestricted form of the FTWindow postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTWindow.

[Definition: Window Operator Restriction. FTWindow can only be applied to an FTOr that is either a single FTWords or a combination of FTWords involving only the operators ftand and ftor.]

If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0011].

5.2.7 FTDistance Operator

The unrestricted form of the FTDistance postfix operator, that can be applied to any kind of FTSelection, is optional. Implementations may choose to enforce the following restriction on the use of FTDistance.

[Definition: Distance Operator Restriction. FTDistance can only be applied to an FTOr that is either a single FTWords or a combination of FTWords involving only the operators ftand and ftor.]

If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0011].

5.2.8 FTTimes Operator

It is optional whether the implementation supports the FTTimes operator. If it does not support FTTimes and encounters one in a full-text query, then it MUST raise an error [err:FTST0005].

5.2.9 FTContent Operator

It is optional whether the implementation supports the FTContent operator. If it does not support FTContent and encounters one in a full-text query, then it MUST raise an error [err:FTST0012].

5.2.10 FTCaseOption

It is optional whether the implementation supports the "lowercase" and "uppercase" choices for the FTCaseOption. If it does not support these choices for the FTCaseOption and encounters an unsupported choice in a full-text query, then it MUST raise an error [err:FTST0015].

5.2.11 FTStopWordOption

It is optional whether the implementation supports the FTStopWordOption. If it does not support FTStopWordOption and encounters one in a full-text query, then it MUST raise an error [err:FTST0006].

It is optional whether the implementation supports the FTStopWordOption in the body of the query. If it supports FTStopWordOption in the prolog, but not in the body of a query, and encounters one in the body of a query it MUST raise an error [err:FTST0006].

It is optional whether the implementation supports the StringLiteral alternative of FTStopWords in the FTStopWordOption. If it does not support the StringLiteral alternative of FTStopWords and encounters such an alternative in a full-text query, then it MUST raise an error [err:FTST0006].

5.2.12 FTLanguageOption

It is optional whether the implementation supports the unrestricted form of FTLanguageOption. Implementations may choose to enforce the following restriction on the use of FTLanguageOption.

[Definition: Single Language Restriction. If a full-text query contains more than one FTLanguageOption in its body and the prolog, then the languages specified must be the same.]

If the implementation does enforce this restriction and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0013].

5.2.13 FTIgnoreOption

The implementation may constrain the set of ignored nodes. If the operand of FTIgnoreOption violates the implementation-defined restriction on that operand, it MUST raise an error [err:FTST0007].

5.2.14 Scoring

The implementation may restrict the allowable expressions used to compute scores. The restrictions are implementation-defined.

If the implementation does enforce such restrictions and encounters a full-text query that does not obey the restriction then it MUST raise an error [err:FTST0014].

5.2.15 Weights

An implementation may constrain the range of valid weights to non-negative values. If an implementation does enforce this restriction and encounters a full-text query that uses a negative weight, it MUST raise an error [err:FTDY0016].