This section describes the full-text selections which contain
the full-text operators in a full-text
contains expression (FTContainsExpr), as well as the
match options which modify the matching semantics of the full-text
selections. In the following, the syntax for each type of full-text
selection is given together with an informal statement of its
meaning.
As shown in the grammar, a full-text selection consists of
search conditions possibly involving logical operators (FTOr), followed by an arbitrary number of
positional filters (FTPosFilter).
The syntax and semantics of the individual full-text selection
operators follow.
This XML document is the source document for examples in this
section.
The first five tokens in this example using the sample
tokenization would be "Improving", "the", "usability", "of", and
"a".
Unless stated otherwise, the results assume a case-insensitive
match.
3.2 Search Tokens and
Phrases
FTWords finds matches that
contain the specified tokens and phrases.
FTWords consists of two parts: a mandatory FTWordsValue part and an optional
FTAnyallOption part.
FTWordsValue specifies the
tokens and phrases that must be contained in the matches. FTAnyallOption specifies how
containment is checked.
In general, the tokens and phrases in FTWordsValue are specified using a
nested XQuery expression. To simplify notation, the enclosing
braces may be omitted if FTWordsValue consists of a single
literal.
The following rules specify how an FTWordsValue matches tokens and
phrases. First, the FTWordsValue is converted to a
sequence of strings as though it were an argument to a function
with the expected type of xs:string*. If the sequence
is empty, the FTWords yields no matches. Otherwise, each of those
strings is tokenized into a sequence of tokens as described in
Section 4.1 Tokenization. Then,
FTAnyallOption is
checked.
If FTAnyallOption is
"any", the sequence of tokens for each string is considered as a
phrase. If the sequence of tokens is empty, then the phrase
contributes nothing to the set of matches for the FTWords.
Otherwise, a match is found in the tokenized form of the text being
searched, whenever that form contains a subsequence of tokens that
corresponds to the sequence of query tokens in an
implementation-defined way and that subsequence of tokens covers
consecutive token positions in the tokenized text. If the value of
the FTWordsValue contains more than one string, the different
strings are considered to be alternatives, i.e. the search context
must contain at least one of the generated phrases. Each resulting
match will contain exactly one such phrase.
If FTAnyallOption is
"all", the sequence of tokens for each string is considered as a
phrase. If any such sequence of tokens is empty, the FTWords yields
no matches. The resulting matches must contain all of the generated
phrases.
If FTAnyallOption is
"phrase", the tokens from all the strings are concatenated in a
single sequence, which is considered as a phrase. If the sequence
of tokens is empty, the FTWords yields no matches. The resulting
matches must contain the generated phrase.
If FTAnyallOption is
"any word", the tokens from all the strings are combined into a
single set. If the set is empty, the FTWords yields no matches. The
search context must contain at least one of the tokens in the set.
Each resulting match will contain exactly one such token.
If FTAnyallOption is
"all words", the tokens from all the strings are combined into a
single set. If the set is empty, the FTWords yields no matches. The
resulting matches must contain all of the tokens in the set.
If the FTWordsValue
evaluates to a single string, the use of "any", "all", and "phrase"
in FTAnyallOption
produces the same results.
If FTAnyallOption is
omitted, "any" is the default.
The following expression returns the sample book
element, because its title element contains the token
"Expert":
//book[./title contains text "Expert"]
The following expression returns the sample book
element, because its title element contains the phrase
"Expert Reviews":
//book[./title contains text "Expert Reviews"]
The following expression returns the sample book
element, because its title element contains the two
tokens "Expert" and "Reviews":
//book[./title contains text {"Expert", "Reviews"} all]
The following expression returns false for our sample document,
because the p element doesn't contain the phrase "Web
Site Usability" although it contains all of the tokens in the
phrase:
//book//p contains text "Web Site Usability"
The following expression returns book numbers of
book elements by "Marigold" with a title about "Web
Site Usability", sorting them in descending score order:
for $book in /books/book[.//author contains text "Marigold"]
let score $score := $book/title/@shortTitle contains text "Web Site Usability"
where $score > 0.8
order by $score descending
return $book/@number
3.4 Match
Options
Full-text match options modify the matching behaviour of the
primary
full-text selection to which they are applied.
[Definition: Match options modify the set
of tokens in the query, or how they are matched against tokens in
the text.]
[Definition: Each of the seven
alternatives of production FTMatchOption corresponds to one
match option group. ] The match options from any given group
are mutually exclusive, i.e., only one of these settings can be in
effect, whereas match options of different groups can be combined
freely.
It is a static error [err:FTST0019] if, within a single FTMatchOptions, there is more
than one match option of any given match option group. For example,
if the FTCaseOption
"lowercase" is specified, then "uppercase" cannot also be specified
as part of the same FTMatchOptions.
Although match options only take effect in the application of
FTWords, the syntax also allows
to specify match options that modify the non-primitive full-text
selection "(" FTSelection ")". Such a higher-level
match option provides a default for the respective match option
group for any embedded FTPrimary, just as match option declarations in the
Prolog provide default match
options for the whole query.
Match options are propagated through the query via the static
context. For each of the seven match option groups, the static
context has a component that contains one option from that group.
The seven settings are initialized by the implementation in
accordance with the table in Appendix C Static Context
Components, and are modified by any FTOptionDecls in the Prolog. The resulting settings are then
propagated unchanged to every FTContainsExpr in the module
(including those in VarDecls and
FunctionDecls, and including any that happen to be
nested within another FTContainsExpr). At any given
FTContainsExpr, the settings from the static context
are copied to the FTContainsExpr's inner settings,
which are then propagated down the syntax tree. At each FTPrimaryWithOptions, the
locally specified match options (if any) overwrite the
corresponding inner setting(s). At each FTWords, the inner settings are used as
the effective match options for tokenizing the query strings and
matching them against the tokens in the text. (These inner settings
could be seen as a parallel set of components in the static
context, but Section 4 Semantics
models them as structures that get passed as parameters to various
semantic functions.)
Thus, when a match option appears in an FTSelection, it applies to the
associated FTPrimary, but not
to any FTContainsExprs that happen to be embedded
within that FTPrimary. Instead, for a nested
FTContainsExpr, the default match options are those
declared in the Prolog or, if not declared in the
Prolog, then supplied by the implementation's initial
values.
An FTMatchOption
applies to the FTPrimary that
immediately precedes it. That FTPrimary is either an FTWords (possibly qualified by an
FTTimes), an FTExtensionSelection, or a
parenthesized FTSelection.
[Definition: The order in
which effective match options for an FTWords are applied is called the
match option application order.] This order is significant
because match options are not always commutative. For example,
synonym(stem(word)) is not always the same as
stem(synonym(word)).
The match option application order is subject to some
constraints:
-
The Language Option must be applied first
-
The Stemming Option must be applied before the Case Option and
the Diacritics Option
Aside from these constraints, the full order of the application
of match options is implementation-defined.
More information on their semantics is given in 4.2.5 Match Options Semantics.
If no match options declarations are present in the prolog and
the implementation does not define any overwriting of the static
context components for the match options, the query:
/books/book/title contains text "usability"
is, assuming "de" is the implementation-defined default
language, equivalent to the query:
/books/book/title contains text "usability"
using language "de"
using no wildcards
using no thesaurus
using no stemming
using case insensitive
using diacritics insensitive
using no stop words
We describe each match option group in more detail in the
following sections.
3.4.1
Language Option
[Definition: A language option modifies
token matching by specifying the language of search tokens and
phrases.]
The StringLiteral following the keyword language
designates one language. It must be castable to
xs:language; otherwise, an error is raised: [err:XPTY0004]XP.
The "language" option influences tokenization, stemming, and
stop words in an implementation-defined way. The
"language" option MAY influence the
behavior of other match options in an implementation-defined way.
The set of standardized language identifiers is defined in
[BCP 47]. The set of valid language
identifiers among the standardized set is implementation-defined. An
implementation MAY choose to use
private extensions introduced by a singleton 'x' for additional
language identifiers, or other singletons for registered extensions
as described in sec. 2.2.6 of [BCP 47]. It is
implementation-defined what
additional language identifiers, if any, are valid. If an invalid
language identifier is specified, then the behavior is implementation-defined. If the
implementation chooses to raise an error in that case, it must
raise [err:FTST0009].
The default language is specified in the static context.
When an XQuery and XPath Full Text processor evaluates text in a
document that is governed by an xml:lang attribute and the portion
of the full-text query doing that evaluation contains an
FTLanguageOption that specifies a different language from the
language specified by the governing xml:lang attribute, the
language-related behavior of that full-text query is implementation-defined.
This is an example where the language option is used to select
the appropriate stop word list:
//book[@number="1"]/content//p contains text "salon de thé"
using stop words default using language "fr"
3.4.2
Wildcard Option
[Definition: A wildcard option modifies
token and phrase matching by specifying whether or not wildcards
are recognized in query strings.]
When the "wildcards" option is used, wildcard syntax may be
included within query strings. A wildcard consists of an indicator
(a period or full stop, "."), optionally followed by a qualifier.
Each wildcard in a query token will match zero or more characters
within a token in the text being searched, as described below. The
number of characters that can be matched depends on the qualifier.
The forms of wildcard syntax specified by this document are:
-
A single period, without any qualifiers: Matches a single
arbitrary character.
-
A period immediately followed by a single question mark, "?":
Matches either no characters or one character.
-
A period immediately followed by a single asterisk, "*": Matches
zero or more characters.
-
A period immediately followed by a single plus sign, "+":
Matches one or more characters.
-
A period immediately followed by a sequence of characters that
matches the regular expression {[0-9]+,[0-9]+}:
Matches a number of characters, where the number is no less than
the number represented by the series of digits before the comma,
and no greater than the number represented by the series of digits
following the comma.
If a period in the query string is immediately followed by a
left curly brace, but the subsequent characters do not conform to
the given regular expression, then an error is raised: [err:FTDY0020].
A question mark, asterisk, plus sign, or left curly brace that
is not immediately preceded by a period is not treated as a
qualifier. For example, using the sample tokenization and
"wildcards", the query string "wil+" does not match the search text
"will" or "willlllll", but only matches the search text "wil". (The
sample tokenization treats the plus sign as punctuation.)
When "wildcards" is used, any character in a query string can be
"escaped" by immediately preceding it with a backslash, "\". That
is, a backslash immediately followed by any character represents
that character literally, preventing any special interpretation
that the "wildcards" option might otherwise attach to it. In
particular:
-
Escaping a period prevents its interpretation as a wildcard.
-
Escaping a question mark, asterisk, plus sign, or left curly
brace ensures that it is not interpreted as a qualifier.
-
An escaped backslash ("\\") represents a literal backslash.
-
If a query string is terminated by an unescaped backslash, an
error is raised: [err:FTDY0020].
Note:
A query string of the form "abc\"xyz" does
not represent the three characters "abc" followed by a
literal double-quote followed by the three characters "xyz".
Instead, this is a malformed StringLiteral, and the processor will
report a syntax error [err:XPST0003]XP.
When the "no wildcards" option is used, no wildcards are
recognized in query strings. Periods, question marks, asterisks,
plus signs, left curly braces, and backslashes are always
recognized as ordinary text characters.
The default is "no wildcards".
The following expression returns true, because the
p element contains "well":
//book[@number="1"]/p contains text "w.ll" using wildcards
The following expression returns true, because the
title element contains "site":
//book[@number="1"]/title contains text ".?site" using wildcards
The following expression returns true, because the
title element contains "improving":
//book[@number="1"]/title contains text "improv.*" using wildcards
The following expression raises error [err:FTDY0020], because the query string
uses incorrect syntax:
//book[@number="1"]/p contains text "wi.{5,7]" using wildcards
The following expression returns true, because the title
contains "site":
//book[@number="1"]/title contains text "\s\i\t\e" using wildcards
The following expression returns true, because the title
contains "Usability":
//book[@number="1"]/title contains text "Usab.+\\" using wildcards
(Note that "\\" represents a literal backslash, which the sample
tokenization treats as punctuation.)
The following expression raises error [err:FTDY0020], because the query string
ends with an unescaped backslash:
//book[@number="1"]/p contains text "will\" using wildcards
The following expression returns false, because the
p element does not contain the phrase "w ll":
//book[@number="1"]/p contains text "w.ll" using no wildcards
(Note that, without wildcards, the sample tokenization will
treat the period in "w.ll" as punctuation, thus producing "w" and
"ll" as separate tokens.)
3.4.3
Thesaurus Option
[Definition: A thesaurus option
modifies token and phrase matching by specifying whether a
thesaurus is used or not.] If thesauri are used, the thesaurus
option specifies information to locate the thesauri either by
default or through a URI reference. It also states the relationship
to be applied and how many levels within the thesaurus to be
traversed.
If the URI specifies a thesaurus that is not found in the
statically known thesauri, an error is raised [err:FTST0018].
Thesauri add related tokens and phrases to the query or change
query tokens. Thus, the user may narrow, broaden, or otherwise
modify the query using synonyms, hypernyms (more generic terms),
etc. The search is performed as though the user has specified all
related query tokens and phrases in a disjunction (FTOr).
Note:
A thesaurus may be standards-based or locally-defined. It may be
a traditional thesaurus, or a taxonomy, soundex, ontology, or topic
map. How the thesaurus is represented is implementation-dependent.
FTThesaurusID specifies the relationship sought between tokens
and phrases written in the query and terms in the thesaurus and the
number of levels to be queried in hierarchical relationships by
including an FTRange "levels". If no levels are specified, the
default is to query all levels in hierarchical relationships.
Relationships include, but are not limited to, the relationships
and their abbreviations presented in [ISO
2788] and their equivalents in other languages. The set of
relationships supported by an implementation is implementation-defined, but
implementations SHOULD support
the relationships defined in [ISO 2788].
The following list of terms have the meanings defined in [ISO 2788]. If a query specifies thesaurus
relationships or levels not supported by the thesaurus, or does not
specify a relationship, the behavior is implementation-defined.
-
equivalence relationships (synonyms): PREFERRED TERM
(USE), NONPREFERRED USED FOR TERM (UF);
-
hierarchical relationships: BROADER TERM (BT), NARROWER
TERM (NT), BROADER TERM GENERIC (BTG), NARROWER TERM GENERIC (NTG),
BROADER TERM PARTITIVE (BTP), NARROWER TERM PARTITIVE (NTP), TOP
Terms (TT); and
-
associative relationships: RELATED TERM (RT).
The "thesaurus" option specifies that string matches include
tokens that can be found in one of the specified thesauri. When
"default" is used in place of a FTThesaurusID, the thesauri
specified in the static context are used, which are either given by
the prolog declaration for the thesaurus option, or, if no such
declaration exists a system-defined default thesaurus with a
system-defined relationship. The default thesaurus may be used in
combination with other explicitly specified thesauri.
The "no thesaurus" option specifies that no thesaurus will be
used.
The default is "no thesaurus".
The following expression returns true, because it finds a
content element containing "task" which the thesaurus
identified as a synonym for "duty":
.//book/content contains text "duty" using
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "UF"
The following expression returns a book element,
because it finds a content element containing "users",
which is a narrower term of "people":
doc("http://bstore1.example.com/full-text.xml")
/books/book[./content contains text "people" using
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "NT" at most 2 levels]
Assuming the thesaurus available at URL
"http://bstore1.example.com/UsabilitySoundex.xml" contains soundex
capabilities, the following query returns a book
element containing "Marigold" which sounds like "Merrygould":
doc("http://bstore1.example.com/full-text.xml")
/books/book[. contains text "Merrygould" using thesaurus at
"http://bstore1.example.com/UsabilitySoundex.xml" relationship
"sounds like"]
3.4.4 Stemming
Option
[Definition: A stemming option modifies
token and phrase matching by specifying whether stemming is applied
or not. ]
The "stemming" option specifies that matches may contain tokens
that have the same stem as the tokens and phrases written in the
query. It is implementation-defined what a stem
of a token is.
The "no stemming" option specifies that the tokens and phrases
are not stemmed.
It is implementation-defined whether the
stemming is based on an algorithm, dictionary, or mixed
approach.
The default is "no stemming".
The following expression returns true, because the
title of the specified book contains
"improving" which has the same stem as "improve":
/books/book[@number="1"]/title contains text "improve" using stemming
3.4.5 Case
Option
| [168] |
FTCaseOption |
::= |
("case" "insensitive")
| ("case" "sensitive")
| "lowercase"
| "uppercase" |
[Definition: A case option modifies the
matching of tokens and phrases by specifying how uppercase and
lowercase characters are considered.]
There are four possible character case options:
-
Using the option "case insensitive", tokens and phrases are
matched, regardless of the case of characters of the query tokens
and phrases.
-
Using the option "case sensitive", tokens and phrases are
matched, if and only if the case of their characters is the same as
written in the query.
-
Using the option "lowercase", tokens and phrases are matched, if
and only if they match the query without regard to character case,
but contain only lowercase characters.
-
Using the option "uppercase", tokens and phrases are matched, if
and only if they match the query without regard to character case,
but contain only uppercase characters.
The default is "case insensitive".
The effect of the case options is also influenced by the query's
default collation (see Section 2.1.1 Static
ContextXQ and Section
4.4 Default Collation DeclarationXQ).
The following table summarizes how these interact.
Case Matrix
| Case option \ Default collation |
UCC (Unicode Codepoint Collation) |
CCS (some generic case-sensitive collation) |
CCI (some generic case-insensitive collation) |
| case insensitive |
compare as if both lower |
case-insensitive variant of CCS if it exists, else error |
CCI |
| case sensitive |
UCC |
CCS |
case-sensitive variant of CCI if it exists, else error |
| lowercase |
compare using UCC after applying fn:lower-case() to the query
string |
compare using CCS after applying fn:lower-case() to the query
string |
CCI |
| uppercase |
compare using UCC after applying fn:upper-case() to the query
string |
compare using CCS after applying fn:upper-case() to the query
string |
CCI |
Note:
In this table, "else error" means "Otherwise, an error is
raised: [err:FOCH0002]FO". The
phrase "if it exists" is used, because the case-sensitive collation
CCS does not always have a case-insensitive variant (and, even if
one exists, it may not be possible to determine it
algorithmically), and because the case-insensitive collation CCI
does not always have a case-sensitive variant (and, even if one
exists, it may not be possible to determine it
algorithmically).
The following expression returns false, because the
title element doesn't contain "usability" in
lower-case characters:
//book[@number="1"]/title contains text "Usability" using lowercase
The following expression returns true, because the character
case is not considered:
//book[@number="1"]/title contains text "usability" using case insensitive
3.4.6
Diacritics Option
[Definition: A diacritics
option modifies token and phrase matching by specifying how
diacritics are considered. ]
There are two possible diacritics options:
-
The option "diacritics" "insensitive" matches tokens and phrases
with and without diacritics. Whether diacritics are written in the
query or not is not considered.
-
The option "diacritics" "sensitive" matches tokens and phrases
only if they contain the diacritics as they are written in the
query.
The default is "diacritics insensitive".
The effect of the diacritics options is also influenced by the
query's default collation (see Section 2.1.1 Static
ContextXQ and Section
4.4 Default Collation DeclarationXQ).
The following table summarizes how these interact.
Diacritics Matrix
| Diacritics option \ Default collation |
UCC (Unicode Codepoint Collation) |
CDS (some generic diacritics-sensitive collation) |
CDI (some generic diacritics-insensitive collation) |
| diacritics insensitive |
UCC comparison, but without considering diacritics |
diacritics-insensitive variant of CDS if it exists, else
error |
CDI |
| diacritics sensitive |
UCC |
CDS |
diacritics-sensitive variant of CDI if it exists, else
error |
Note:
In this table, "else error" means "Otherwise, an error is
raised: [err:FOCH0002]FO". The
phrase "if it exists" is used, because the diacritics-sensitive
collation CDS does not always have a diacritics-insensitive variant
(and, even if one exists, it may not be possible to determine it
algorithmically), and because the diacritics-insensitive collation
CDI does not always have a diacritics-sensitive variant (and, even
if one exists, it may not be possible to determine it
algorithmically).
The following expression returns true, because the token "Véra"
in the editor element is matched, as the acute accent
is not considered in the comparison:
//book[@number="1"]//editor contains text "Vera" using diacritics insensitive
This returns false, because the editor element does
not contain the token "Vera" in this exact form, i.e. without any
diacritics:
//book[@number="1"]/editors contains text "Vera" using diacritics sensitive
3.4.7 Stop
Word Option
[Definition: A stop word option
controls matching of FTWords by specifying whether stop words are
used or not. Stop words are tokens in the query that match any
token in the text being searched. ] Normally a stop word matches
exactly one token, but there may be implementation-defined conditions,
under which a stop word may match a different number of tokens.
FTStopWords specifies
the list of stop words either explicitly as a comma-separated list
of string literals, or by the keyword at followed by a
literal URI. If the URI specifies a list of stop words that is not
found in the statically known stop word lists, an error is raised
[err:FTST0008].
Whether the stop word list is resolved from the statically known
stop word lists or given explicitly, no tokenization is performed
on the stop words: they are used as they occur in the list.
The "stop words" option specifies that if a token is within the
specified collection of stop words, it is removed from the search
and any token may be substituted for it. Stop words retain their
position numbers and are counted in FTDistance and FTWindow searches.
Multiple stop word lists may be combined using "union" or
"except". The keywords "union" and "except" are applied from left
to right. If "union" is specified, every string occurring in the
lists specified by the left-hand side or the right-hand side is a
stop word. If "except" is specified, only strings occurring in the
list specified by the left-hand side but not in the list specified
by the right-hand side are stop words.
The "default stop words" option specifies that an implementation-defined collection
of stop words is used.
The "no stop words" option specifies that no stop words are
used. This is equivalent to specifying an empty list of stop
words.
The default is "no stop words".
Note:
Some implementations may apply stop word lists during indexing
and be unable to comply with query-time requests to not apply those
stop words. An implementation may still support stop-word options
(and therefore not raise [err:FTST0006]) by applying any additional stop
words specified in the query. Pre-application of irrevocable stop
word lists falls under implementation-defined tokenization behavior
in this case, and a query that specifies "no stop words" may still
have some words ignored.
The following expression returns true, because the document
contains the phrase "propagating few errors":
/books/book[@number="1"]//p contains text "propagating of errors"
using stop words ("a", "the", "of")
Note the asymmetry in the stop word semantics: the property of
being a stop word is only relevant to query terms, not to document
terms. Hence, it is irrelevant for the above-mentioned match
whether "few" is a stop word or not, and on the other hand we do
not want the query above to match "propagating" followed by 2 stop
words, or even a sequence of 3 stop words in the document.
The following expression returns false. In this case specifying
"few" as a stop word has no effect, since "few" does not appear in
the query. Although the words "propagating" and "errors" appear in
the text being searched, the phrase "propagating errors" cannot be
matched, since that phrase does not occur.
/books/book[@number="1"]//p contains text "propagating errors"
using stop words ("few")
The following expression returns false, because "of" is not in
the p element between "propagating" and "errors":
/books/book[@number="1"]//p contains text "propagating of errors"
using no stop words
The following expression uses the stop words list specified at
the URL. Assuming that the specified stop word list contains the
word "then", this query is reduced to a query on the phrase
"planning X conducting", allowing any token as a substitute for X.
It returns a book element, because its
content element contains "planning then conducting".
It would also return the book if the phrases "planning
and conducting" and "planning before conducting" had been in its
content:
doc("http://bstore1.example.com/full-text.xml")
/books/book[.//content contains text "planning then
conducting" using stop words at
"http://bstore1.example.com/StopWordList.xml"]
The following expression returns books containing
"planning then conducting", but not does not return
books containing "planning and conducting", since it
is exempting "then" from being a stop word:
doc("http://bstore1.example.com/full-text.xml")
/books/book[.//content contains text "planning then conducting"
using stop words at "http://bstore1.example.com/StopWordList.xml"
except ("the", "then")]
3.4.8
Extension Option
[Definition: An extension option is a
match option that acts in an implementation-defined way.
]
An extension option consists of an identifying QName and a
StringLiteral. Typically, a particular option will be recognized by
some implementations and not by others. The syntax is designed so
that option declarations can be successfully parsed by all
implementations.
The QName of an extension option must resolve to a namespace URI
and local name, using the statically known namespaces.
Note:
There is no default namespace for options.
Each implementation recognizes an implementation-defined set of
namespace URIs used to denote extension options.
If the namespace part of the QName is not a namespace recognized
by the implementation as one used to denote extension option, then
the extension option is ignored.
Otherwise, the effect of the extension option, including its
error behavior, is implementation-defined. For
example, if the local part of the QName is not recognized, or if
the StringLiteral does not conform to the rules defined by the
implementation for the particular extension option, the
implementation may choose whether to report an error, ignore the
extension option, or take some other action.
Implementations may impose rules on where particular extension
options may appear relative to other match options, and the
interpretation of an option declaration may depend on its
position.
An extension option must not be used to change the syntax
accepted by the processor, or to suppress the detection of static
errors. However, it may be used without restriction to modify the
set of tokens in the query or how they are matched against tokens
in the text being searched. An extension option has the same scope
as other match options.
The following examples illustrate several possible uses for
extension options:
This extension option is set as part of the static context of
all full-text expressions in the module and might be used to ensure
that queries are insensitive to Arabic short-vowels.
declare namespace exq = "http://example.org/XQueryImplementation";
declare ft-option using option exq:diacritics "short-vowel insensitive";
This extension option applies only to the matching in the
full-text selection in which it is found and might be used to
specify how compound words should be matched.
declare namespace exq = "http://example.org/XQueryImplementation";
//para[. contains text
("Kinder" ftand "Platz" distance exactly 1 words)
using stemming
using option exq:compounds "distance=1" ]
3.5
Logical Full-Text Operators
Full-text selections can be combined with the logical
connectives ftor (full-text or), ftand
(full-text and), not in (mild not), and
ftnot (unary full-text not).
3.5.1 Or-Selection
[Definition: An or-selection combines two
full-text selections using the ftor operator.]
An or-selection finds all matches that satisfy at least one of
the operand full-text selections.
The following expression returns the book element
written by "Millicent":
//book[.//author contains text "Millicent" ftor "Voltaire"]
3.5.2 And-Selection
[Definition: An and-selection combines
two full-text selections using the ftand
operator.]
An and-selection finds matches that satisfy all of the operand
full-text selections simultaneously. A match of an and-selection is
formed by combining matches for each of the operand full-text
selections as described in 4.2.6.2
FTAnd.
For example, "usability" ftand "testing" will find
two matches in //book[@number="1"]/title: each of the
two matches for the FTWords selection "usability" (the
two occurrences of "usability" in the string value of the title
element) is combined with the single match for the FTWords
"testing" (only one occurrence of "testing" in the
title). Since the above and-selection has at least one match, the
following expression will return "true".
//book[@number="1"]/title contains text ("usability" ftand "testing")
The following expression returns false, because "Millicent" and
"Montana" are not contained by the same author element
in any book element:
//book/author contains text "Millicent" ftand "Montana"
No author element in any book element
contains both "Millicent" and "Montana". Therefore, for any such
author element, there are either one match for the
FTWords "Millicent" and zero matches for the FTWords
"Montana", or vice versa, or no matches for both of
them. In any of these cases, the and-selection will have zero
matches.
3.5.3 Mild-Not
Selection
[Definition: A mild-not
selection combines two full-text selections using the not
in operator.]
The not in operator is a milder form of the
operator combination ftand ftnot. The selection
A not in B matches a token sequence that matches
A, but not when it is a part of a match of
B. In contrast, A ftand ftnot B only
finds matches when the token sequence contains A and
does not contain B.
As an example, consider a search for "Mexico" not in "New
Mexico". This may return, among others, a document which is
all about "Mexico" but mentions at the end that "New Mexico was
named after Mexico". The occurrence of "Mexico" in "New Mexico" is
not considered, but other occurrences of "Mexico" are matched. Note
that this document would not be matched by the full-text selection
"Mexico" ftand ftnot "New Mexico".
A match to a mild-not selection must contain at least one token
that satisfies the first condition and does not satisfy the second
condition. If it contains a token that satisfies both the first and
the second condition, the token is not considered as a match.
The following expression returns true, because "usability"
appears in the title and the p elements
and the token within the phrase "Usability Testing" in the
title element is not considered:
/books/book contains text "usability" not in "usability testing"
If either operand of a mild-not selection yields an
AllMatches that contains a Match that contains a
StringExclude, then a dynamic error [err:FTDY0017] is raised.
Note:
This situation can arise if the operand contains a not-selection
or a cardinality constraint (FTTimes) involving
exactly, at most, or from ...
to.
3.5.4 Not-Selection
[Definition: A not-selection is a
full-text selection starting with the prefix operator
ftnot.]
A not-selection selects matches that do not satisfy the operand
full-text selection. Details about how such matches are constructed
are given in 4.2.6.3
FTUnaryNot.
The following expression returns the empty sequence, because all
book elements contain "usability":
//book[. contains text ftnot "usability"]
The following expression returns true, because book
elements contain "improving" and "usability" but not "improving
usability":
//book contains text "improving" ftand
"usability" ftand ftnot "improving usability"
The following expression returns book elements
containing "web site usability" but not "usability testing":
//book[title/@shortTitle contains text "web site usability" ftand
ftnot "usability testing"]
3.6 Positional
Filters
[Definition: Positional filters are
postfix operators that serve to filter matches based on various
constraints on their positional information.]
Recall that the grammar rule for FTSelection allows an arbitrary
number of positional filters to follow an FTOr. Multiple adjacent positional filters
are applied from left to right, i.e., the first filter is applied
to the result of the FTOr, the
second is applied to the result of that first application, and so
on.
An FTOr consists of one or more
FTAnds (separated by
ftor), each of which could be an FTPosFilter applied to an embedded
FTOr, enclosed in parentheses.
3.6.1 Ordered Selection
[Definition: An ordered selection
consists of a full-text selection followed by the postfix operator
"ordered".] An ordered selection constrains the order of tokens and
phrases to be the same as the order in which they are written in
the operand selection.
The default is unordered. Unordered is in effect when ordered is
not specified in the query. Unordered cannot be written explicitly
in the query.
An ordered selection selects matches which satisfy the operand
full-text selection and which also satisfy the following
constraint: the order that the matching tokens or phrases have in
the text being searched is the same order that the corresponding
query tokens or phrases have in the operand selection. In both
cases, the ordering is determined from the minimum start positions
of the contituent tokens.
The following expression returns true, because titles of
book elements contain "web site" and "usability" in
the order in which they are written in the query, i.e., "web site"
must precede "usability":
//book/title contains text ("web site" ftand "usability") ordered
The following expression returns false, because although
"Montana" and "Millicent" both appear in the book
element, they do not appear in the order they are written in the
query:
//book[@number="1"] contains text ("Montana" ftand "Millicent") ordered
3.6.2 Window
Selection
[Definition: A window selection
consists of a full-text selection followed by one of the (complex)
postfix operators derived from FTWindow.] A window selection selects
matches which satisfy the operand full-text selection and for which
the matched tokens and phrases, more precisely the individual
StringIncludes of that match, are found within a number of FTUnits (words, sentences, and
paragraphs). The number of FTUnits is specified by an AdditiveExpr
that is converted as though it were an argument to a function with
the expected type of xs:integer.
A window selection may cross element boundaries. The size of the
window is not affected by the presence or absence of element
boundaries. Stop words are included in the computation of the
window size whether they are ignored by the query or not.
A window selection examines the matches generated by the
preceding portion of the FTSelection, and selects those for
which the matched tokens and phrases (more precisely, the
individual StringIncludes of that match) are all found within a
window whose size is a specified number of FTUnits (words,
sentences, or paragraphs); for each such window, the window
selection then generates a match containing the merge of those
StringIncludes, plus any StringExcludes that fall within the
window.
The following expression returns true, because "web", "site",
and "usability" are within a window of 5 tokens in the
title element:
/books/book/title contains text "web" ftand "site"
ftand "usability" window 5 words
The following expression returns true, because "web" and "site"
in the order they are written in the query and either "usability"
or "testing" are within a window of at most 10 tokens:
/books/book contains text ("web" ftand "site" ordered)
ftand ("usability" ftor "testing") window 10 words
The following expression returns false, because the instances of
"web site" and "usability" in the title element are
not within a window of 3. The phrase "Web Site Usability" in the
attribute does not apply because the attribute is not part of the
string value of the node. A similar query with a window of 5 would
return true.
/books/book//title contains text "web site" ftand
"usability" window 3 words
The following expression returns the sample book
element, because its number attribute is 1 and it
contains a window of 2 words which contains an occurrence of
"efficient" but not an occurrence of "and". There is just one such
matching window in the sample text and it contains "enable
efficient".
/books/book[@number="1" and . contains text "efficient"
ftand ftnot "and" window 2 words]
The following expression returns the empty sequence, because in
the selected book element, there is no occurrence of
"efficient" within a window of 3 tokens which would not also
contain an occurrence of "and":
/books/book[@number="1" and . contains text "efficient"
ftand ftnot "and" window 3 words]
In order to allow meaningful results for nested positional
filters, e.g., a window selection embedded inside a distance
selection, the resulting matches for window selections are formed
from the input matches that satisfy the window constraint as
follows. All StringIncludes of such a match are coerced into a
single StringInclude that spans all token positions from the
smallest to the largest position of any input StringIncludes. This
is explained in more detail in Section 3.6.3 Distance Selection.
3.6.3 Distance
Selection
[Definition: A distance
selection consists of a full-text selection followed by one of
the (complex) postfix operators derived from FTDistance.]
A distance selection selects matches which satisfy the operand
full-text selection and for which the matched tokens and phrases
satisfy the specified distance conditions.
Distances in the search context are measured in units of tokens,
sentences, or paragraphs. Roughly speaking, the distance between
two matches is the number of intervening units, so a distance of
zero tokens (sentences, paragraphs) means no intervening tokens
(sentences, paragraphs). More precisely, given two matches, we
first determine their order by sorting on starting position and if
necessary on ending position. Let M1 be the "earlier" and M2 be the
"later". (If there are overlapping tokens involved, the
designations "earlier" and "later" may not be intuitively obvious.)
Then the distance between the two is M2's starting position minus
M1's ending position, minus 1.
When computing distances in the search context, a distance
selection may cross element boundaries; they affect the distance
computed only to the extent that they affect the tokenization of
the search context. Stop words are counted in those computations
whether they are ignored or not.
When a distance selection applies a distance condition to more
than two matches, the distance condition is required to hold on
each successive pair of matches.
An FTDistance expresses a
distance condition in terms of an FTUnit and an FTRange. An FTUnit can be words,
sentences, or paragraphs, where
words refers to a distance measured in tokens.
An FTRange specifies a range
of integer values by providing a minimum and/or maximum value for
some integer quantity. (Here, where the FTRange appears in an
FTDistance, that quantity is a distance. When it appears in an
FTTimes, the quantity is a number of occurrences.) Each one of the
AdditiveExpr specified in an FTRange is converted as though it were
an argument to a function with the expected parameter type of
xs:integer.
Let the value of the first (or only) operand be M. If "from" is
specified, let the value of the second operand be N.
If "exactly" is specified, then the range is the closed interval
[M, M]. If "at least" is specified, then the range is the
half-closed interval [M, unbounded). If "at most" is specified,
then the range is the half-closed interval (unbounded, M]. If
"from-to" is specified, then the range is the closed interval [M,
N]. Note: If M is greater than N, the range is empty.
Here are some examples of FTRanges:
-
'exactly 0' specifies the range [0, 0].
-
'at least 1' specifies the range [1,unbounded).
-
'at most 1' specifies the range (unbounded, 1].
-
'from 5 to 10' specifies the range [5, 10].
The following expression returns false, because "completion" and
"errors" are less than 11 tokens apart:
/books/book contains text ("completion" ftand "errors"
distance at least 11 words)
The following expression returns true:
/books/book contains text "web" ftand "site" ftand
"usability" distance at most 2 words
The search context contains two occurrences of the phrase "the
usability of a web site" (once in the <title> and once in the
<content>). In this phrase, the tokens "usability" and "web"
have a distance of 2 words, and the tokens "web" and "site" have a
distance of 0 words, both of which satisfy the constraint
distance at most 2 words. (The tokens "usability" and
"site" have a distance of 3 words, but this does not cause the
distance filter to fail, because these are not successive matches.)
Thus, the full-text selection yields two matches, and the whole
expression yields true. (The phrase "Improving Web Site Usability"
would also satisfy the given full-text selection, but in the sample
document it occurs in an attribute value, and so does not
contribute to the string value or the tokenization of the book
element.)
The following expression returns the empty sequence, because
between any token "usability" and the token in any occurrence of
the phrase "web site" that is the nearest to the token "usability"
there is always more than one intervening token:
/books/book[.//p contains text "web site"
ftand "usability" distance at most 1 words]
The following expression returns the book title,
because for the occurrences of the tokens "web" and "users" in the
note element only one intervening token appears:
/books/book[. contains text "web"
ftand "users" distance at most 1 words]/title
In order to allow meaningful results for nested positional
filters, e.g., a distance selection embedded inside another
distance selection, the resulting matches for distance selections
are formed from the input matches that satisfy the distance
constraint as follows. All StringIncludes of such a match are
coerced into a single StringInclude that spans all token positions
from the smallest to the largest position of any input
StringIncludes. Thus, a distance selection that embeds a window or
a distance selection takes the result of the embedded selection as
a single unit.
The following gives an example of nested distance
selections:
/books/book contains text ((("richard" ftand "nixon") distance at most 2 words)
ftand
(("george" ftand "bush") distance at most 2 words)
distance at least 20 words)
This expression allows to find book elements that
contain, for instance, "Richard M. Nixon" and "George W. Bush" at
least 20 words apart. The matches for the inner distance selections
are treated as single units (represented by StringIncludes) by the
outer distance selection. Suppose such phrases are present in the
search context, then the outer distance selection enforces a
constraint on the number of intervening tokens ("at least 20")
between the last token of "Richard M. Nixon" and the first token of
"George W. Bush".
3.6.4 Scope Selection
[Definition: A scope selection consists
of a full-text selection followed by one of the (complex) postfix
operators derived from FTScope.]
A scope selection selects matches which satisfy the operand
full-text selection and for which the matched tokens and phrases
are contained in the same scope or in different scopes.
Possible scopes are sentences and paragraphs.
By default, there are no restrictions on the scope of the
matches.
The following expression returns false, because the tokens
"usability" and "Marigold" are not contained within the same
sentence:
//book contains text "usability" ftand "Marigold" same sentence
The following expression returns true, because the tokens
"usability" and "Marigold" are contained within different
sentences:
//book contains text "usability" ftand "Marigold" different sentence
The following expression returns a book element,
because it contains "usability" and "testing" in the same
paragraph:
//book[. contains text "usability" ftand "testing" same paragraph]
The following expression returns a book element,
because "site" and "errors" appear in the same sentence:
//book[. contains text "site" ftand "errors" same sentence]
It is possible that both "same sentence" and "different
sentence" conditions are simultaneously safisfied for several
tokens and/or phrases within the same document fragment. This can
be observed if there are occurrences of the tokens and/or phrases
both within the same sentence and within difference sentences. For
example, consider the following document fragment.
<introduction>
... The usability of a Web site is how well the site supports the user in
achieving specified goals. ... Expert reviews and usability testing are methods of
identifying problems in layout, terminology, and navigation. ...
</introduction>
This sample will satisfy both conditions ("usability"
ftand "reviews") different sentence and ("usability"
ftand "reviews") same sentence. The tokens "usability" and
"reviews" occur both in different sentences (the first and second
shown sentences) and in the same sentence (the second shown
sentences.)
The above observation also holds for the "same paragraph" and
"different paragraph" conditions.
3.6.5 Anchoring
Selection
| [165] |
FTContent |
::= |
("at" "start") | ("at" "end") | ("entire"
"content") |
[Definition: An anchoring
selection consists of a full-text selection followed by one of
the postfix operators "at start", "at end", or "entire
content".]
An anchoring selection selects matches which satisfy the operand
full-text selection and for which the matched tokens and phrases
are the first, last, or all tokens in the tokenized form of the
items being searched.
-
Using the "at start" operator, tokens or phrases are matched, if
they cover the first token position in the tokenized string value
of the item being searched.
-
Using the "at end" operator, tokens or phrases are matched, if
they cover the last token position in the tokenized string value of
the item being searched.
-
Using the "entire content" operator, tokens or phrases are
matched, if they cover all token positions of the tokenized string
value of the item being searched.
The following expression returns each title element
starting with the phrase "improving the usability of a web
site":
/books//title[. contains text "improving the usability
of a web site" at start]
The following expression returns the p element of
the sample, because it ends with the phrase "propagating few
errors":
/books//p[. contains text "propagat.*" using wildcards ftand "few
errors" distance at most 2 words at end]
Since the distance operator doesn't imply an ordering, the last
example would also yield a match if the p element
ended with, say, "few errors are propagated".
The following expression returns each note element
whose entire content is "this book has been approved by the web
site users association":
/books//note[. contains text "this book has been
approved by the web site users association" entire content]
The following example returns true because both the
content and the note elements match:
/books//* contains text "Association" at end
3.8 Extension Selections
[Definition: An extension
selection is a full-text selection whose semantics are
implementation-defined.]
Typically, a particular extension will be recognized by some
implementations and not by others. The syntax is designed so that
extension selections can be successfully parsed by all
implementations, and so that fallback behavior can be defined for
implementations that do not recognize a particular extension.
An extension selection consists of one or more pragmas followed
by a full-text selection enclosed in curly braces. See Section 3.14
Extension ExpressionsXQ for
information on pragmas in general. A pragma is denoted by the
delimiters (# and #), and consists of an
identifying QName followed by implementation-defined
content. The content of a pragma may consist of any string of
characters that does not contain the ending delimiter
#). The QName of a pragma must resolve to a namespace
URI and local name, using the statically known namespaces.
Note:
Since there is no default namespace for pragmas, a pragma QName
must include a namespace prefix.
Each implementation recognizes an implementation-defined set of
namespace URIs used to denote pragmas.
If the namespace part of a pragma QName is not recognized by the
implementation as a pragma namespace, then the pragma is ignored.
If all the pragmas in an FTExtensionSelection are
ignored, then full-text extension selection is just the full-text
selection enclosed in curly braces; if this full-text selection is
absent, then a static error is raised [err:XQST0079]XQ.
If an implementation recognizes the namespace of one or more
pragmas in an FTExtensionSelection, then
the value of the FTExtensionSelection,
including its error behavior, is implementation-defined. For
example, an implementation that recognizes the namespace of a
pragma QName, but does not recognize the local part of the QName,
might choose either to raise an error or to ignore the pragma.
It is a static error [err:XQST0013]XQ if an
implementation recognizes a pragma but determines that its content
is invalid.
If an implementation recognizes a pragma, it must report any
static errors in the following full-text selection even if it will
not apply that selection.
The following examples illustrate three ways in which extension
selections might be used.
A pragma can be used to furnish a hint for how to evaluate the
following full-text selection, without actually changing the
result. For example:
declare namespace exq = "http://example.org/XQueryImplementation";
/books/book/author[name contains text (# exq:use-index #) {'Berners-Lee'}]
An implementation that recognizes the exq:use-index
pragma might use an index to evaluate the full-text selection that
follows. An implementation that does not recognize this pragma
would evaluate the full-text selection in its normal way.
A pragma might be used to modify the semantics of the following
full-text selection in ways that would not (in the absence of the
pragma) be conformant with this specification. For example, a
pragma might be used to change distance counting so that adjacent
words are at a distance of 1 (otherwise they would be at a distance
of 0):
declare namespace exq = "http://example.org/XQueryImplementation";
/books/book[.//p contains text (# exq:distance #) { "web site"
ftand "usability" distance at most 1 words }]
Such changes to the language semantics must be scoped to the
expression contained within the curly braces following the
pragma.
A pragma might contain syntactic constructs that are evaluated
in place of the following full-text selection. In this case, the
following selection itself (if it is present) provides a fallback
for use by implementations that do not recognize the pragma. For
example:
declare namespace exq = "http://example.org/XQueryImplementation";
//city[. contains text (# exq:classifier with class 'Animals' #)
{"animal" using thesaurus at "http://example.org/thesaurus.xml"
relationship "RT"}]
Here an implementation that recognizes the pragma will return
the result of evaluating the proprietary syntax with class
'animals', while an implementation that does not recognize
the pragma will instead return the result of the thesaurus option.
If no fallback expression is required, or if none is feasible, then
the expression between the curly braces may be omitted, in which
case implementations that do not recognize the pragma will raise a
static error.
