Please refer to the errata for this document, which may include some normative corrections.
See also translations.
Copyright © 2009 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
The Protocol for Web Description Resources (POWDER) facilitates the publication of descriptions of multiple resources such as all those available from a Web site. This document describes how sets of IRIs can be defined such that descriptions or other data can be applied to the resources obtained by dereferencing IRIs that are elements of the set. IRI sets are defined as XML elements with relatively loose operational semantics. This is underpinned by the formal semantics of POWDER which include a semantic extension, defined separately. A GRDDL transform is associated with the POWDER namespace that maps the operational to the formal semantics.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This document is a W3C Recommendation that was developed by the POWDER Working Group.
Please see the Working Group's implementation report and Disposition of Last Call Comments. The disposition of comments received during previous calls are also available. Changes since the previous version of this document are minor in nature and are fully documented in the Change log.
Publication of this Recommendation is synchronized with several other documents:
The W3C Membership and other interested parties are invited to review the document and send comments to public-powderwg@w3.org (with public archive).
This document has been reviewed by W3C Members, by software developers, and by other W3C groups and interested parties, and is endorsed by the Director as a W3C Recommendation. It is a stable document and may be used as reference material or cited from another document. W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
includeregex
The Protocol for Web Description Resources (POWDER) facilitates the publication of descriptions of multiple resources such as all those available from a Web site. These descriptions are attributable to a named individual, organization or entity that may or may not be the creator of the described resources. This contrasts with more usual metadata that typically apply to a single resource, such as a specific document's title, which is usually provided by its author.
Description Resources (DRs) are described separately [DR]. This document sets out how groups (i.e. sets) of resources may be defined, either for use in DRs or in other contexts. Set theory has been used throughout as it provides a well-defined framework that leads to unambiguous definitions. However, it is used solely to provide a formal version of what is written in the natural language text.
POWDER uses a limited set of XML elements to define sets of resources and these have relatively loose semantics. However, a GRDDL [GRDDL] transform is associated with the POWDER root namespace through which formal semantics are accessible as RDF/OWL. This is known as Semantic POWDER or POWDER-S. The details of the GRDDL transform and the formal semantics are defined separately [FORMAL] and outlined in Section 1.4 below. The use cases, a primer, test suite and schema namespace documents complete the document set.
The POWDER schema namespace is http://www.w3.org/2007/05/powder#
for which we use the prefix wdr
. The POWDER-S namespace is http://www.w3.org/2007/05/powder-s#
for which we use the prefix wdrs
and All namespaces and prefixes used in this document are shown in the table below.
Prefix | Namespace |
---|---|
wdr |
http://www.w3.org/2007/05/powder# |
wdrs |
http://www.w3.org/2007/05/powder-s# |
rdf |
http://www.w3.org/1999/02/22-rdf-syntax-ns# |
rdfs |
http://www.w3.org/2000/01/rdf-schema#" |
owl |
http://www.w3.org/2002/07/owl# |
xsd |
http://www.w3.org/2001/XMLSchema |
ex |
An arbitrary prefix used to denote an 'example vocabulary' |
In this document, the words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as described in RFC 2119 [RFC2119].
POWDER makes substantial use of XML, the processing rules for which MUST be followed faithfully. The processing rules for attribute-value normalization are particularly relevant when considering the white space separated lists of values that occur in POWDER. A space-separated list is a string of which the items are separated by one or more space characters (in any order). The string may also be prefixed or suffixed with zero or more of those characters. To obtain the values from a space-separated list user agents MUST replace any sequence of space characters with a single #x20 character, dropping any leading or trailing #x20 character, and then chopping the resulting string at each occurrence of a #x20 character, dropping that character in the process.
The (unqualified) terms POWDER, POWDER Document and Description Resource (DR) refer to operational representations and semantics. The term POWDER-S refers to documents and data that express the formal semantics of POWDER. Unqualified XML element names are in the POWDER (wdr
) namespace
In designing a system to define sets of resources we have drawn on earlier work [Rabin] carried out in the Web Content Label Incubator Activity [WCL-XG], and taken into account the following considerations.
Operationally, POWDER does not define resource sets, rather, it facilitates the definition of sets of IRIs (International Resource Identifiers) [IRIS], which can be used to denote resources in terms of their identifiers. We use the notion of IRIs instead of URIs[URIS] since IRIs are a superset of URIs. Therefore, an IRI set definition may denote a set of IRIs as well as a set of URIs.
Defining a resource set by specifying the characteristics that the identifiers of resources in the set share is clearly an indirect approach, albeit a very useful one in the real world. In a logical sense, the definition must be interpreted to arrive at the full set.
More formally, an IRI Set definition D denotes a set of IRIs IS = DI, where DI is the interpretation of D, i.e., the set of IRIs sharing the characteristics denoted by D.
We take this further and allow an IRI set definition to be built up in stages.
An IRI set IS is denoted by an IRI set definition DIS in terms of one or more characteristics that the elements of the set have in common. Each characteristic is expressed by an IRI constraint C, and IRI constraints C1, C2, … Cn give rise to IRI set definitions D, 1, D2, … Dn, so that the complete IRI set definition DIS comprises D1, D2, … Dn.
The IRI set IS is the intersection of the IRI sets denoted by the IRI set definitions in DIS.
Formally:
IS = DISI = D1I ∩ D2I ∩ … ∩ DnI = (D1 ∧ D2 ∧ … ∧ Dn)I.
For example, suppose that an IRI set IS is denoted by the following definitions:
example.org
”/foo
“Then, DIS will be defined as follows: “the top level components of the host component of the IRI exactly match example.org
” AND “the path component of the IRI begins with /foo
.”
Whether the IRI of a specific resource R, known as the candidate resource, is a member of IRI Set IS or not is determined by comparing its characteristics with those denoted by the set definitions used in DIS. It must be an element of the intersection of the sets defined by the interpretation of D1, D2, …, Dn to be an element of IS.
If an IRI set definition contains no constraints, then its interpretation is by definition the empty set ∅. Formally:
Let IS be an IRI Set, and let DIS be the set of IRI Set definitions denoting the IRIs in IS: if DIS = ∅, then IS = ∅.
The POWDER XML schema [WDR] defines the set of XML elements and attributes to be used for enforcing the operational semantics of an IRI set definition.
More precisely, we define an XML element iriset
to take the place of the IRI set, and its child elements denote the set of IRI constraints C1, C2, …, Cn. The example reported in the previous section can therefore be written as follows:
<iriset> <includehosts>example.org</includehosts> <includepathstartswith>/foo</includepathstartswith> </iriset>
The operational semantics described above are underpinned by formal semantics. A GRDDL [GRDDL] transform
is associated with the POWDER namespace that allows the XML data to be rendered and processed as RDF/OWL with one important
proviso — that a semantic extension is understood. Defined fully in the Formal Semantics document
[FORMAL], this allows a candidate resource's IRI to be matched against regular expressions that are values
of an OWL data type property wdrs:matchesregex
(or wdrs:notmatchesregex
in
the case of patterns that are to be excluded). An OWL class takes the place of the IRI set
and resources whose IRIs match all the property restrictions defined using wdrs:matchesregex
and wdrs:notmatchesregex
are
instances of that class. The regular expression syntax used is defined by XML schema as modified by XQuery 1.0 and
XPath 2.0 Functions and Operators [XQXP].
As shown in Example 1-1 above, the POWDER XML elements generally take strings as values. These are converted into regular expressions as a first step in the GRDDL transform which renders POWDER documents in an intermediate format known as POWDER-BASE. It is POWDER-BASE that is then transformed into POWDER-S. For clarity, this two-stage process is not referred to in the main section of this document on defining a resource set which only presents POWDER and POWDER-S examples. POWDER-BASE is, however, an important part of the extension mechanism of POWDER Resource Grouping. The Formal Semantics document gives full details of the transformation of all elements of POWDER documents to POWDER-BASE and POWDER-S.
The result of the GRDDL transformation on Example 1-1 above is shown below.
<owl:Class rdf:nodeID="iriset_1"> <owl:equivalentClass> <owl:Class> <owl:intersectionOf rdf:parseType="Collection"> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)?(example\.org)(:([0-9]+))?\/</owl:hasValue> </owl:Restriction> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?(\/foo)</owl:hasValue> </owl:Restriction> </owl:intersectionOf> </owl:Class> </owl:equivalentClass> </owl:Class>
A Resource Set is defined in terms of the IRIs of resources that are its members. Determining whether a candidate resource is, or is not, a member of the set, can therefore be done by comparing its IRI with the data in the set definition. Importantly, defining the Resource Set in terms of IRIs allows us to verify whether the candidate resource is in the set without having to fetch and parse it, or perform a DNS lookup, thus maximizing processing efficiency in many environments.
We define a range of methods to support set definition by IRI, and provide support for methods defined in other Recommendations.
The syntax of an IRI, as defined in RFC 3987 [IRIS], provides a generic framework for identification schemes that goes beyond what is demanded by the POWDER use cases [USECASES]. We therefore limit our work to IRIs with the syntax: scheme://iuser@ihost:port/ipath?iquery#ifragment
, as shown below:
http://jdoe@www.example.com:1234/example1/example2?query=help#fragment \ / \ / \ / \ /\ / \ / \ / -- -- ------------- -- ---------------- -------- ------ | | | | | | | scheme iuser ihost port ipath iquery ifragment info
The following Regular Expression, elaborated from that offered in RFC 3986 [Rabin], provides a means of splitting both URIs and IRIs of this type into their component parts.
^(([^:/?#]+):)?(//((([^/?#]*)@)?([^/?#:]*)(:([^/?#]*))?))?([^?#]*)(\?([^#]*))?(#(.*))?
If the IRI of the candidate resource is valid, this yields the components as shown in Table 2 (strings that are not valid IRIs will inevitably lead to unpredictable results).
Component | RE variable |
---|---|
scheme | $2 |
iuserinfo | $6 |
ihost | $7 |
port | $9 |
ipath | $10 |
iquery | $12 |
ifragment | $14 |
For the scheme, ihost, port, ipath, and iquery IRI components we define corresponding IRI constraints, the value of most of which is a white space-separated list of strings, any one of which must match the relevant portion of the IRI of the candidate resource. The exceptions are the iquery and ihost components, which we discuss further in sections 2.1.2 and 2.1.3.
The iuserinfo and ifragment components are not used in POWDER IRI set definitions directly as it is felt that these may add a layer of unnecessary complexity with few practical applications. That said, it is important not to discard these components when processing the candidate resource's IRI. Furthermore, IRI sets may be defined using additional vocabularies as set out in Section 3. That extension method, or the use of the includeregex
and excluderegex
properties (see Section 2.3 below), means that user info and fragments can be used in IRI set definitions if required.
Formally, an IRI set definition D is expressed by one or more IRI constraints of the form C = IRI_component_matches(?x, {string1 | string2 | … | stringn}), where ?x is a variable denoting the IRI component under consideration, and {string1 | string2 | … | stringn} denotes a set consisting either of string string1 OR string2 OR … OR stringn.
Any number of IRI constraints C1, C2, …, Cn can be declared, and, as stated in Section 1.2, the overall IRI set is the intersection of the sets that can be interpreted from IRI set definitions corresponding to Cn. With some exceptions, each particular IRI constraint can only appear 0 or 1 times.
Strings are matched according to one of four rules:
startsWith
, meaning that the IRI component starts with any of the strings listed in the value of the relevant IRI constraint;endsWith
, meaning that the IRI component ends with any of the strings listed in the value of the relevant IRI constraint;exact
, meaning that there is an exact match between the candidate IRI component and at least one of the strings listed in the value of the relevant IRI constraint;contains
, meaning that at least one of the strings listed in the value of the relevant IRI constraint appears somewhere in the IRI component.Recognizing the great diversity of potential uses and set definition requirements, multiple IRI constraints are defined relating to the path component. Furthermore, for each constraint there is a ‘negative’ constraint, that is, a constraint whose value is a list of strings that must not be present in the relevant IRI component.
IRI constraint | IRI component | Matching rule | Negative constraint |
---|---|---|---|
includeschemes |
scheme |
exact |
excludeschemes |
includehosts |
ihost |
endsWith |
excludehosts |
includeexactpaths |
ipath |
exact |
excludeexactpaths |
includepathcontains † |
contains |
excludepathcontains |
|
includepathstartswith |
startsWith |
excludepathstartswith |
|
includepathendswith |
endsWith |
excludepathendswith |
|
includeports |
port |
exact |
excludeports |
includepathcontains
may appear any number of times within an IRI set definition, so that it is easy to create one in which multiple strings must be present in paths. This is in contrast to all other terms in Table 3 which can only occur 0 or 1 times, since the IRI of a candidate resource can only have one scheme, one host etc.As a quick example, the set of all resources on example.org
, whether fetched using specifically http
or https
, where the path component of their IRIs starts with foo
, and where the path does not end with .png
or .jpg
is defined thus:
<iriset> <includeSchemes>http https</includeSchemes> <includeHosts>example.org</includeHosts> <includePathStartsWith>/foo</includePathStartsWith> <excludePathEndsWith>.png .jpg</excludePathEndsWith> </iriset>
As outlined in Section 1.4, the POWDER GRDDL transform maps the IRI constraints in Table 3 to regular expressions against which the candidate IRI can be matched. These are shown in Table 4 below where var means the value of the XML element following processing as set out in the formal semantics document [FORMAL]. In brief this turns white space separated lists of strings into alternative values within the regular expression such that:
<includehosts>example.org example.com </includehosts>
becomes
(example\.org|example\.com)
.
IRI Constraint ( include / exclude …) |
Regular Expression |
---|---|
schemes |
^ var\:\/\/ |
hosts |
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)? var(\:([0-9]+))?\/ |
ports |
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)*[^\:\/\?\#\@]+\:var\/ |
exactpaths |
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))? var($|\?|\#) |
pathcontains |
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?\/[^\?\#]* var[^\?\#]*[\?\#]? |
pathstartswith |
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))? var |
pathendswith |
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?\/[^\?\#]* var($|\?|\#) |
These template regular expressions may be useful in processing POWDER documents directly but other methods of determining whether a candidate IRI does or does not match a particular constraint are equally valid.
Example 2-2 below uses the regular expressions from Table 4 in the POWDER-S version of Example 2-1.
1 <owl:Class rdf:nodeID="iriset_1"> 2 <owl:equivalentClass> 3 <owl:Class> 4 <owl:intersectionOf rdf:parseType="Collection"> 5 <owl:Restriction> 6 <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> 7 <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">^(http|https)\:\/\/</owl:hasValue> 8 </owl:Restriction> 9 <owl:Restriction> 10 <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> 11 <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)?(example\.org)(:([0-9]+))?\/</owl:hasValue> 12 </owl:Restriction> 13 <owl:Restriction> 14 <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> 15 <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?(\/foo)</owl:hasValue> 16 </owl:Restriction> 17 <owl:Restriction> 18 <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#notmatchesregex" /> 19 <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?\/[^\?\#]*(\.png|\.jpg)($|\?|\#)</owl:hasValue> 20 </owl:Restriction> 21 </owl:intersectionOf> 22 </owl:Class> 23 </owl:equivalentClass> 24 </owl:Class>
Note the use of notmatchesregex
in line 18 to encode the excludepathendswith
element.
includeports
and excludeports
Although ports are clearly integers, POWDER treats them as a string in the same way as the other constraints in Table 3. Port ranges (such as 80-100) are not supported but note that the value of includeports
and excludeports
is a white space separated list so that multiple ports may be enumerated.
includequerycontains
and excludequerycontains
Query strings typically contain a series of name-value pairs separated by ampersands thus:
?name1=value1&name2=value2
These are usually acted on by the server to generate content in real time and the order of the name-value pairs is unimportant. For practical purposes ?name1=value1&name2=value2
is equivalent to ?name2=value2&name1=value1
. As a result, a significant amount of processing must be done to determine whether or not a candidate IRI is or is not an element of an IRI set that includes either the includequerycontains
or excludequerycontains
IRI Constraints.
To keep such processing manageable, the includequerycontains
and excludequerycontains
IRI Constraints take a single value not a white space separated list of values. Including two or more
name/value pairs in an includequerycontains
or excludequerycontains
means that all those
pairs must be present in the query string for the candidate IRI to be a member of the set.
Section 2.6 includes a further discussion on creating unions of multiple IRI sets
where it is necessary to create a set defined in terms of alternative name/value pairs.
By default, the POWDER GRDDL transform assumes that the delimiting character in a query string is the ampersand (&
). However, an alternative delimiter can be specified as the value for the delimiter
attribute on includequerycontains
and excludequerycontains
constraints. Example 2-3 below shows this.
<iriset> <includehosts>socialnetwork.example.com</includehosts> <includequerycontains delimiter=",">id=abcdef,group=12345</includequerycontains> </iriset>
The GRDDL transform splits the value provided for the includequerycontains
or excludequerycontains
IRI Constraints into its constituent pairs at the delimiting character and the presence of each name-value pair within the candidate IRI is then tested for independently. The template regular expression for such a test is:
\:\/\/(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?\/[^\?\#]*\?([^\#]*d)?q(d|$)
Where d is the delimiting character and q is the name-value pair. The Formal Semantics document [FORMAL] sets this out in more detail.
An important consequence of this processing model is that within the query string, only complete name-value pairs or value-less parameters are matched. More precisely, only complete query conjuncts in the query string are matched. As complete query conjuncts we consider any minimal substring of the query string that has ?
or d before the first character and d or $
after the last character, where, as in the template regular expression, d is the query delimiter and $
is the end-of-string.
If the value of includequerycontains in Example 2-3 were changed to simply abcdef
(rather than id=abcdef,group=12345
) then:
http://socialnetwork.example.com/?group=12345,id=abcdef
would not match, even though the query does contain the string abcdef
http://socialnetwork.example.com/?abcdef
would match.http://socialnetwork.example.com/?abcdef=ijklm
would not matchAgain, a POWDER processor may use alternative methods to determine whether a given name-value pair is present in a candidate IRI but the template regular expression is used in the GRDDL transform to generate the POWDER-S shown in Example 2-4. Notice that the pre-processing described here allows POWDER-S to use the same restriction on the wdrs:matchesregex
data property as the other elements in Table 3.
<owl:Class rdf:nodeID="iriset_1"> <owl:equivalentClass> <owl:Class> <owl:intersectionOf rdf:parseType="Collection"> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)?(socialnetwork\.example\.com)(:([0-9]+))?\/</owl:hasValue> </owl:Restriction> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?\/[^\?\#]*\?([^\#]*,)?id=abcdef(,|$)</owl:hasValue> </owl:Restriction> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]*)(\:([0-9]+))?\/[^\?\#]*\?([^\#]*,)?id=abcdef(,|$)</owl:hasValue> </owl:Restriction> </owl:intersectionOf> </owl:Class> </owl:equivalentClass> </owl:Class>
includehosts/excludehosts
Equivalence of internationalized domain names is defined in terms of ASCII case insensitive
comparison of the output of the ToASCII
operation (section 2 of [RFC 3490]).
If a regular expression is constructed to match the ihost component of an IRI, then it MUST be constructed to match the ASCII representation of that component.
Specifically, to generate a regular expression from an includehosts
or excludehosts
constraint,
the value of each space separated token under the ToASCII
operation, with the
UseSTD3ASCIIRules
flag unset and the AllowUnassigned
flag set, MUST
be used. [RFC 3490]. Behavior if the ToASCII
operation fails is undefined.
When internationalized (non-ASCII) domain names appear in the values of the includehosts
and excludehosts
constraints, they SHOULD be represented either as the output of a successful
ToASCII
operation, or as a string of Unicode characters that is mapped to itself when the
ToASCII
and ToUnicode
operations are successfully applied, in this order.
(Note, in particular, that behavior for these strings is well-defined, since the ToASCII
operation must succeed by definition.)
Before determining whether an IRI is or is not a member of a given set, it is important to make sure that the IRI is represented in the same way as the data against which it is being matched. We set out below a series of steps that are designed to help processors make a best effort in this regard, which are consistent with RFC 3986 [URIS], RFC 3987 [IRIS] and URISpace [URISpace].
%c3%a7
becomes ç
etc.). Note the hexadecimal digits are case-insensitive.
However space characters (%20) and reserved characters as per Section 2.2 of RFC 3986 [URIS]
must not be converted to literals, as that may invalidate the URI/IRI - the reason a URI would contain
(for example) %2F
instead of /
would be to distinguish between a literal /
,
such as in 'his/hers', and the /
which is used as a path separator.Input IRI/URI | Canonical form |
---|---|
http://example.com/staff/Fran%c3%a7ois |
http://www.example.com/staff/François |
http://example.com/my%20doc.doc |
http://www.example.com/my doc.doc |
In this next example the %2F is a literal slash, not a path separator, and so is left as %2F |
|
http://www.example.com/foo/his%2Fhers |
http://www.example.com/foo/his%2Fhers |
authority
is present, but the scheme
is absent, the scheme
should default to http
.path
is absent, a path
of /
is appended..
characters in the host
are removed, i.e. http://www.example.com./
becomes http://www.example.com/
ToASCII
operation to the host
string, with the UseSTD3ASCIIRules
flag unset and
the AllowUnassigned
flag set [RFC 3490].
Note that behavior if the ToASCII
operation fails is undefined.scheme
and host
strings to ASCII lowercase.port
is specified, but it is the default port for the scheme, it is removed.The following table gives some examples.
Input IRI | Canonical form |
---|---|
www.example.com | http://www.example.com/ |
http://www.example.com | http://www.example.com/ |
HTTPS://WWW.EXAMPLE.COM/FOO | https://www.example.com/FOO |
http://www.example.com./foo | http://www.example.com/foo |
http://www.example.com:80/foo | http://www.example.com/foo |
http://sigmaσ.example.org/ | http://xn--sigma-kde.example.org |
To complement the IRI canonicalization steps described in the previous section, related processing steps must also be carried out on the strings supplied as set defining data. The following steps should therefore be applied to each item in the list separately.
host
, trailing .
characters are removed.host
, and does not completely consist of ASCII characters,
the ToASCII
operation is applied as described in Section 2.1.3scheme
or host
, it is normalized to
ASCII lower case.includepathstartswith
, excludepathstartswith
, includeexactpaths
or excludeexactpaths
must begin with the /
character which is pre-pended if absent.If the IRI set definition includes values related to the port
then matching of the data against the candidate resource's IRI must be carried out as follows:
includeports
then, when matching, if the default port for the candidate resource's IRI is present in the list of supplied values, but the candidate resource's IRI does not specify the port, the candidate resource IS an element of the set IF all other conditions are met.excludeports
then, when matching, if the default port for the candidate resource's IRI is present in the list of supplied values, but the candidate resource's IRI does not specify the port, the candidate resource is NOT an element of the IRI Set.includeiripattern
and excludeiripattern
IRI constraintsIt is anticipated that resource groups will typically be defined in terms of the
domains and sub domains from which they are available. In order to provide as much flexibility as possible
in this regard, the includeiripattern
and
excludeiripattern
properties allow domains and sub-domains
to be substituted by a wildcard character (*) according to the following EBNF:
iri-pattern ::= (scheme "://")? domain-pattern (":" port)? | "*" domain-pattern ::= domain | "*." domain
scheme
and port
are used as defined in
RFC 3986 [URIS]. domain
is an internationalized
domain name as defined in RFC 3490 [RFC3490].
It follows that:
<includehosts>example.com</includehosts>
and
<includeiripattern>example.com</includeiripattern>
are equivalent. However, *.example.com
, meaning resources on sub-domains of example.com
but not on example.com
itself, is not a valid value for includehosts
.
In contrast to the IRI constraints shown in Table 3, includeiripattern
and excludeiripattern
take a single pattern, not a white space separated list of values. Note that paths and query strings MUST NOT be included in the pattern. If these are required in an IRI set definition, the relevant IRI constraints from Table 3 can be used.
Any processing method that accurately tests a candidate IRI against the value of an includeiripattern
or
excludeiripattern
element is valid but the POWDER GRDDL transform does it in the same way as the other
IRI constraints, namely by creating a restriction on the
wdrs:matchesregex
and wdrs:notmatchesregex
properties as shown
in the example below. Full details of the transformation are provided in the Formal Semantics
document [FORMAL] however it is worth noting here once again that
if a regular expression is constructed to match the host
component of an IRI, then
it MUST be constructed to match the ASCII representation of that component.
POWDER
<iriset> <includeiripattern>http://example.org</includeiripattern> <excludeiripattern>search.example.com:81</excludeiripattern> </iriset>
POWDER-S
<owl:Class rdf:nodeID="iriset_1">
<owl:equivalentClass>
<owl:Class>
<owl:intersectionOf rdf:parseType="Collection">
<owl:Restriction>
<owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" />
<owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">^http\:\/\/([^\:\/\?\#\@]+\.)+example.org(\:[0-9]+)?</owl:hasValue>
</owl:Restriction>
<owl:Restriction>
<owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#notmatchesregex" />
<owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">^[A-Za-z]+\:\/\/([^\:\/\?\#\@]+\.)*search.example.com\:81</owl:hasValue>
</owl:Restriction>
</owl:intersectionOf>
</owl:Class>
</owl:equivalentClass>
</owl:Class>
includeregex
and excluderegex
IRI constraintsIt is believed that the IRI constraints discussed above will be easy to use and cover the overwhelming
majority of POWDER use cases. However, the use of strings with fixed matching rules clearly presents a restriction
on flexibility. To support fully flexible set definition by IRI, the
includeregex
and excluderegex
properties take a Regular Expression and should be applied to the candidate resource's complete IRI (after following
the canonicalization steps above and ensuring that regular expressions are
constructed using the ASCII representation of IRI components related to the host
as
discussed in Section 2.1.3). In common with
includepathcontains
and excludepathcontains
, and unlike the other IRI constraints,
includeregex
and excluderegex
may occur any number of times in an IRI set. This
follows from the GRDDL transformation associated with POWDER introduced in
Section 1.4. The same section notes that the regular expression
syntax used is that defined by XML schema as modified by XQuery 1.0 and XPath 2.0 Functions and
Operators [XQXP].
For POWDER-S, the regular expressions are copied verbatim as values
for the wdrs:matchesregex
and wdrs:notmatchesregex
properties.
N.B. The value of the includeregex
and excluderegex
properties MUST be a single Regular Expression, not a white space-separated list.
As an example, the set of all the resources hosted either by example.org
or example.net
, where the path component of their IRIs starts either with foo
or bar
, can be defined thus:
POWDER:
<iriset> <includeregex>^(([^:/?#]+):)//([^:/?#]+.)?example.(org|net)/(foo|bar)</includeregex> </iriset>
POWDER-S:
<owl:Class rdf:nodeID="iriset_1"> <owl:equivalentClass> <owl:Class> <owl:intersectionOf rdf:parseType="Collection"> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">^(([^:/?#]+):)//([^:/?#]+.)?example.(org|net)/(foo|bar)</owl:hasValue> </owl:Restriction> </owl:intersectionOf> </owl:Class> </owl:equivalentClass> </owl:Class>
It is important to note that Example 2-6 does not take account of the need to escape certain characters.
The following characters are used as meta characters in regular expressions and MUST therefore be escaped if used in a pattern given as the value of the includeregex
property:
. \ ? * + { } ( ) [ ]
In addition, the < (less than) and & (ampersand) characters MUST always be escaped to < and & as per the XML specification [XML].
As a safeguard against unintended consequences, other characters that always or typically have special meaning within IRI strings and/or XML SHOULD also be escaped, namely:
! " # % ' , - / : ; = > @ [ ] _ ` ~
As a result, Example 2-6 should properly be written as shown in Example 2-7 below.
POWDER:
<iriset> <includeregex>^(([^\:\/\?\#]+)\:)//([^\:\/\?\#]+\.)?example\.(org|net)/(foo|bar)</includeregex> </iriset>
POWDER-S:
<owl:Class rdf:nodeID="iriset_1"> <owl:equivalentClass> <owl:Class> <owl:intersectionOf rdf:parseType="Collection"> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">^(([^\:\/\?\#]+)\:)//([^\:\/\?\#]+\.)?example\.(org|net)/(foo|bar)</owl:hasValue> </owl:Restriction> </owl:intersectionOf> </owl:Class> </owl:equivalentClass> </owl:Class>
includeregex
Example 2-7 uses a modified version of the regular expression given Section 2.1. This is the safest method but is not, perhaps, the most natural way to proceed. If a less rigorous approach is taken it is easy to make mistakes when specifying regular expressions, and incorrect regular expressions in set definitions will have one of two possible (and obvious) consequences
Example 2-8 below shows how this can happen.
<iriset> <includehosts>example.org</includehosts> <includeregex>https</includeregex> </iriset>
The intention of the regular expression given in Example 2-8 is probably to say "all resources on
example.org
with a URI beginning with https
." However, as the regular expression is not
anchored at either end, what this actually means is "all resources on example.org
where the URI
includes https
". Thus this IRI set includes both of:
https://www.example.org/page.html
http://www.example.org/why_we_use_https.html
Adding in anchors at the beginning and end of the regular expression can have equally undesirable consequences.
<iriset> <includehosts>example.org</includehosts> <includeregex>^https$</includeregex> </iriset>
In Example 2-9, the intention is, again probably, to define the set of "all resources on example.org
fetched using https
only". However, adding both the ^
and $
anchors at the beginning and end of the regular expression means that the whole IRI must be https
from start to finish — which can never be true so this IRI set is equivalent to the empty set.
Example 2-10 shows one possible way to encode the intended set definition.
<iriset> <includehosts>example.org</includehosts> <includeregex>^https</includeregex> </iriset>
Whilst Example 2-10 'works', the potential dangers of using regular expressions mean that it is generally better to use component strings where possible. Example 2-10 is therefore better written as shown in Example 2-11 below.
<iriset> <includehosts>example.org</includehosts> <includeschemes>https</includeschemes> </iriset>
It is noteworthy that POWDER does not define any special procedures where the host component of an IRI is expressed as an IP address. These are treated as strings, not as a sequence of digits. If the intention is to define an IRI set that encompasses a particular group of resources however they are accessed then it may be appropriate to include both the domain name and associated IP address as two space separated values in an includehosts
element for example. However, this assumes that there is a one to one relationship between the domain name and the IP address which, of course, is often not the case.
As noted in Section 1.2, POWDER defines sets of IRIs, not of the resources that they identify. IRI sets must therefore be defined with care. For operational reasons, a user agent MAY perform a DNS or reverse DNS lookup to match domains names and IP addresses but this is very much application-specific.
includeresources
and excluderesources
ConstraintsIt is useful to be able to include or exclude IRIs from sets by simple listing. The includeresources
and excluderesources
constraints support this, both of which take white space separated lists of IRIs. To give a simple example, the set of all resources on example.org
except its stylesheet and JavaScript library can be encoded as shown in Example 2-12 below.
<iriset> <includehosts>example.org</includehosts> <excluderesources>http://www.example.org/stylesheet.css http://www.example.org/jslib.js</excluderesources> </iriset>
The white space separated list of values is processed as set out in the Formal Semantics document [FORMAL] to create a pattern var that can be inserted into the simple template regular expression:
^var$
Thus Example 2-12 is transformed into the following POWDER-S.
<owl:Class rdf:nodeID="iriset_1"> <owl:equivalentClass> <owl:Class> <owl:intersectionOf rdf:parseType="Collection"> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#matchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)?(example\.org)(:([0-9]+))?\/</owl:hasValue> </owl:Restriction> <owl:Restriction> <owl:onProperty rdf:resource="http://www.w3.org/2007/05/powder-s#notmatchesregex" /> <owl:hasValue rdf:datatype="http://www.w3.org/2001/XMLSchema-datatypes#string">^(http\:\/\/www\.example\.org\/stylesheet\.css|http\:\/\/www\.example\.org\/jslib\.js)$</owl:hasValue> </owl:Restriction> </owl:intersectionOf> </owl:Class> </owl:equivalentClass> </owl:Class>
As emphasized throughout this document, each constraint and its value creates a set definition of its own and the full IRI set is the intersection of those sets. Thus an alternative way of looking at Example 2-12 is to say that a candidate IRI is a member of the IRI set IF it is on example.org
AND does not have the IRI http://www.example.org/stylesheet.css
AND does not have the IRI http://www.example.org/jslib.js
.
POWDER allows a DR to express any grouping of resources whatsoever, no matter how complex.
Atomic negation is achieved by complementing each IRI constraint that includes certain IRI
components by one that excludes them, and vice versa; furthermore, all includeX
and excludeX
constraints
are mutually exclusive. The analogous properties matchesregex
and notmatchesregex
properties are used in POWDER-S. Negation of complex constraints is not supported.
Conjunction of atomic propositions (both positive and negative) is inherent in the basic model - an IRI must match all the constraints if it is to be an element of the set. The GRDDL transform uses owl:intersectionOf
to render in POWDER-S iriset
elements with multiple constraints.
The disjunction of conjunctions of atomic propositions (both positive and negative) is also possible, as a DR may contain multiple iriset
elements, and if any of them holds, then the DR holds. The GRDDL transform encodes multiple iriset
elements as multiple clauses in POWDER-S.
It follows from the above, that POWDER allows the expression of Disjunctive Normal Form propositions. Since arbitrarily complex propositions can be brought into DNF (DNF Theorem), it follows that POWDER allows the expression of any proposition.
Example 2-14 shows a Description Resource defining the set of IRIs on example.com with a path beginning with /foo and those on example.org where the path starts with /bar.
<?xml version="1.0"?> <powder xmlns="http://www.w3.org/2007/05/powder#" xmlns:ex="http://example.org/vocab#"> <attribution> <issuedby src="http://authority.example.org/company.rdf#me" /> <issued>2007-12-14T00:00:00</issued> </attribution> <dr> <iriset> <includehosts>example.com</includehosts> <includepathstartswith>/foo</includepathstartswith> </iriset> <iriset> <includehosts>example.org</includehosts> <includepathstartswith>/bar</includepathstartswith> </iriset> <descriptorset> <ex:color>red</ex:color> <ex:shape>square</ex:shape> <displaytext>Everything on example.com where the path starts with /foo and everything on example.org where the path starts with /bar is red and square</displaytext> <displayicon>http://example.org/icon.png</displayicon> </descriptorset> </dr> </powder>
In this document we have specified various methods for defining sets of resource identifiers. The elements are clearly designed to be used with information resources available on the Web, identified by IRIs containing host names, directory paths, port numbers, and so on. The POWDER grouping vocabulary can be easily extended by new elements, defined via GRDDL transformation, which build upon the elements defined by POWDER. As examples, in Sections 3.1 and 3.2 we show how other methods of defining IRI sets that may suit particular situations can be transformed into POWDER-BASE.
Furthermore, there is no fundamental reason to constrain the domain of POWDER descriptions to HTTP IRIs, so there should not be unnecessary constraints on how the protocol works. In other words, the domain of grouping extensions does not need to be HTTP IRIs, but may be any kind of IRIs. As an example, in Section 3.3 we show such an extension for ISAN numbers.
It should be noted that the treatment of non-HTTP IRIs is one of the basic motivations behind the two-step GRDDL transform from POWDER to POWDER-BASE to POWDER-S, outlined in
Section 1.4 and fully specified in the Formal Semantics document [FORMAL]. If POWDER were rendered into POWDER-S in a single direct transform, the only XML language from which to derive extensions would be POWDER, which would oblige POWDER extensions to include HTTP-specific IRI restrictions such as includehosts
, even if they are meaningless for the domain of the extension.
In the intermediate POWDER-BASE language, on the other hand, all HTTP-specific elements have been rendered as regular expressions, using the includeregex
and excluderegex
IRI restrictions, as POWDER-BASE only requires that these two restrictions are supported. Developers of non-HTTP extensions and tools are advised to use POWDER-BASE to derive their extension from, instead of POWDER, as this relieves them of the obligation to also implement the HTTP-specific IRI restrictions in their tools.
XML elements suitable for defining sets of URIs or IRIs from schemes other than HTTP may be created
and a GRDDL transform defined that renders such IRI sets in POWDER-BASE. This is an generic extension mechanism since a conformant POWDER Processor, as defined in the Description Resources document [DR], MUST be able to process POWDER-BASE. For clarity: POWDER-BASE is not a separate encoding of POWDER — it is all done in the wdr
namespace — merely a restricted form of POWDER that just has the two possible child elements of iriset
.
Developers of POWDER tools MAY directly implement extensions they know about, and MAY include support for transformation technologies such as XSLT so that unknown extensions can be processed.
As an example of a service-specific extension, consider a service which uses unix shell wildcards instead of regular expressions, so that www.example.org/*
means "all the resources on www.example.org fetched using HTTP." Such a system is easily used within an IRI set, only requiring the definition of a near copy of the POWDER schema [WDR] with a single IRI constraint shell:includepattern
as child element of its IRI set element (good practice when defining shell:includepattern
would be to also define shell:excludepattern
).
A publisher of a document using shell:includepattern
SHOULD define a GRDDL transform
that will generate a POWDER-BASE document as shown the example below.
Custom IRI pattern:
<shell:iriset> <shell:includepattern>www.example.org/*</shell:includepattern> </shell:iriset>
POWDER-BASE:
<iriset> <includeregex>http\:\/\/www\.example\.org\/.*</includeregex> </iriset>
Note that the custom IRI pattern SHOULD NOT be used in a document with its root element in the POWDER namespace since the only valid child elements of the iriset
element within a POWDER document are those defined in this document.
Many content providers serve dynamic content stored in a database, so that IRIs express queries to that database. This kind of IRI will have certain structure but this is typically neither obvious nor easily human-interpreted.
As an example, consider sport.example.com
, a sports news site, where IRIs look like the one shown in Example 3-2. The adopted scheme is systematic so that sport=2&countryID=16
provides a front page with news about Greek basketball and links to various Greek basketball leagues, sport=3&countryID=16
a front page about Greek volleyball, etc.
http://sport.example.com/matches.asp?sport=1&countryID=16&champID=2
A POWDER document providing metadata about this Web site would have to use regular expression matching with explicit reference to the numerical values in the country
and sport
fields of the query. This process is error-prone, and requires extensive changes if the underlying database schema is modified or extended.
As an alternative, the site developer may provide a POWDER-like scheme that abstracts away from the specific database fields to allow reference to sports and countries, as shown in Example 3-3. Description Resource authors can then use the properties in this extension to generate POWDER-BASE documents that are valid even if the site schema is modified, as long as the site developer updates the relevant transformations.
Custom IRI constraint:
<sport:iriset> <wdr:includehosts>sport.example.com</wdr:includehosts> <sport:countries>Greece</sport:countries> <sport:sports>Football Basketball</sport:sports> </sport:iriset>
Corresponding POWDER-BASE IRI set:
<iriset> <includeregex>(([^\/\?\#]*)\@)?([^\:\/\?\#\@]+\.)?(sport\.example\.com)(:([0-9]+))?\/</includeregex> <includeregex>country=16</includeregex> <includeregex>sport=[1|2]</includeregex> </iriset>
The International Standard Audiovisual Number [ISAN1] is a globally-unique, centrally managed and permanent numbering system for the identification of audiovisual works and versions. Following ISO 15706 [ISAN3], [ISAN3-2], the ISAN numbers are written as 24 bit hexadecimal digits in the following format [ISAN2].
-----root----- |
episode |
-version- |
||||
ISAN |
1881-66C7-3420 |
- |
0000 |
-7- |
9F3A-0245 |
-U |
The root segment of an ISAN number is assigned to a core work. When the core work is a serial, episodes are identified with a non null episode segment. Versions are assigned in the version segment and refer to changes in the audiovisual content, being a different language or soundtrack, subtitles, editions, promotional trailers, and so on.
Since ISAN numbers are URNs [URN], and hence IRIs of the urn:
scheme [URIS], a vocabulary can readily be defined to allow IRI Sets to be defined based on ISAN numbers. The terms might be along the lines of:
includeRoots
— the value of which would be a white space separated of hexadecimal digits and hyphens that would be matched against the first three blocks in the ISAN number.
includeEpisodes
— a white space separated list of hexadecimal digits and hyphens that would be matched against the 4th block of 4 digits in the ISAN number.
includeVersions
— a white space separated list of hexadecimal digits and hyphens that would be matched against the 5th and 6th blocks of 4 digits in the ISAN number.
The set of all audio visual resources that relate to two particular works might then be defined as shown in Example 3-4.
Custom ISAN pattern:
<ex_isan:iriset> <ex_isan:includeRoots>1881-66C7-3420 1881-66C7-3421</ex_isan:includeRoots> </ex_isan:iriset>
Corresponding POWDER-BASE IRI Set:
<iriset> <includeregex>^urn:isan:(1881-66C7-3420)|(1881-66C7-3421)</includeregex> </iriset>
An IRI set definition is a Conformant IRI set definition if it adheres to the specification described in this document.
More precisely:
The editors duly acknowledge the earlier work in this area carried out by Jo Rabin. Jeremy Carroll and David Booth developed the operational and formal semantics model which was further developed by Stasinos Konstantopoulos. Eric Prud'hommeaux, Addison Phillips and Thomas Roessler made significant contributions to the development of the canonicalization section. The editors gratefully acknowledge the further contributions made by Régis Flad of ISANIA and members of the POWDER Working Group.
Changes since the 3 April 2009 draft are as follows:
Element Name | Content | Attributes | Cardinality | Introduced |
---|---|---|---|---|
iriset |
Any of
includeschemes ,
excludeschemes ,
includehosts ,
excludehosts ,
includeexactpaths ,
excludeexactpaths ,
includepathcontains ,
excludepathcontains ,
includepathstartswith ,
excludepathstartswith ,
includepathendswith ,
excludepathendswith ,
includeports ,
excludeports
|
At least 1 must be a child element of a dr |
Section 1.3 | |
includeschemes |
Token list | 0 or 1 | Section 2.1 | |
excludeschemes |
||||
includehosts |
||||
excludehosts |
||||
includeexactpaths |
||||
excludeexactpaths |
||||
includepathcontains |
Token list | any number | ||
excludepathcontains |
||||
includepathstartswith |
Token list | 0 or 1 | ||
excludepathstartswith |
||||
includepathendswith |
||||
excludepathendswith |
||||
includeports |
||||
excludeports |
||||
includequerycontains |
Single value | delimiter (any single character). Default value is: & (“ampersand”) |
0 or 1 | Section 2.1.2 |
excludequerycontains |
||||
includeiripattern |
Single value | 0 or 1 | Section 2.2 | |
excludeiripattern |
||||
includeregex |
Single value | Any number | Section 2.3 | |
excluderegex |
||||
includeresources |
Token list | 0 or 1 | Section 2.5 | |
excluderesources |