Issue PATHNAME-WILD Writeup

Status: Passed, as amended, Jun 89 X3j13

Issue:        PATHNAME-WILD

Forum:        Cleanup

References:   Pathnames (pp410-413)

Related issues: PATHNAME-COMPONENT-VALUE, PATHNAME-LOGICAL

Category:     ADDITION

Edit history: 21-Jul-88, Version 1 by Pitman

              06-Oct-88, Version 2 by Pitman

               9-May-89, Version 3 by Moon (small fixes)

              10-May-89, Version 4 by Moon (add two more functions)

              13-May-89, Version 5 by Moon (minor cleanups, add clarification)

              19-Jun-89, Version 6 by Moon (revise based on extensive

                        discussion in the cleanup subcommittee; rewrite

                        the description of TRANSLATE-PATHNAME so it is

                        possible to understand it)

              23-Jun-89, Version 7 by Moon (simplify TRANSLATE-PATHNAME

                        based on last minute discussion of logical pathnames)

	       2-Jul-89, Version 8 by Masinter (T -> True as per Jun89 X3J13)

	

Problem Description:


  Some file systems provide more complex conventions for wildcards than

  simple component-wise wildcards (:WILD). For example,


  "F*O" might mean:

    - a normal three character name

    - a three-character name, with the middle char wild

    - at least a two-character name, with the middle 0 or more chars wild

    - a wild match spanning multiple directories


  ">foo>*>bar" might imply:

    - the middle directory is named "*"

    - the middle directory is :WILD

    - there may be zero or more :WILD middle directories

    - the middle directory name matches any one-letter name


  ">foo>**>bar" might mean

    - the middle directory is named "**"

    - the middle directory is :WILD

    - there may be zero or more :WILD middle directories

    - the middle directory name matches any two-letter name


  Some file systems support even more complex wildcards, for example

  regular expressions.


  The CL pathname model does not specify a way to represent complex

  wildcards, which means, for example, that (MAKE-PATHNAME :NAME "F*O")

  cannot be recognized by portable code as containing a wildcard.


  Common Lisp provides only the first of these four common operations

  on wildcard pathnames:

  (1) Enumerate the set of existing files that match the pathname;

  this is provided by the DIRECTORY function.

  (2) Test whether a pathname contains wildcards.

  (3) Test whether a pathname matches a wildcard pathname.

  (4) Translate one pathname into another according to a mapping specified

  by a pair of wildcard pathnames.


Proposal (PATHNAME-WILD:NEW-FUNCTIONS):


  Introduce the following three functions:


  1. WILD-PATHNAME-P pathname &optional field-key

  

    Tests a pathname for the presence of wildcard components.  If the first

    argument is not a pathname, string, or file stream an error of type

    TYPE-ERROR is signalled.

  

    If no <field-key> is provided, or the <field-key> is NIL, the result is

    true if <pathname> has any wildcard components, NIL if <pathname> has none.

    If a non-null <field-key> is provided, it must be one of :HOST, :DEVICE,

    :DIRECTORY, :NAME, :TYPE, or :VERSION.  In this case, the result is true

    if the indicated component of <pathname> is a wildcard, NIL if the

    component is not a wildcard.  Note that not all implementations

    support wildcards in all fields, according to PATHNAME-COMPONENT-VALUE.



  2. PATHNAME-MATCH-P pathname wildcard


    true if <pathname> matches <wildcard>, otherwise NIL.  The matching rules

    are implementation-defined but should be consistent with the

    DIRECTORY function.  Missing components of <wildcard> default to :WILD.


    If either argument is not a pathname, string, or file stream an error

    of type TYPE-ERROR is signalled.  It is valid for <pathname> to be a

    wild pathname; a wildcard field in <pathname> will only match a

    wildcard field in <wildcard>, i.e. the function is not commutative.

    It is valid for <wildcard> to be a non-wild pathname.



  3. TRANSLATE-PATHNAME source from-wildcard to-wildcard &key


    Translates the pathname <source>, which matches <from-wildcard>, into

    a corresponding pathname <result>, which matches <to-wildcard>, and

    returns <result>.


    The pathname <result> is <to-wildcard> with each wildcard or missing

    field replaced by a portion of <source>.  A "wildcard field" is a

    pathname component with a value of :WILD, a :WILD element of a

    list-valued directory component, or an implementation-defined portion

    of a component, such as the "*" in the complex wildcard string

    "foo*bar" that some implementations support.  An implementation that

    adds other wildcard features, such as regular expressions, must define

    how TRANSLATE-PATHNAME extends to those features.  A "missing field" is

    a pathname component with a value of NIL.


    The portion of <source> that is copied into <result> is implementation

    defined.  Typically it is determined by the user interface conventions

    of the file systems involved.  Usually it is the portion of <source>

    that matches a wildcard field of <from-wildcard> that is in the same

    position as the wildcard or missing field of <to-wildcard>.  If there

    is no wildcard field in <from-wildcard> at that position, then usually

    it is the entire corresponding pathname component of <source>, or in

    the case of a list-valued directory component, the entire corresponding

    list element.  For example, if the name components of <source>,

    <from-wildcard>, and <to-wildcard> are "gazonk", "gaz*", and "h*"

    respectively, then in most file systems, the wildcard fields of the

    name component of <from-wildcard> and <to-wildcard> are each "*", the

    matching portion of <source> is "onk", and the name component of

    <result> is "honk".  However, the exact behavior of TRANSLATE-PATHNAME

    cannot be dictated by the Common Lisp language and must be allowed to

    vary, depending on the user interface conventions of the file systems

    involved.


    During the copying of a portion of <source> into <result>, additional

    implementation-defined translations of alphabetic case or file naming

    conventions might occur, especially when <from-wildcard> and

    <to-wildcard> are for different hosts.


    If any of the first three arguments is not a pathname, string, or file

    stream an error of type TYPE-ERROR is signalled.  It is valid for

    <source> to be a wild pathname; in general this will produce a wild

    result.  It is valid for <from-wildcard> and/or <to-wildcard> to be

    non-wild pathnames.  (PATHNAME-MATCH-P <source> <from-wildcard>) must

    be true or an error is signalled.

    

    There are no specified keyword arguments for TRANSLATE-PATHNAME, but

    implementations are permitted to extend it by adding keyword arguments.

    There is one specified return value from TRANSLATE-PATHNAME;

    implementations are permitted to extend it by returning additional

    values.


    Implementation guideline: one file system performs this operation by

    examining each piece of the three pathnames in turn, where a piece is a

    pathname component or a list element of a structured component such as

    a hierarchical directory.  Hierarchical directory elements in

    <from-wildcard> and <to-wildcard> are matched by whether they are

    wildcards, not by depth in the directory hierarchy.  If the piece in

    <to-wildcard> is present and not wild, it is copied into the result.

    If the piece in <to-wildcard> is :WILD or NIL, the piece in <source> is

    copied into the result.  Otherwise, the piece is <to-wildcard> might be

    a complex wildcard such as "foo*bar" and the piece in <from-wildcard>

    should be wild; the portion of the piece in <source> that matches the

    wildcard portion of the piece in <from-wildcard> replaces the wildcard

    portion of the piece in <to-wildcard> and the value produced is used in

    the result.


  4. Clarify that the functions OPEN (and WITH-OPEN-FILE), PROBE-FILE,

  FILE-WRITE-DATE, FILE-AUTHOR, and TRUENAME only accept non-wildcard

  pathnames and signal an error if given a pathname for which

  WILD-PATHNAME-P returns true.


  5. Clarify that the functions RENAME-FILE, DELETE-FILE, LOAD, and

  COMPILE-FILE have implementation-defined consequences when given a

  wildcard pathname.  Each function might signal an error or might operate

  on all files that match the wildcard pathname.



Examples:


  ;The following examples are not portable.  They are written to run

  ;with particular file systems and particular wildcard conventions.

  ;Other implementations will behave differently.  These examples are

  ;intended to be illustrative, not to be prescriptive.


  (WILD-PATHNAME-P (MAKE-PATHNAME :NAME :WILD)) => T

  (WILD-PATHNAME-P (MAKE-PATHNAME :NAME :WILD) :NAME) => T

  (WILD-PATHNAME-P (MAKE-PATHNAME :NAME :WILD) :TYPE) => NIL

  (WILD-PATHNAME-P (PATHNAME "S:>foo>**>")) => T   ;Lispm

  (WILD-PATHNAME-P (PATHNAME :NAME "F*O")) => T    ;Most places


  ;This example assumes one particular set of wildcard conventions

  ;Not all file systems will run this example exactly as written

  (DEFUN RENAME-FILES (FROM TO)

    (DOLIST (FILE (DIRECTORY FROM))

      (RENAME-FILE FILE (TRANSLATE-PATHNAME FILE FROM TO))))

  (RENAME-FILES "/usr/me/*.lisp" "/dev/her/*.l")

    ;Renames /usr/me/init.lisp to /dev/her/init.l

  (RENAME-FILES "/usr/me/pcl*/*" "/sys/pcl/*/")

    ;Renames /usr/me/pcl-5-may/low.lisp to /sys/pcl/pcl-5-may/low.lisp

    ;In some file systems the result might be /sys/pcl/5-may/low.lisp

  (RENAME-FILES "/usr/me/pcl*/*" "/sys/library/*/")

    ;Renames /usr/me/pcl-5-may/low.lisp to /sys/library/pcl-5-may/low.lisp

    ;In some file systems the result might be /sys/library/5-may/low.lisp

  (RENAME-FILES "/usr/me/foo.bar" "/usr/me2/")

    ;Renames /usr/me/foo.bar to /usr/me2/foo.bar

  (RENAME-FILES "/usr/joe/*-recipes.text" "/usr/jim/cookbook/joe's-*-rec.text")

    ;Renames /usr/joe/lamb-recipes.text to /usr/jim/cookbook/joe's-lamb-rec.text

    ;Renames /usr/joe/pork-recipes.text to /usr/jim/cookbook/joe's-pork-rec.text

    ;Renames /usr/joe/veg-recipes.text to /usr/jim/cookbook/joe's-veg-rec.text


  ;This example assumes one particular set of wildcard conventions

  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "foo*" "*baz")) => "barbaz"

  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "foo*" "*"))    => "foobar"

  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "*"    "foo*")) => "foofoobar"

  (PATHNAME-NAME (TRANSLATE-PATHNAME "bar"    "*"    "foo*")) => "foobar"


  ;Using Unix syntax and the wildcard conventions used by the

  ;particular version of Unix on which I tried this:

  (NAMESTRING

    (TRANSLATE-PATHNAME "/usr/dmr/hacks/frob.l"

                        "/usr/d*/hacks/*.l"

                        "/usr/d*/backup/hacks/backup-*.*"))

   => "/usr/dmr/backup/hacks/backup-frob.l"

  (NAMESTRING

    (TRANSLATE-PATHNAME "/usr/dmr/hacks/frob.l"

                        "/usr/d*/hacks/fr*.l"

                        "/usr/d*/backup/hacks/backup-*.*"))

   => "/usr/dmr/backup/hacks/backup-ob.l"


  ;This is similar to the above example but uses two different hosts,

  ;U: which is a Unix and V: which is a VMS.  Note the translation

  ;of file type and alphabetic case conventions.

  (NAMESTRING

    (TRANSLATE-PATHNAME "U:/usr/dmr/hacks/frob.l"

                        "U:/usr/d*/hacks/*.l"

                        "V:SYS$DISK:[D*.BACKUP.HACKS]BACKUP-*.*"))

   => "V:SYS$DISK:[DMR.BACKUP.HACKS]BACKUP-FROB.LSP"

  (NAMESTRING

    (TRANSLATE-PATHNAME "U:/usr/dmr/hacks/frob.l"

                        "U:/usr/d*/hacks/fr*.l"

                        "V:SYS$DISK:[D*.BACKUP.HACKS]BACKUP-*.*"))

   => "V:SYS$DISK:[DMR.BACKUP.HACKS]BACKUP-OB.LSP"


  ;This example presumes background information described in PATHNAME-LOGICAL

  (DEFUN TRANSLATE-LOGICAL-PATHNAME-1 (PATHNAME RULES)

    (LET ((RULE (ASSOC PATHNAME RULES :TEST #'PATHNAME-MATCH-P)))

      (UNLESS RULE (ERROR "No translation rule for ~A" PATHNAME))

      (TRANSLATE-PATHNAME PATHNAME (FIRST RULE) (SECOND RULE))))

  (TRANSLATE-LOGICAL-PATHNAME-1 "FOO:CODE;BASIC.LISP"

                        '(("FOO:DOCUMENTATION;" "MY-UNIX:/doc/foo/")

                          ("FOO:CODE;"          "MY-UNIX:/lib/foo/")

                          ("FOO:PATCHES;*;"     "MY-UNIX:/lib/foo/patch/*/")))

   => the pathname MY-UNIX:/lib/foo/basic.l


Rationale:


  1,2,3. These three functions provide a standardized interface to the

  idiosyncratic wildcard functionality of each host file system.


  1. WILD-PATHNAME-P makes it possible to detect wild pathnames reliably and

  do something useful (give up, merge out the bothersome components, call

  DIRECTORY for a list of matching pathnames, etc.)


  2,3. TRANSLATE-PATHNAME is needed by many application programs that deal

  with wildcard pathnames.  PATHNAME-MATCH-P and TRANSLATE-PATHNAME are

  needed by logical pathnames.  The PATHNAME-LOGICAL proposal cannot be

  implemented without these features.  Implementing PATHNAME-LOGICAL could

  involve adding additional capabilities to TRANSLATE-PATHNAME, depending

  on the type of file system used, but those capabilities do not need to

  be in the standard.


  4. Since these functions return a value connected with one file, there

  is no meaningful way to extend them to work on wildcard pathnames.  It

  seems best to specify that they signal an error, rather than leaving

  the consequences undefined.


  5. The consequences are proposed to be implementation-defined because

  current practice varies and no one wants to change.


Current Practice:


  Presumably no implementation supports the proposal exactly as stated.

  Symbolics Genera has had similar features under different names for many

  years:


    (SEND pathname :WILD-P) returns a value such as NIL, :NAME, :TYPE,

    etc., indicating the first wild field.


    (SEND pathname :NAME-WILD-P), (SEND pathname :DIRECTORY-WILD-P),

    etc. test individual fields.


    The :TRANSLATE-WILD-PATHNAME, :TRANSLATE-WILD-PATHNAME-REVERSIBLE, and

    :PATHNAME-MATCH messages resemble TRANSLATE-PATHNAME and

    PATHNAME-MATCH-P.


  The Explorer also supports the messages :WILD-P (although it only

  returns NIL or T), :NAME-WILD-P, etc., :TRANSLATE-WILD-PATHNAME, and

  :PATHNAME-MATCH.


  Points 4 and 5 are current practice as far as the authors are aware.

  The Explorer permits DELETE-FILE on a wild pathname, meaning to delete

  all files that match.


Cost to Implementors:


  Many implementations probably have a substrate which is capable of this

  or something similar already. In such cases, it's a relatively small

  matter to add the proposed interface.


  Even in cases where an implementation doesn't have ready code, it's clearly

  better for the implementor to write that code once and for all than to ask

  each user of wildcards to write it.


  Since the detailed behavior is at the implementor's discretion, the cost

  is unlikely to be large.  Some file systems will do all the work and the

  implementor need only provide an interface to the file system or to a

  standard library routine.  For other file systems the implementor has to

  write the actual matching and translation algorithms.


Cost to Users:


  None.  This change is upward compatible.


Cost of Non-Adoption:


  Wild pathnames would continue to be mistaken for ordinary pathnames in

  many situations.  User programs that deal with wildcard pathnames would

  have to operate on implementation-dependent representations and hence

  would not be easily portable.


  The biggest cost is that the logical pathnames proposal would be stymied.


Performance Impact:


  None.


Benefits:


  A more complete set of wildcard pathname operations.  Portable user

  programs that deal with wildcard pathnames will be more consistent

  and reliable.  A portable system construction tool can be written

  and the foundations are laid for a `logical pathname' facility

  (proposed separately in PATHNAME-LOGICAL).


Aesthetics:


  This change would make some portable code less kludgey.


Discussion:


  There was some question about the name. The name PATHNAME-WILD-P

  suggests a ``slot'' of a pathname (like PATHNAME-HOST),

  while WILD-PATHNAME-P suggests a type (like INPUT-STREAM-P).

  The committee was split on what to call it. Since it is more

  like a type than a slot, the name WILD-PATHNAME-P was chosen.


  It's been suggested that WILD-PATHNAME-P and PATHNAME-MATCH-P be allowed

  to return a value other than T to represent "truth", which would

  somehow encode some additional information.