Issue DEFINE-COMPILER-MACRO Writeup

Forum:          Cleanup

Issue:          DEFINE-COMPILER-MACRO

Related Issues: Issue DEFINE-OPTIMIZER

                Issue SYNTACTIC-ENVIRONMENT-ACCESS

                Issue LISP-SYMBOL-REDEFINITION

                Issue DEFINING-MACROS-NON-TOP-LEVEL

                Issue EVAL-WHEN-NON-TOP-LEVEL

References:     CLtL p. 151, p. 152

Category:       ADDITION

Edit-History:   28-Jun-89, Version 1 by JonL White and Steve Haflich

                12-Jul-89, Version 2 by Loosemore

                24-Oct-89, Version 3 by Loosemore

                19-Oct-90, Version 4 by Loosemore

Status:         Version 2 (proposal NEW-FACILITY) passed at June 89 meeting

                Version 4 (proposal X3J13-NOV89) passed at Nov 89 meeting

                    (replaces proposal NEW-FACILITY)



Problem Description:


Occasionally one would like to define a macro which is expanded only

"in the compiler", but which would not normally affect the actions of

the interpreter.  For example, the OSS/Generator proposal has several

functions for which it would like to specify some alternative source

code sequences for the compiler to compiler, rather than just

compiling a closed-call to the function.


Also, it is occasionally desirable for a macro expansion to be

different based on the various compiler optimization qualities (e.g.,

SPEED, SAFETY, and so on); but if the expansion is for the interpreter

rather than the compiler, then such variation based on compiler

optimizers is not needed.


So-called "compiler optimizers" are just a special case of macro-like

expansions, which are limited to being done "in the compiler" and

which are generally required to produce semantically equivalent code

to replace an apparent function call.  There is a need for a facility

that at least covers this capability.





Proposal: (DEFINE-COMPILER-MACRO:X3J13-NOV89):


Add the concept of "compiler macros" to the language, along with the

defining macro DEFINE-COMPILER-MACRO and accessor function

COMPILER-MACRO-FUNCTION.



(1) What compiler macros are


  The purpose of this facility is to permit selective source-code

  transformations as optimization advice to the compiler.  When a

  nonatomic form is being processed (as by the compiler), if the

  operator names a compiler macro then that compiler macro may be

  invoked on the form, and the resulting expansion recursively processed

  in preference to performing the usual processing on the original form

  according to its normal interpretation as a function or macro call.


  A compiler macro function, like an ordinary macro function, is a

  function of two arguments: the entire call form and the environment.

  Unlike an ordinary macro, a compiler macro can decline to provide an

  expansion merely by returning a value that is EQL to the original

  form.  The consequences are undefined if a compiler macro function

  destructively modifies any part of its form argument.


  The form passed to the compiler macro function can either be a list

  whose CAR is the function name, or a list whose CAR is FUNCALL and

  whose CADR is a list (FUNCTION <name>); note that this affects

  destructuring of the form argument by the compiler macro function.

  DEFINE-COMPILER-MACRO arranges for destructuring of arguments to be

  performed correctly for both possible formats.



(2) Naming of compiler macros


  Compiler macros may be defined for function names that name macros as

  well as functions.  It is not permitted to define a compiler macro for

  names which are external symbols in the COMMON-LISP package; see issue

  LISP-SYMBOL-REDEFINITION.


  Compiler macro definitions are strictly global.  There is no provision

  for defining local compiler macros in the way that MACROLET defines

  local macros.  Lexical bindings of a function name shadow any compiler

  macro definition associated with the name as well as its global

  function or macro definition.


  Note that the presence of a compiler macro definition does not affect

  the values returned by FUNCTION-INFORMATION, or other accessors such

  as FBOUNDP or MACROEXPAND.  Compiler macros are global and the function

  COMPILER-MACRO-FUNCTION is sufficient to resolve their interaction

  with other lexical and global definitions.



(3) When compiler macros might/must/must not be used


  The presence of a compiler macro definition for a function or macro

  indicates that it is desirable for the compiler to use the expansion

  of the compiler macro instead of the original function call or macro

  call form.  However, it is not required for any language processor

  (compiler, evaluator, or other code walker) to actually invoke compiler

  macro functions, or to make use of the resulting expansion.


  There two situations in which a compiler macro definition must not be

  applied by any language processor:


      - The global function name binding associated with the compiler

        macro is shadowed by lexical rebinding of the function name.


      - The function name has been declared or proclaimed NOTINLINE and

        the call form appears within the scope of the declaration.


  When a compiler macro function is called as part of processing by the

  evaluator or compiler, it is invoked by calling the function that is

  the value of *MACROEXPAND-HOOK*.


  When COMPILE-FILE chooses to expand a top-level compiler macro call,

  the expansion is also treated as a top-level form for the purposes of

  EVAL-WHEN processing, in the same way as the expansion of an ordinary

  macro.




(4) Specification of DEFINE-COMPILER-MACRO


  DEFINE-COMPILER-MACRO name lambda-list

        [[ {declaration}* | doc-string ]] {form}*                [macro]


  DEFINE-COMPILER-MACRO is the normal mechanism for defining a compiler

  macro function.  Its syntax resembles that of DEFMACRO.  The function

  is defined in the lexical environment in which the DEFINE-COMPILER-MACRO

  form appears; see issue DEFINING-MACROS-NON-TOP-LEVEL.


  The <name> must be a function name.


  The <lambda-list> supports destructuring and may include &ENVIRONMENT

  and &WHOLE, in the same way as a DEFMACRO lambda-list.  The &WHOLE

  argument is bound to the form argument that is passed to the compiler

  macro function.  The remaining lambda-list parameters are specified

  as if this form contained the function name in the CAR and the actual

  arguments in the CDR, but if the CAR of the actual form is the symbol

  FUNCALL, then the destructuring of the the arguments will actually be

  performed using its CDDR instead.


  When a call to DEFINE-COMPILER-MACRO appears at top-level in a file

  being compiled by COMPILE-FILE, the compiler macro definition is made

  known to the file compiler (analagous to the way top-level DEFMACRO

  calls are handled).





(5) Specification of COMPILER-MACRO-FUNCTION


  COMPILER-MACRO-FUNCTION name &optional env                       [function]


  This is the accessor for the compiler macro definition associated

  with a given name.


  The <name> argument must be a function name.  If there is a compiler

  macro definition associated with that name in the given environment

  <env>, COMPILER-MACRO-FUNCTION returns that function; otherwise it

  returns NIL.


  If there is a local function or macro named <name> defined in the

  environment <env>, this definition shadows any global compiler macro

  definition for that <name> and COMPILER-MACRO-FUNCTION must return

  NIL.


  SETF may be used with COMPILER-MACRO-FUNCTION to install a compiler

  macro function for the name <name>, analogously to using SETF on

  MACRO-FUNCTION.  The value must be a function of two arguments, as

  described above.  It is also permissible to provide a value of NIL to

  remove any existing compiler macro definition.  The <env> argument to

  COMPILER-MACRO-FUNCTION must be omitted when it appears as a SETF

  place.





Proposal: (DEFINE-COMPILER-MACRO:NEW-FACILITY):


Introduce a new facility by additions as follows:


Macro:


DEFINE-COMPILER-MACRO name lambda-list {doc-string} {declarations}* {form}*


  This is just like DEFMACRO except the definition isn't stored in the

  symbol function cell of 'name', and isn't seen by MACROEXPAND-1 (but

  is seen by COMPILER-MACROEXPAND-1 -- see below).  Like DEFMACRO, the

  lambdalist may include &ENVIRONMENT and &WHOLE.  The definition is

  "global"; there is no explicit provision for defining local compiler

  macros in the way that MACROLET defines local macros.


  A toplevel call to DEFINE-COMPILER-MACRO in a file being compiled by

  COMPILE-FILE has an effect on the compilation environment similar to

  what a call to DEFMACRO would have, except it is noticed as a

  "compiler macro".



Function:


COMPILER-MACRO-FUNCTION name &OPTIONAL env


  If 'name' is a symbol that has been defined as a compiler macro, then

  calling COMPILER-MACRO-FUNCTION on it returns the macro expansion

  function; otherwise it returns NIL.  'name' must be a symbol.  The

  local lexical environment may override a global definition for 'name'

  by defining a local function or local macro (such as by FLET,

  MACROLET, etc.), in which case NIL is returned; the optional argument

  'env' is provided so that clients may pass in &environment objects for

  this purpose.


  SETF may be used with COMPILER-MACRO-FUNCTION to install a function as

  the expansion function for the compiler macro 'name', analogously to

  using SETF on MACRO-FUNCTION.  SETF'ing to NIL removes any existing

  compiler macro definition.  Like MACRO-FUNCTION, the SETF value (if not

  NIL) must be a function of two arguments:  the entire macro call, and

  the environment.  The second argument to COMPILER-MACRO-FUNCTION must

  be omitted when it is SETFed.



Functions:


COMPILER-MACROEXPAND form &OPTIONAL env

COMPILER-MACROEXPAND-1 form &OPTIONAL env


  This is just like MACROEXPAND and MACROEXPAND-1 (see CLtL p.151)

  except that the expander function is obtained as if by a call to

  COMPILER-MACRO-FUNCTION on the CAR of 'form' rather than by a call to

  MACRO-FUNCTION.  There are three cases wherein no expansion happens:


    (1) There is no compiler macro definition for the CAR of 'form';

    (2) There is such a definition but there is also a NOTINLINE

        declaration, either globally or in the lexical environment 'env';

    (3) A global compiler macro definition is shadowed by a local

        function or macro definition (such as by FLET, LABELS, or MACROLET).


  Note that if there is no expansion, the original form is returned as

  the first value, and NIL as the second value.


  When COMPILER-MACROEXPAND-1 discovers that there is to be an expansion

  it does it by calling the function in *MACROEXPAND-HOOK* (see CltL p.152).



The purpose of this facility is to permit selective source-code

transformations based on whether the compiler is processing the code.

When the compiler is about to compile a nonatomic form, it first calls

COMPILER-MACROEXPAND-1 repeatedly until there is no more expansion

(there might not be any to begin with).  Then it continues its

remaining processing, which may include calling MACROEXPAND-1 etc.


The compiler is required to expand compiler macros; it is unspecified

whether the interpreter does so.  The intention is that only the

compiler will do so, but the range of possible "compiled-only"

implementation strategies precludes any firm specification.


Note that a compiler macro may decline to provide any expansion merely

by returning the original form; this is useful when using the facility

to put "compiler optimizers" on various function names.  For example,

here is a compiler macro that "optimizes" the 0- and 1-argument cases of

a function called PLUS:


    (define-compiler-macro plus (&whole form &rest args)

        (case (length args)

            (0 0)

            (1 (car args))

            (t form)))


The issue LISP-SYMBOL-REDEFINITION precludes user definition of any

compiler macros for symbols external in the Lisp package that have a

definition as a function, macro, or special form.


Note that compiler macros do not appear in information returned by

FUNCTION-INFORMATION; they are global, and their interaction

with other lexical and global definitions can be reconstructed by

COMPILER-MACRO-FUNCTION.  It is up to code-walking programs to decide

whether to invoke compiler macro expansion.



  Rationale:


  Many implementations have it.  Many users have requested a way to add

  source-code "optimizers" to the compiler.


  Other than INLINE declarations for functions there is no other way to

  customize how calls to a specific function are compiled.  DEFMACRO is

  not usable for this purpose since it requires use of the

  symbol-function cell, which would prevent the functional definition

  from being active in the compilation environment.




Current Practice:


Lucid, Franz, and Symbolics have very similar facilities.  Hunoz about

the others?



Cost to Implementors:


Minor: implement a method for storing named expansion functions, and

tweak the compiler in one or two places.



Cost to Users:


None.  This is an upward-compatible addition.



Benefits:


Increased portability for clients of the existing facilities.



Discussion:


There has been extensive discussion under the issue DEFINE-OPTIMIZER.