Issue MACRO-ENVIRONMENT-EXTENT Writeup

Forum:		Compiler

Issue:		MACRO-ENVIRONMENT-EXTENT

References:	CLtL p. 145-146

		Issue COMPILER-LET-CONFUSION

		Issue MACRO-CACHING

		Issue EVAL-WHEN-NON-TOP-LEVEL

		Issue SYNTACTIC-ENVIRONMENT-ACCESS

		CLOS Chapter 3 (89-003)

Category:	CLARIFICATION,CHANGE

Edit History:   V1, 10 Jan 1989, Sandra Loosemore

		V2, 09 Mar 1989, Sandra Loosemore

		V3, 13 Mar 1989, Sandra Loosemore (last-minute discussion)

Status:		**DRAFT**



Problem Description:


What is the extent of environment objects received as the &ENVIRONMENT

argument of a macro function?


CLtL says that &ENVIRONMENT is "useful primarily in the rare cases

where a macro definition must explicitly expand any macros in a

subform of the macro call before computing its own expansion".  While

this suggests that macro environment objects are typically used within

the dynamic scope of the macro function, the use of the word

"primarily" (rather than "only" or "exclusively" or some similarly

strong language) suggests that there may be other legitimate uses for

environment objects.  But, because CLtL is silent about what those

uses might be, many users and implementors are under the impression

that environment objects have only dynamic extent.


There are some situations where using environment objects as if they

had indefinite extent provides a very useful viewpoint from which to

solve a problem.  Consider the following example:


  (defmacro typed-var (var) var)


  (defmacro local-type-declare (declarations &body forms &environment env)

      `(macrolet ((typed-var (&whole w var)

		    (let ((type  (assoc var ',declarations)))

		      (if type 

		          `(the ,(cadr type) ,var)

                          (macroexpand w ',env)))))

	 ,@forms))


  (defun f (x y)

    (local-type-declare ((x fixnum) (y float))

      (+ (typed-var x) (typed-var y))))


Here, local macro TYPED-VAR is defined to look first in the innermost

lexical environment for information about the variable, and if there

isn't any then it recurses on the next outermost lexical environment.

The global definition of TYPED-VAR provides a terminal case to stop

the recursion.


There are other situations where the extent of macro environment

objects comes into play.  For example, if we allow caching of macro

expansions (issue MACRO-CACHING), environments must have indefinite

extent.  It is unclear whether CLOS would be affected by allowing

macro environments to have only dynamic extent.  (The descriptions of

the CLOS defining macros in document 89-003 seem to imply that the

value of the &ENVIRONMENT argument appears in the expansion of the

macro, but there now seems to be sentiment that the model of how the

defining macros work that is presented there is broken.)



Proposal MACRO-ENVIRONMENT-EXTENT:INDEFINITE:


State that macro environment objects received with the &ENVIRONMENT

argument of a macro function or as the argument to a *MACROEXPAND-HOOK*

function have indefinite extent.


Note that implementations are not permitted to destructively modify

lexical information in environment objects once they have been passed

to a macro function.  It is, however, permissible to add or remove

global definitions that are accessible through the environment.


  Rationale:


  This legitimizes the use of idioms which depend on macro environments

  having indefinite extent.


  Since data objects in Lisp otherwise have indefinite extent, it is

  more consistent to give environment objects indefinite extent as

  well.



Proposal MACRO-ENVIRONMENT-EXTENT:DYNAMIC:


State that macro environment objects received with the &ENVIRONMENT

argument of a macro function or with a *MACROEXPAND-HOOK* function

have only dynamic extent; the consequences are undefined if they are

referred to outside the dynamic extent of that macro function or hook

function.


  Rationale:


  This allows implementations to use somewhat more efficient techniques

  for representing environment objects.  For example, the storage could

  be stack-allocated, or environments could be bashed destructively

  instead of always being freshly heap-allocated.



Proposal MACRO-ENVIRONMENT-EXTENT:DYNAMIC-WITH-COPIER:


State that macro environment objects received with the &ENVIRONMENT

argument of a macro function or with a *MACROEXPAND-HOOK* function

have only dynamic extent; the consequences are undefined if they are

referred to outside the dynamic extent of that macro function or hook

function.


Add a new function:


COPY-ENVIRONMENT environment				[function]


This function returns an environment object that is semantically

equivalent to "environment" (which must be an object of the type

received with an &ENVIRONMENT argument to a macro or as an argument to

a *MACROEXPAND-HOOK* function), but which may safely be referred to

outside the dynamic extent of the macro function.  This function is

permitted to return an object that is EQ to its argument if that 

object may be safely used.


  Rationale:


  This allows implementations to use somewhat more efficient techniques

  for representing environment objects.  For example, the storage could

  be stack-allocated, or environments could be bashed destructively

  instead of always being freshly heap-allocated.


  It also allows programmers to use idioms that rely on environment

  objects having indefinite extent.



Current Practice:


Macro environments appear to have indefinite extent in Lucid Common

Lisp, Kyoto Common Lisp, CMU Common Lisp (at least in the

interpreter), Utah Common Lisp, and A-Lisp.  A-Lisp internally uses

the kind of idiom shown in the example above to implement FLET,

LABELS, and FUNCTION as macros.


Macro environments are stack-allocated in Symbolics Genera, and hence

have dynamic extent.


Macro environments also have dynamic extent on the TI Explorer,

because the compiler uses special variables are used to keep track of

lexical definitions.



Cost to implementors:


For proposal INDEFINITE, some implementations would need to change.  A

simple implementation of macro environments that would fit the

requirements of this proposal is to represent them as lists, pushing

information for inner contours onto the front of the list as the

contour is entered and popping the list when the contour is exited.


For proposal DYNAMIC, there is no associated implementation cost.


For proposal DYNAMIC-WITH-COPIER, the implementation cost is unknown

but probably fairly small in most implementations.



Cost to users:


For proposal INDEFINITE, there is no associated cost to users.


For proposal DYNAMIC, users would not be able to portably use a

simple and elegant approach to solving certain kinds of problems.


For proposal DYNAMIC-WITH-COPIER, users would have to remember to make

a copy of an environment object in some situations.



Benefits:


It is made clear whether treating environment objects as if they had

indefinite extent is portable usage.



Discussion:


Proposal SYNTACTIC-ENVIRONMENT-ACCESS:ADD-FUNCTIONAL-INTERFACE

includes adding a function called AUGMENT-ENVIRONMENT which could also

be used to create a copy of an environment.  However, it returns an

object with the same extent as its argument, and therefore can not

replace the function COPY-ENVIRONMENT under proposal

DYNAMIC-WITH-COPIER.


We have also considered a couple of other alternatives on this 

issue.


One alternative would be to give "remote" environments created by

COMPILE-FILE the extent of that call to COMPILE-FILE, while "local"

environments (the null lexical environment and environments created by

COMPILE and EVAL) would have indefinite extent.


Another alternative would be to say that environments created by

COMPILE-FILE, COMPILE, or EVAL have a dynamic extent that includes the

time when any other macro calls appearing lexically within the

expansion returned by the macro function are expanded.  This is

similar to the extent of the special bindings made by COMPILER-LET.


Both of these proposals could be combined with adding a copier

function to deal with those implementations where environments are

stack-allocated.  They would both solve the extent problem for the

example given in the problem description section, but not the general

problem of macro caching.  In conjunction with the proposals for issue

SYNTACTIC-ENVIRONMENT-ACCESS, they would both require some

modifications to implementations that currently give macro

environments dynamic extent.


Loosemore supports proposal MACRO-ENVIRONMENT-EXTENT:INDEFINITE.


Moon says:

  My opinion is that anything in CLOS that seems to depend on indefinite

  extent for macro environments is broken and needs to be fixed.  It's not

  broken because of the environment extent, but for other reasons.

  Thus I believe in dynamic extent for environments.


Neil Goldman says:

  In my code walker I have a pretty ugly way of dealing with MACROLET

  that would have been trivial if I could have counted on the

  ENVIRONMENT having indefinite extent.  There may be some cleaner way

  than what I did, but I just looked at PCL's code walker, and it also

  is much more complex than would be necessary if environments had

  indefinite extent.