CLHS: Issue EVAL-WHEN-NON-TOP-LEVEL Writeup

Issue EVAL-WHEN-NON-TOP-LEVEL Writeup

Issue: EVAL-WHEN-NON-TOP-LEVEL

Forum: Compiler

References: EVAL-WHEN (CLtL pp69-70),

Issue DEFINING-MACROS-NON-TOP-LEVEL

Issue COMPILED-FUNCTION-REQUIREMENTS

Issue IN-PACKAGE-FUNCTIONALITY

Issue LOCALLY-TOP-LEVEL

Category: CLARIFICATION/CHANGE

Edit History: 06-May-88, Version 1 by Sandra Loosemore

16-Dec-88, Version 2 by Loosemore (alternate direction)

30-Dec-88, Version 3 by Loosemore (minor wording changes)

07-Jan-89, Version 4 by Loosemore (update discussion)

09-Feb-89, Version 5 by Pitman and Moon (some major changes)

09-Mar-89, Version 6 by Loosemore (clean up wording)

22-Mar-89, Version 7 by Loosemore (order of processing)

Status: Ready for release

Problem Description:

The current description of how the compiler should handle EVAL-WHEN

only makes sense when it appears as a top-level form in the file being

compiled. Is it legitimate for EVAL-WHEN to appear in non-top-level

locations? Even if it is legitimate, what does it mean?

Another issue, referred to here as ``the EVAL-WHEN shadowing problem,''

is that some people have complained that shadowing the symbols EVAL,

COMPILE, or LOAD means that you have to also either shadow EVAL-WHEN

and define it to recognize the new symbol, or else you must resign

yourself to writing (EVAL-WHEN (... LISP:EVAL ...) ...),etc. all over.

While the goal here is not to solve this problem, it might be possible

to solve both problems at once.

There are two proposals presented here, GENERALIZE-EVAL and

GENERALIZE-EVAL-NEW-KEYWORDS.

Background/Analysis:

The proposal which follows was constructed with the following goals

in mind:

1. The lexical and dynamic environment for the EVAL-WHEN body should

be the same for each situation. That is, the body should ``mean

the same thing'' regardless of which situation is being processed.

2. The evaluation context for EVAL-WHEN should be the current

lexical environment.

3. At execution time, EVAL-WHEN should always return the result of

its last form if execution of the body occurred, or NIL if the

body was not executed.

4. If a top-level EVAL-WHEN has a LOAD keyword, its body should

inherit top-level-ness during normal processing. This permits the

use of (EVAL-WHEN (EVAL COMPILE LOAD) ...) at top-level to mean

simply "Do whatever would normally be done for this body, but

also do something at compile time." This, in turn, will later be

the key to allowing defining forms to be usefully described in

terms of EVAL-WHEN.

5. Non-top-level expressions should have no effect until they are

executed. This is the key to making sure that any necessary

environment is present. Since the COMPILE keyword forces effects

to occur earlier than execution time, it follows from this that

any correct solution must not allow the COMPILE keyword to have

an effect at other than top-level.

To accomplish these goals, we formulated the following model:

The purpose of EVAL-WHEN is to accomodate the fact that some of the

semantic processing of an expression may usefully be partitioned

between compile time and run time in some circumstances.

(EVAL-WHEN (EVAL) <code>)

describes a general technique for accomplishing some particular goal

at normal program execution time. However, the pair of expressions

(EVAL-WHEN (COMPILE) <code-A>)

(EVAL-WHEN (LOAD) <code-B>)

can be used to describe an alternate technique for implementing part

of the effect (A) at compile-time, and part of the effect (B) at

load-time.

Proposal (EVAL-WHEN-NON-TOP-LEVEL:GENERALIZE-EVAL):

Replace the description of EVAL-WHEN with the following:

EVAL-WHEN ({situation}*) {form}* [Special Form]

The body of an EVAL-WHEN form is processed as an implicit PROGN, but

only in the situations listed. Each SITUATION must be a symbol,

either COMPILE, LOAD, or EVAL.

The use of COMPILE and LOAD controls whether and when processing

occurs for top-level forms. The use of EVAL controls whether

processing occurs for non-top-level forms.

The EVAL-WHEN construct may be more precisely understood in terms of

a model of how the file compiler, COMPILE-FILE, processes forms in a

file to be compiled.

Successive forms are read from the file by the file compiler using

READ. These top-level forms are normally processed in what we call

`not-compile-time' mode. There is one other mode, called

`compile-time-too' mode, which can come into play for top-level

forms. The EVAL-WHEN special form is used to annotate a program

in a way that allows the program doing the processing to select

the appropriate mode.

Processing of top-level forms in the file compiler works as follows:

* If the form is a macro call, it is expanded and the result is

processed as a top-level form in the same processing mode

(compile-time-too or not-compile-time).

* If the form is a PROGN form, each of its body forms is

sequentially processed as top-level forms in the same processing

mode.

* If the form is a COMPILER-LET, MACROLET, or SYMBOL-MACROLET,

the file compiler makes the appropriate bindings and recursively

processes the body forms as an implicit top-level PROGN with those

bindings in effect, in the same processing mode.

* If the form is an EVAL-WHEN form, it is handled according to

the following table:

COMPILE LOAD EVAL compile-time-too Action

Yes Yes -- -- Process body in compile-time-too mode

No Yes Yes Yes Process body in compile-time-too mode

No Yes Yes No Process body in not-compile-time mode

No Yes No -- Process body in not-compile-time mode

Yes No -- -- Evaluate body

No No Yes Yes Evaluate body

No No Yes No do nothing

No No No -- do nothing

"Process body" means to process the body as an implicit top-level

PROGN. "Evaluate body" means to evaluate the body forms as in

implicit PROGN in the dynamic execution context of the compiler and

in the lexical environment in which the EVAL-WHEN appears.

* Otherwise, the form is a top-level form that is not one of the

special cases. If in compile-time-too mode, the compiler first

evaluates the form and then performs normal compiler processing

on it. If in not-compile-time mode, only normal compiler

processing is performed. [The nature of this processing is

defined more precisely in issue COMPILED-FUNCTION-REQUIREMENTS.]

Any subforms are treated as non-top-level forms.

Note that top-level forms are guaranteed to be processed in the order

in which they textually appear in the file, and that each top-level

form read by the compiler is processed before the next is read.

However, the order of processing (including, in particular, macro

expansion) of subforms that are not top-level forms is unspecified.

For an EVAL-WHEN form that is not a top-level form in the file compiler

(that is, one of: in the interpreter; in COMPILE; or in the file

compiler but not at top-level), if the EVAL situation is specified,

its body is treated as an implicit PROGN. Otherwise, the EVAL-WHEN

form returns NIL.

Clarifications/Consequences:

The following effects are logical consequences of the above proposal:

* It is never the case that the execution of a single EVAL-WHEN

expression will execute the body code more than once.

* The keyword `EVAL' is a misnomer because execution of

the body need not be done by EVAL. In compiled code, such as

(DEFUN FOO () (EVAL-WHEN (EVAL) (PRINT 'FOO)))

the call to PRINT should be compiled.

* Macros intended for use in top-level forms should arrange for all

side-effects to be done by the forms in the macro expansion.

The macro-expander itself should not do the side-effects.

Wrong: (defmacro foo ()

(really-foo)

`(really-foo))

Right: (defmacro foo ()

`(eval-when (compile eval load) (really-foo)))

Adherence to this convention will mean that such macros will behave

intuitively when placed in non-top-level positions.

* Placing a variable binding around an EVAL-WHEN reliably captures the

binding because the `compile-time-too' mode cannot occur (because

introducing a variable binding would mean we were not at top level).

For example,

(LET ((X 3))

(EVAL-WHEN (EVAL LOAD COMPILE) (PRINT X)))

will print 3 at execution [load] time, and will not print anything at

compile time. This is important so that expansions of DEFUN and

DEFMACRO can be done in terms of EVAL-WHEN and can correctly capture

the lexical environment.

(DEFUN BAR (X) (DEFUN FOO () (+ X 3)))

might expand into

(DEFUN BAR (X)

(PROGN (EVAL-WHEN (COMPILE)

(COMPILER::NOTICE-FUNCTION-DEFINITION 'FOO '(X)))

(EVAL-WHEN (EVAL LOAD)

(SETF (SYMBOL-FUNCTION 'FOO) #'(LAMBDA () (+ X 3))))))

which would be treated the same as

(DEFUN BAR (X)

(SETF (SYMBOL-FUNCTION 'FOO) #'(LAMBDA () (+ X 3))))

by the above rules.

Test Cases:

;; #1: The EVAL-WHEN in this case is not at top-level, so only the EVAL

;; keyword is considered. At compile time, this has no effect.

;; At load time (if the LET is at top level), or at execution time

;; (if the LET is embedded in some other form which does not execute

;; until later) this sets (SYMBOL-FUNCTION 'FOO1) to a function which

;; returns 1.

(LET ((X 1))

(EVAL-WHEN (EVAL LOAD COMPILE)

(SETF (SYMBOL-FUNCTION 'FOO1) #'(LAMBDA () X))))

;; #2: If this expression occurs at the top-level of a file to be compiled,

;; it has BOTH a compile time AND a load-time effect of setting

;; (SYMBOL-FUNCTION 'FOO2) to a function which returns 2.

(EVAL-WHEN (EVAL LOAD COMPILE)

(LET ((X 2))

(EVAL-WHEN (EVAL LOAD COMPILE)

(SETF (SYMBOL-FUNCTION 'FOO2) #'(LAMBDA () X)))))

;; #3: If this expression occurs at the top-level of a file to be compiled,

;; it has BOTH a compile time AND a load-time effect of setting the

;; function cell of FOO3 to a function which returns 3.

(EVAL-WHEN (EVAL LOAD COMPILE)

(SETF (SYMBOL-FUNCTION 'FOO3) #'(LAMBDA () 3)))

;; #4: This always does nothing. It simply returns NIL.

(EVAL-WHEN (COMPILE)

(PRINT 'FOO4)))

;; #5: If this form occurs at top-level of a file to be compiled, FOO5 is

;; printed at compile time. If this form occurs in a non-top-level

;; position, nothing is printed at compile time. Regardless of context,

;; nothing is ever printed at load time or execution time.

(EVAL-WHEN (COMPILE)

(EVAL-WHEN (EVAL)

(PRINT 'FOO5)))

;; #6: If this form occurs at top-level of a file to be compiled, FOO6 is

;; printed at compile time. If this form occurs in a non-top-level

;; position, nothing is printed at compile time. Regardless of context,

;; nothing is ever printed at load time or execution time.

(EVAL-WHEN (EVAL LOAD)

(EVAL-WHEN (COMPILE)

(PRINT 'FOO6)))

Rationale:

This is compatible with any guarantees made by CLtL, and extends the

behavior usefully to non-top-level situations.

This gives a useful meaning to EVAL-WHEN that supports useful and

predictable behavior if defining macros are used in a non-top-level

situation.

The constraints on the order in which top-level forms are processed

ensure that the compile-time effects of defining macros and EVAL-WHENs

at the beginning of the file are visible during the processing of

forms that appear later in the file, which is what most users expect.

Leaving the order of processing of non-top-level forms unspecified

allows the compiler to perform certain kinds of transformations that

change the textual order of subforms. Users should not depend on

side-effects from macros that require them to be expanded in any

particular order.

Proposal (EVAL-WHEN-NON-TOP-LEVEL:GENERALIZE-EVAL-NEW-KEYWORDS):

As in GENERALIZE-EVAL, but rename the EVAL keyword to :EXECUTE,

the COMPILE keyword to :COMPILE-TOPLEVEL, and LOAD keyword to

:LOAD-TOPLEVEL.

Deprecate the use of keywords EVAL, COMPILE, and LOAD to EVAL-WHEN.

For compatibility, they are supported in EVAL-WHEN

at top-level, but their meaning is not defined elsewhere.

Rationale:

The fact that the situation keywords chosen are not the same as

those now used means that the change can be added in a way that

is truly upward compatible (not only with CLtL but with existing

practice in implementations which have chosen to extend or `clarify'

the definition given in CLtL) since the meaning of EVAL, COMPILE,

and LOAD in non-top-level situations (which was never spelled

out in CLtL) can legitimately differ from the meaning of these

new keywords.

Using other names and/or the keyword package for the names of

situations solves the EVAL-WHEN shadowing problem.

The name `execute' does not promote the confusion that the body of an

EVAL-WHEN must be executed only in the evaluator. It also does not

promote the confusion that the body of an EVAL-WHEN, regardless of when

executed, must run interpreted.

The names `compile-toplevel' and `load-toplevel' emphasize the fact

that these cases are not interesting in non-top-level positions.

Current Practice:

In Symbolics Genera, the interpreter permits EVAL-WHEN in non-top-level

positions in a way that is compatible with this proposal but both the

COMPILE and COMPILE-FILE functions complain about EVAL-WHEN in a

non-top-level position.

Both Lucid Common Lisp and Kyoto Common Lisp already interpret the

EVAL keyword to mean "execute" in non-top-level situations. Both of

these implementations also make (EVAL-WHEN (LOAD) ...) suppress

compile-time "magic" from defining macros such as DEFMACRO.

IIM describes its EVAL-WHEN as:

(defmacro eval-when (situations &body body &environment env)

(if (not (compiler-environment-p env))

(when (member 'eval situations) `(progn ,@body))

(progn

(when (member 'compile situations)

(if (compiler-at-top-level-p env)

(mapc #'eval body)

(warn "Top-level form encountered at non-top-level.")))

(when (member 'load situations) `(progn ,@body)))))

Note that the interpretation of the EVAL situation and the nesting

behavior is different.

Cost to Implementors:

The actual change to EVAL-WHEN in both cases is probably fairly

localized and straightforward to make in most or all implementations.

The second-order costs of proposal GENERALIZE-EVAL will vary depending

on whether existing implementations have extended the definition of

EVAL-WHEN in incompatible ways. If an implementation has made such

extensions, there may be user and system code which depends on them

and the cost of converting that code may be non-trivial. There is

presumably also documentation impact.

Proposal GENERALIZE-EVAL-NEW-KEYWORDS avoids most or all of the

second-order costs of proposal GENERALIZE-EVAL.

The compiler processing for top-level forms might be implemented

something like:

;;; Forms read by the file compiler are passed to PROCESS-TOP-LEVEL-FORM

;;; with a env compile-time-too both NIL.

(defun process-top-level-form (form env compile-time-too)

(setq form (macroexpand form env))

(cond ((not (consp form))

nil)

((eq (car form) 'progn)

(dolist (f (cdr form))

(process-top-level-form f env compile-time-too)))

((eq (car form) 'compiler-let)

(process-compiler-let form env compile-time-too))

((eq (car form) 'macrolet)

(process-macrolet form env compile-time-too))

((eq (car form) 'symbol-macrolet)

(process-symbol-macrolet form env compile-time-too))

((eq (car form) 'eval-when)

(process-eval-when form env compile-time-too))

(if compile-time-too

(internal-eval form env))

(compile-form form env))

))

(defun process-eval-when (form env compile-time-too)

(let* ((situations (cadr form))

(body (cddr form))

(compile-p (member 'compile situations))

(load-p (member 'load situations))

(eval-p (member 'eval situations)))

(cond ((or (and compile-p load-p)

(and eval-p load-p compile-time-too))

(process-top-level-form `(progn ,@body) env t))

(load-p

(process-top-level-form `(progn ,@body) env nil))

((or compile-p

(and eval-p compile-time-too))

(dolist (f body)

(internal-eval f env)))

nil))))

;;; PROCESS-COMPILER-LET, PROCESS-MACROLET, and PROCESS-SYMBOL-MACROLET

;;; do the obvious things.

;;; INTERNAL-EVAL evaluates "form" in lexical environment "env".

Cost to Users:

Technically, none. Either proposal is technically upward compatible

with CLtL.

Proposal GENERALIZE-EVAL might force some extended implementations to

change incompatibly. As such, some users who depend on

implementation-dependent extensions might have to adjust their code

somewhat to deal with those changes.

Proposal GENERALIZE-EVAL-NEW-KEYWORDS does not force implementations

to change incompatibly, so has no forced impact on users.

Cost of Non-Adoption:

EVAL-WHEN is a mess. Using it as the low-level substrate into which

defining macros should expand, and guaranteeing any predictable effects

of those macros in non-top-level situations is currently difficult and

would continue to be so in the absence of some resolution on this issue.

Benefits:

The costs of non-adoption would be avoided: it would be possible to

use EVAL-WHEN in many situations where it cannot currently be used

reliably.

The portability of many existing tools which use EVAL-WHEN internally

in macros will be enhanced.

Aesthetics:

This generalization of the meaning makes the purpose and uses of

EVAL-WHEN less mysterious. In that sense, aesthetics are simplified

somewhat.

Discussion:

The cleanup issue LOCALLY-TOP-LEVEL would make LOCALLY also "pass

through" top-level-ness to its body. The reason why that is not

addressed in this issue is that it involves making LOCALLY a special

form.

Pitman and Moon don't care whether we say `top level,' `top-level,' or

`toplevel.' The spelling choices in this writeup are arbitrary. If

necessary, the proposal GENERALIZE-EVAL-NEW-KEYWORDS could be amended

to propose :COMPILE-TOP-LEVEL, etc.

Pitman, Moon, and Bob Laddaga (a Symbolics Cloe implementor) support

both of these proposals. Pitman and Laddaga have a preference for

GENERALIZE-EVAL-NEW-KEYWORDS. Moon is neutral about which should be

preferred.

Sandra Loosemore says:

I still feel somewhat uncomfortable with the definition of EVAL-WHEN

presented here, mostly because its nesting behavior is so unintuitive

(as in test case number 6). We have also had a hard time in deciding

what the term "top-level" really means; the definition presented here

is rather arbitrary. However, since we have run out of time in which

to come up with acceptable alternatives, I'm willing to go along with

proposal GENERALIZE-EVAL. It is compatible with the description in

CLtL but presented in a more coherent way, and I think it is an

improvement. On the other hand, I don't really like the idea of

changing the names of the keywords; if we are going to make an

incompatible change, the right thing to do would be to throw out

EVAL-WHEN entirely and start from scratch.

Treating MACROLET and SYMBOL-MACROLET (and possibly LOCALLY)

complicates the treatment of top-level DEFMACROs and other

defining macros that cause functions to be created at compile-time

(because the lexical environment the functions are defined in may

not be null). See issue DEFINING-MACROS-NON-TOP-LEVEL for details.-------