FIRST tokens is determined. The FIRST set of
a nonterminal defines all terminal tokens that can be encountered when
beginning to recognize that nonterminal.
FIRST set, the grammar itself is
analyzed. Starting from the start rule all possible syntactically correct
derivations of the grammar are determined.
parse() function)
processes input according to the tables generated by the parser generator.
All the above phases are illustrated and discussed in the next sections. Additional details of the parsing process can be found in various books about compiler construction, e.g., in Aho, Sethi and Ullman's (2003) book Compilers (Addison-Wesley).
In the sections below, the following grammar is used to illustrate the various phases:
%token NR
%left '+'
%%
start:
start expr
|
// empty
;
expr:
NR
|
expr '+' expr
;
The grammar is interesting as it has a rule containing an empty
alternative and because it harbors a shift-reduce conflict. The shift-reduce
conflict is solved by explictly assigning a priority and association to the
'+' token.
The analysis starts by defining an additional rule, which is recognized
(reduced) at end of input. This rule and the rules specified in the grammar
together define what is known as the augmented grammar. In the coming
sections the symbol $ is used to indicate `end of input'. From the above
grammar the following augmented grammar is derived:
1. start: start expr
2. start: // empty
3. expr: NR
4. expr: expr '+' expr
5. start_$: start (input ends here)
bisonc++ itself produces an extensive analysis of any grammar it is offered
when the option --construction is provided.
FIRST set defines all terminal tokens that can be encountered when
beginning to recognize a grammatical symbol. For each grammatical symbol
(terminal and nonterminal) a FIRST set can be determined as follows:
FIRST set of a terminal symbol is the symbol itself.
FIRST set of an empty alternative is the empty set. The empty
set is indicated by ε
and is considered an actual element of the
FIRST set (So, a FIRST set could contain two elements:
'+' and ε
).
X: X1 X2 X3..., Xi, ...Xn, then
initialize FIRST(X) to empty (i.e., not even holding ε
). Then,
for each Xi (1..n):
FIRST(Xi) to FIRST(X)
FIRST(Xi) does not contain ε
FIRST(Xn) does not contain ε
remove ε
from FIRST(X) (unless
analyzing another production rule) ε
is already part of FIRST(X).
When starting this algorithm, only the nonterminals need to be
considered. Also, required FIRST sets may not yet be available. Therefore
the above algorithm iterates over all nonterminals until no changes were
observed. In the algorithm $ is not considered.
Applying the above algorithm to the rules of our grammar we get:
| nonterminal | rule | FIRST set | |
start_$ | start | not yet available | |
start | start expr | not yet available | |
start | // empty | ε | |
expr | NR | NR |
|
expr | expr '+' expr | NR |
|
| changes in the next cycle: | |||
start | start expr | NR ε
|
|
start | // empty | NR ε
|
|
| changes in the next cycle: | |||
start_$ | start | NR ε
|
|
| no further changes | |||
FIRST set, bisonc++ determines the
states of the grammar. The analysis starts at the augmented grammar rule
and proceeds until all possible states have been determined.
In this analysis the concept of the dot symbol is used. The dot shows
the position we are at when analyzing production rules defined by a grammar.
Using the provided example grammar the analysis proceeds as follows:
start_$ -> . startThe above kernel item results in the addition of the following non-kernel items:
start -> . start expr
start -> .
From each of the items new states may be derived. New states are reached when the symbol to the right of the dot has been recognized. In that case a transition (a goto) to the next state takes place, where the dot has moved one postition to the right, defining a kernel item of the new state. Once the dot has reached the end of the rule, a reduction may take place. Following a reduction a transition based on rule's LHS is performed. This procedure is discussed in more detail in section 7.0.5.
Looking at the current state's items, two actions are possible:
start, to a state in which start has been seen (state 1)
start -> .
start_$ -> start .
start -> start . expr
expr is a nonterminal to the right of the dot, we add
all expr rules as this state's non-kernel items:
expr -> . NR
expr -> . expr '+' expr
start_$ rule has been recognized, and so the input was
syntactically correct. But in this state transitions to other states are also
possible:
expr to state 2
start -> start expr .
expr -> expr . '+' expr
'+' to state 4
start according to its first item (removing
two elements from the parser's stack).
expr -> NR .
expr
(removing one element from the parser's stack).
expr -> expr '+' . expr
expr is a nonterminal to the right of the dot, we add
all expr rules as this state's non-kernel items:
expr -> . NR
expr -> . expr '+' expr
expr to state 5
NR we reach the situation expr -> NR . which has
already been encountered at state 3. That's OK, so on NR there is
a transition to state 3.
expr -> NR . kernel item.
expr -> expr '+' expr .
expr -> expr . '+' expr
'+' to state 4
expr according to its first item (removing three
elements from the parser's stack).
expr -> expr '+' expr .
expr -> expr . '+' expr
%left '+'
specification, which is explained in this and the next section.
When analyzing a grammar all states that can be reached from the augmented
start rule are determined. In the current grammar's fifth state bisonc++ must
decide which action to take: should it shift on '+' or should it reduce
according to the item `expr -> expr '+' expr .'? What choice will bisonc++
make?
Here the fact that bisonc++ implements a parser for a Look Ahead Left to Right (1) (LALR(1)) grammar becomes relevant. Bisonc++ computes look-ahead sets to determine which alternative to select when confronted with a choice. The look-ahead set can be used to favor one action over another when generating tables for the parsing function.
Sometimes the look-ahead sets allow bisonc++ simply to remove one action from the
set of possible actions. When bisonc++ is called to process the example grammar
while specifying the --construction option state five only shows the
reduction and not the shifting action, as bisonc++ has removed that latter
action from the action set. In state five the choice is between shifting a
'+' token on the stack, or reducing the stack according to the rule
expr -> expr '+' expr
Here, as we will shortly see, the '+' is also an element of the
look-ahead set of the reducible item, creating a conflict: what to do
on '+'?
In this case the grammar designer has provided bisonc++ with a way out: the
%left directive tells bisonc++ to favor a reduction over a shift, and so it
removed expr -> expr . '+' expr from its set of actions in state five.
Whenever a token is read, it is not immediately shifted; first it becomes the look-ahead token, which is not yet shifted on the stack. This allows the parser to perform one or more reductions, with the look-ahead token still waiting to be processed. Only when all available reductions have been performed the look-ahead token is shifted on the stack. The phrase `all available reductions' does not necessarily mean all possible reductions. Depending on the look-ahead token, a shift rather than a reduce may be performed in states in which both actions are possible.
Here is a simple case where a look-ahead token is required. The production
rules define expressions which may contain binary addition operators and
postfix unary factorial operators (`!'), as well as parentheses for
grouping expressions:
expr:
term '+' expr
|
term
;
term:
'(' expr ')'
|
term '!'
|
NUMBER
;
Suppose that the tokens `1 + 2' have been read and shifted; what
should be done? If the following token is `)', then the first three
tokens must be reduced, forming an expr. This is the only valid course,
because shifting the `)' would produce the sequence of symbols
term 'CLOSEPAR'
which is not syntactically correct.
But if the next token is `!', then that token must be shifted so that
`2 !' can be reduced to recognize a term. If in this case the parser
would perform a reduction then `1 + 2' would become an expr. In that
case the `!' can't be shifted because doing so would result in the
sequence
expr '!'
which is also syntactically incorrect.
S_$: . S, where S is the grammar's start rule) the LA sets of all
items of all of the grammar's states are determined. By definition, the LA set
of state 0's kernel item equals $, representing end-of-file.
Starting from the function State::determineLAsets, which is called for
state 0, the LA sets of all items of all states are computed.
For each state, the LA sets of its items are computed first. Once they have
been computed, the LA sets of items from where transitions to other states are
possible are then propagated to the matching kernel items of those destination
states. When the LA sets of kernel items of those destination states are
enlarged then their state indices are added to a set todo. LA sets of the
items of states whose indices are stored in the todo set are (re)computed
(by calling determineLAsets for those states) until todo is empty, at
which point all LA sets have been computed. Initially todo only contains
0, the index of the initial state, representing the augmented grammar's
production rule.
To compute the LA sets of a state's items the LA set of each of its kernel
items is distributed (by the member State::distributeLAsetOf) over the
items which are implied by the item being considered. E.g., for item X: a
. Y z, where a and z are any sequence of grammar symbols and X
and Y are nonterminal symbols, all of Y's production rules are added
as new items to the current state.
Then the member distributeLAsetOfItem(idx) matches the item's rule
specification with the specification a.Bc, where a and c are
(possibly empty) sequences of grammatical symbols, and B is a (possibly
empty) nonterminal symbol appearing immediately to the right of the item's
dot position. if B is empty then there are no additional production rules
and distributeLAsetOf may return. Otherwise, the set b = FIRST(c) is
computed. This set holds all symbols which may follow B. If b contains
ε
(i.e., the element representing the empty set), then the currently
defined LA set of the item can also be observed. In that case ε
is
removed, and the currently defined LA set is added to b. Finally, the LA
sets of all items representing a production rule for B are inspected: if
b contains unique elements compared to the LA sets of those items, then
the unique elements of b are added to the LA sets of those items. Finally,
distributeLAsetOfItem is recursively called for those items whose LA sets
were enlarged.
Once the LA sets of the items of a state have thus been computed,
inspectTransitions is called to propagate the LA sets of items from where
transitions to other states are possible to the affected (kernel) items of
those other (destination) states. The member inspectTransitions inspects
all Next objects of the current state's d_nextVector. Next objects
provide
To illustrate an LA-set computation we will now compute the LA sets of (some of) the items of the states of the grammar introduced at the beginning of this chapter. Its augmented grammar consists of the following production rules:
1. start: start expr
2. start: // empty
3. expr: NR
4. expr: expr '+' expr
5. start_$: start
When analyzing this grammar, we found the following five states, consisting of
several items and transitions (kernel items are marked with K following their
item indices). Next to the items, where applicable, the goto-table is shown:
the state to go to when the mentioned grammatical symbol has been recognized:
Goto table
-----------
State 0: start
0K: start_$ -> . start 1
1: start -> . start expr 1
2: start -> .
State 1: expr NR
0K: start_$ -> start .
1K: start -> start . expr 2
2: expr -> . NR 3
3: expr -> . expr '+' expr
State 2: '+'
0K: start -> start expr .
1K: expr -> expr . '+' expr 4
State 3:
0K: expr -> NR .
State 4: expr NR
0K: expr -> expr '+' . expr 5
1: expr -> . NR 3
2: expr -> . expr '+' expr 5
State 5: '+'
0K: expr -> expr '+' expr .
1K: expr -> expr . '+' expr 4
Item 0 of state 0 by definition has LA symbol $, and LA computation therefore
always starts at item 0 of state 0. The interesting part of the LA set
computation is encountered in the recursive member distributeLAsets:
distributeLAsetsOfItem(0)
start_$ -> . start: LA: {$}, B: start, c: {}, so b: {$}
items 1 and 2 refer to production rules of B (start) and are inspected:
1: LA(1): {}: b contains unique elements. Therefore:
LA(1) = {$}
distributeLAsetsOfItem(1):
start -> . start expr: LA: {$}, B: start, c: {expr}, so b: {NR}
inspect items 1 and 2 as they refer to production rules of B (start):
1: LA(1): {}: b contains unique elements. Therefore:
LA(1) = {$,NR}
distributeLAsetsOfItem(1)
start -> . start expr: LA: {$,NR}, B: start, c: {expr}, so b: {NR}
inspect items 1 and 2 as they refer to prod. rules of B (start):
1: LA(1): {$,NR}, so b does not contain unique elements: done
2: LA(2): {}, b contains unique elements
LA(2) = {NR}
distributeLAsetsOfItem(2)
start -> .: LA: {NR}, B: -, c: {}, so b: {NR}
inspect items 1 and 2 as they refer to prod. rules of B (start):
1: LA(1): {$,NR}, b does not contain unique elements: done
2: LA(2): {NR}, so b does not contain unique elements: done
2: LA(2): {NR}, so b does not contain unique elements: done
2: LA(2): {NR}: b contains unique elements. Therefore:
LA(2) = {$,NR}
distributeLAsetsOfItem(2)
start -> .: LA: {$,NR}, B: -, c: {}
B empty, so return.
So, item 0 has LA set {$}, items 1 and 2 have LA sets {$,NR}.
The next step involves propagating the LA sets to kernel items of the states to where transitions are possible:
{$}, and 1 (state 1's
index) is inserted into the todo set.
{$,NR}, and 1 (state 1's
index) is inserted into the todo set.
Following this LA set propagation the LA sets of all items of state 1 are computed, which in turn is followed by LA propagation to other states (states 2 and 3), etc. etc.
In this grammar there are no transitions to the current state (i.e.,
transitions from state x to state x). If such transitions are encountered then
they can be ignored by inspectTransitions as the LA sets of the items of a
state have already be computed by the time inspectTransitions is called.
parse() is implemented using a finite-state
machine. The values pushed on the parser stack are not simply token type
codes; they represent the entire sequence of terminal and nonterminal symbols
at or near the top of the stack. The current state collects all the
information about previous input which is relevant to deciding what to do
next.
Each time a look-ahead token is read, the current parser state together with the current (not yet processed) token are looked up in a table. This table entry can say Shift the token. This also specifies a new parser state, which is then pushed onto the top of the parser stack. Or it can say Reduce using rule number n. This means that a certain number of tokens or nonterminals are removed from the stack, and that the rule's nonterminal becomes the `next token' to be considered. That `next token' is then used in combination with the state then at the stack's top, to determine the next state to consider. This (next) state is then again pushed on the stack, and a new token is requested from the lexical scanner, and the process repeats itself.
There are two special situations the parsing algorithm must consider:
parse() returns the value 0, indicating a successful
parsing.
Once bisonc++ has successfully analyzed the grammar it generates the tables that are used by the parsing function to parse input according to the provided grammar. Each state results in a state transition table. For the example grammar used so far there are five states. Each table consists of rows having two elements. The meaning of the elements depends on their position in the table.
| NORMAL | Despite its name, it's not used |
| ERR_ITEM | The state allows error recovery |
| REQ_TOKEN | The state requires a token (which may already be available) |
| ERR_REQ | combines ERR_ITEM and REQ_TOKEN |
| DEF_RED | This state has a default reduction |
| ERR_DEF | combines ERR_ITEM and DEF_RED |
| REQ_DEF | combines REQ_TOKEN and DEF_RED |
| ERR_REQ_DEF | combines ERR_ITEM, REQ_TOKEN and DEF_RED |
--thread-safe was specified)
PARSE_ACCEPT rather than 0) may be
used as well.
SR_ s_0[] =
{
{ { DEF_RED}, { 2} },
{ { 258}, { 1} }, // start
{ { 0}, { -2} },
};
SR_ s_1[] =
{
{ { REQ_TOKEN}, { 4} },
{ { 259}, { 2} }, // expr
{ { 257}, { 3} }, // NR
{ { EOF_}, { PARSE_ACCEPT} },
{ { 0}, { 0} },
};
SR_ s_2[] =
{
{ { REQ_DEF}, { 2} },
{ { 43}, { 4} }, // '+'
{ { 0}, { -1} },
};
SR_ s_3[] =
{
{ { DEF_RED}, { 1} },
{ { 0}, { -3} },
};
SR_ s_4[] =
{
{ { REQ_TOKEN}, { 3} },
{ { 259}, { 5} }, // expr
{ { 257}, { 3} }, // NR
{ { 0}, { 0} },
};
SR_ s_5[] =
{
{ { REQ_DEF}, { 1} },
{ { 0}, { -4} },
};
parse(). This
function obtains its tokens from the member lex() and processes all tokens
until a syntactic error, a non-recoverable error, or the end of input is
encountered.
The algorithm used by parse() is the same, irrespective of the used
grammar. In fact, the parse() member's behavior is completely determined
by the tables generated by bisonc++.
The parsing algorithm is known as the shift-reduce (S/R) algorithm, and it
allows parse() to perform two actions while processing series of tokens:
NR token is observed in
state 1 of the example's grammar) a transition to state 3 is performed.
The parsing function maintains two stacks, which are manipulated by the above
two actions: a state stack and a value stack. These stacks are not accessible
to the parser: they are private data structures defined in the parser's base
class. The parsing member parse() may use the following member functions
to manipulate these stacks:
push_(stateIdx) pushes stateIdx on the state stack and pushes
the current semantic value (i.e., LTYPE_ d_val_) on the value stack;
pop_(size_t count = 1) removes count elements from the two
stacks;
top_() returns the state currently on top of the state stack;
Apart from the state- and semantic stacks, the S/R algorithm itself sometimes
needs to push a token on a two-element stack. Rather than using a formal
stack, two variables (d_token_ and d_nextToken_) are used to
implement this little token-stack. The member function pushToken_()
pushes a new value on the token stack, the member popToken_()
pops a previously pushed value from the token stack. At any time,
d_token_ contains the topmost element of the token stack.
The member nextToken() determines the next token to be processed. If the
token stack contains a value it is returned. Otherwise, lex() is called to
obtain the next token to be pushed on the token stack.
The member lookup() looks up the current token in the current state's
SR_ table. For this a simple linear search algorithm is used. If
searching fails to find an action for the token an UNEXPECTED_TOKEN_
exception is thrown, which starts the error recovery. If an action was found,
it is returned.
Rules may have actions associated with them. These actions are executed when a
grammatical rule has been completely recognized. This is always at the end of
a rule: mid-rule actions are converted by bisonc++ into pseudo nonterminals,
replacing mid-rule action blocks by these pseudo nonterminals. The pseudo
nonterminals show up in the verbose grammar output as rules having LHSs
starting with #. So, once a rule has been recognized its action (if
defined) is executed. For this the member function executeAction() is
available.
Finally, the token stack can be cleared using the member clearin().
Now that the relevant support functions have been introduced, the S/R algorithm itself turns out to be a fairly simple algorithm. First, the parser's stack is initialized with state 0 and the token stack is cleared. Then, in a never ending loop:
REQ_TOKEN has been specified for
that state), nextToken() is called to obtain the next token;
lookup() determines the next
action;
reduce_()):
the semantic and state stacks are reduced by the number of elements found in
that production rule, and the production rule's LHS is pushed on the token
stack
EOF is encountered in state 1) then the parsing function
terminates, returning 0.
The following table shows the S/R algorithm in action when the example grammar
is given the input 3 + 4 + 5. The first column shows the (remaining)
input, the second column the current token stack (with - indicating an
empty token stack), the third column the state stack. The fourth column
provides a short description. The leftmost elements of
the stacks represent the tops of the stacks. The information shown below is
also (in more elaborate form) shown when the --debug option is provided to
Bisonc++ when generating the parsing function.
| remaining input | token stack | state stack | description | |||
3 + 4 + 5 | - | 0 |
initialization | |||
3 + 4 + 5 | start | 0 |
reduction by rule 2 | |||
3 + 4 + 5 | - | 1 0 |
shift `start' | |||
+ 4 + 5 | NR | 1 0 |
obtain NR token | |||
+ 4 + 5 | - | 3 1 0 |
shift NR | |||
+ 4 + 5 | expr | 1 0 |
reduction by rule 3 | |||
+ 4 + 5 | - | 2 1 0 |
shift `expr' | |||
4 + 5 | + | 2 1 0 |
obtain `+' token | |||
4 + 5 | - | 4 2 1 0 |
shift `+' | |||
+ 5 | NR | 4 2 1 0 |
obtain NR token | |||
+ 5 | - | 3 4 2 1 0 |
shift NR | |||
+ 5 | expr | 4 3 1 0 |
reduction by rule 3 | |||
+ 5 | - | 5 4 3 1 0 |
shift `expr' | |||
5 | + | 5 4 3 1 0 |
obtain `+' token | |||
5 | expr + | 1 0 |
reduction by rule 4 | |||
5 | + | 2 1 0 |
shift `expr' | |||
5 | - | 4 2 1 0 |
shift '+' | |||
| NR | 4 2 1 0 |
obtain NR token | |||
| - | 3 4 2 1 0 |
shift NR | |||
| expr | 4 2 1 0 |
reduction by rule 3 | |||
| - | 5 4 2 1 0 |
shift `expr' | |||
| EOF | 5 4 2 1 0 |
obtain EOF | |||
| expr EOF | 1 0 |
reduction by rule 4 | |||
| EOF | 2 1 0 |
shift `expr' | |||
| start EOF | 2 1 0 |
reduction by rule 1 | |||
| EOF | 1 0 |
shift `start' | |||
| EOF | 1 0 |
ACCEPT | |||
if and if-else
statements, with a pair of rules like this:
if_stmt:
IF '(' expr ')' stmt
|
IF '(' expr ')' stmt ELSE stmt
;
Here we assume that IF and ELSE are terminal symbols for specific
keywords, and that expr and stmnt are defined nonterminals.
When the ELSE token is read and becomes the look-ahead token, the contents
of the stack (assuming the input is valid) are just right for reduction by
the first rule. But it is also legitimate to shift the ELSE, because
that would lead to eventual reduction by the second rule.
This situation, where either a shift or a reduction would be valid, is called
a shift/reduce conflict. Bisonc++ is designed to resolve these conflicts
by implementing a shift, unless otherwise directed by operator precedence
declarations. To see the reason for this, let's contrast it with the other
alternative.
Since the parser prefers to shift the ELSE, the result is to attach the
else-clause to the innermost if-statement, making these two inputs
equivalent:
if (x) if (y) then win(); else lose();
if (x)
{
if (y) then win(); else lose();
}
But if the parser would perform a reduction whenever possible rather
than a shift, the result would be to attach the else-clause to the
outermost if-statement, making these two inputs equivalent:
if (x) if (y) then win(); else lose();
if (x)
{
if (y) win();
}
else
lose();
The conflict exists because the grammar as written is ambiguous:
either parsing of the simple nested if-statement is legitimate. The
established convention is that these ambiguities are resolved by attaching the
else-clause to the innermost if-statement; this is what bisonc++ accomplishes
by implementing a shift rather than a reduce. This
particular ambiguity was first encountered in the specifications of Algol 60
and is called the dangling else ambiguity.
To avoid warnings from bisonc++ about predictable, legitimate shift/reduce
conflicts, use the %expect n directive. There will be no warning as long
as the number of shift/reduce conflicts is exactly n. See section
4.5.6.
The definition of if_stmt above is solely to blame for the conflict, but
the plain stmnt rule, consisting of two recursive alternatives will of
course never be able to match actual input, since there's no way for the
grammar to eventually derive a sentence this way. Adding one non-recursive
alternative is enough to convert the grammar into one that does derive
sentences. Here is a complete bisonc++ input file that actually shows the
conflict:
%token IF ELSE VAR
%%
stmt:
VAR ';'
|
IF '(' VAR ')' stmt
|
IF '(' VAR ')' stmt ELSE stmt
;
Looking again at the dangling else problem note that there are multiple ways
to handle stmnt productions. Depending on the particular input that is
provided it could
either be reduced to a stmt or the parser could continue to consume input
by processing an ELSE token, eventually resulting in the recognition of
IF '(' VAR ')' stmt ELSE stmt as a stmt.
There is little we can do but resorting to %expect to handle the dangling
else problem. The default handling is what most people intuitively expect and
so in this case using %expect 1 is an easy way to prevent bisonc++ from
reporting a shift/reduce conflict. But shift/reduce conflicts are most often
solved by specifying disambiguating rules specifying priorities or
associations, usually in the context of arithmetic expressions, as discussed
in the next sections.
However, shift-reduce conflicts can also be observed in grammars where a state contains items that could be reduced to a certain nonterminal and items in which a shift is possible in an item of a production rule of a completely different nonterminal. Here is an example of such a grammar:
%token ID
%left '-'
%left '*'
%right UNARY
%%
expr:
expr '-' term
|
term
;
term:
term '*' factor
|
factor
;
factor:
'-' expr %prec UNARY
|
ID
;
Why these grammars show shift reduce conflicts and how these are solved is
discussed in the next section.
1 - 2 * 3' can be parsed in two different ways):
expr:
expr '-' expr
|
expr '*' expr
|
expr '<' expr
|
'(' expr ')'
...
;
Suppose the parser has seen the tokens `1', `-' and `2';
should it reduce them via the rule for the addition operator? It depends on
the next token. Of course, if the next token is `)', we must reduce;
shifting is invalid because no single rule can reduce the token sequence `-
2 )' or anything starting with that. But if the next token is `*'
or `<', we have a choice: either shifting or reduction would allow the
parse to complete, but with different results.
To decide which one bisonc++ should do, we must consider the results. If
the next operator token op is shifted, then it must be reduced first in
order to permit another opportunity to reduce the sum. The result is (in
effect) `1 - (2 op 3)'. On the other hand, if the subtraction is reduced
before shifting op, the result is `(1 - 2) op 3'. Clearly, then, the
choice of shift or reduce should depend on the relative precedence of the
operators `-' and op: `*' should be shifted first, but not
`<'.
What about input such as `1 - 2 - 5'; should this be `(1 - 2) - 5' or
should it be `1 - (2 - 5)'? For most operators we prefer the former, which
is called left association. The latter alternative, right association,
is desirable for, e.g., assignment operators. The choice of left or right
association is a matter of whether the parser chooses to shift or reduce when
the stack contains `1 - 2' and the look-ahead token is `-': shifting
results in right-associativity.
%left and %right. Each such directive contains a list of
tokens, which are operators whose precedence and associativity is being
declared. The %left directive makes all those operators left-associative
and the %right directive makes them right-associative. A third alternative
is %nonassoc, which declares that it is a syntax error to find the same
operator twice `in a row'. Actually, %nonassoc is not currently (0.98.004)
punished that way by bisonc++. Instead, %nonassoc and %left are
handled identically.
The relative precedence of different operators is controlled by the order in
which they are declared. The first %left or %right directive in the
file declares the operators whose precedence is lowest, the next such
directive declares the operators whose precedence is a little higher, and so
on.
%left '<'
%left '-'
%left '*'
In a more complete example, which supports other operators as well, we
would declare them in groups of equal precedence. For example, '+' is
declared with '-':
%left '<' '>' '=' NE LE GE
%left '+' '-'
%left '*' '/'
(Here NE and so on stand for the operators for `not equal' and so
on. We assume that these tokens are more than one character long and therefore
are represented by names, not character literals.)
Finally, the resolution of conflicts works by comparing the precedence of the
rule being considered with that of the look-ahead token. If the token's
precedence is higher, the choice is to shift. If the rule's precedence is
higher, the choice is to reduce. If they have equal precedence, the choice is
made based on the associativity of that precedence level. The verbose output
file made by `-V' (see section 9) shows how each conflict was
resolved.
Not all rules and not all tokens have precedence. If either the rule or the look-ahead token has no precedence, then the default is to shift.
%token ID
%left '-'
%left '*'
%right UNARY
%%
expr:
expr '-' term
|
term
;
term:
term '*' factor
|
factor
;
factor:
'-' expr %prec UNARY
|
ID
;
Even though operator precedence and association rules are used the grammar still displays a shift/reduce conflict. One of the grammar's states consists of the following two items:
0: expr -> term .
1: term -> term . '*' factor
and bisonc++ reduces to item 0, dropping item 1 rather than shifting a '*' and
proceeding with item 0.
When considering states where shift/reduce conflicts are encountered the
`shiftable' items of these states shift when encountering terminal tokens that
are also in the follow sets of the reducible items of these states. In the
above example item 1 shifts when '*' is encountered, but '*' is also
an element of the set of look-ahead tokens of item 0. Bisonc++ must now decide what
to do. In cases we've seen earlier bisonc++ could make the decision because the
reducible item itself had a well known precedence. The precedence of a
reducible item is defined as the precedence of the rule's LHS. Item
0 in the above example is an item of the rule expr -> term.
The precedence of a production rule is defined as follows:
%prec is used then the precedence of the production rule is
equal to the precedence of the terminal that is specified with the %prec
directive;
%prec is not used then the production rule's precedence is
equal to the precedence of the first terminal token that is used in the
production rule;
Since expr -> term does not contain a terminal token and does not use
%prec, its precedence is the maximum possible precedence. Consequently in
the above state the shift/reduce conflict is solved by reducing rather
than shifting.
Some final remark as to why the above grammar is peculiar. It is peculiar as
it combines precedence and association specifying directives with auxiliary
nonterminals that may be useful conceptually (or when implementing an
expression parser `by hand') but which are not required when defining grammars
for bisonc++. The following grammar does not use term and factor but
recognizes the same grammar as the above `peculiar' grammar without reporting
any shift/reduce conflict:
%token ID
%left '-'
%left '*'
%right UNARY
%%
expr:
expr '-' expr
|
expr '*' expr
|
'-' expr %prec UNARY
|
ID
;
The bisonc++ precedence directives, %left, %right and %nonassoc, can only be used once for a given token; so a token has only one precedence declared in this way. For context-dependent precedence, you need to use an additional mechanism: the %prec modifier for rules.
The %prec modifier declares the precedence of a particular rule by specifying a terminal symbol whose precedence should be used for that rule. It's not necessary for that symbol to appear otherwise in the rule. The modifier's syntax is:
%prec terminal-symbol
and it is written after the components of the rule. Its effect is to assign the rule the precedence of terminal-symbol, overriding the precedence that would be deduced for it in the ordinary way. The altered rule precedence then affects how conflicts involving that rule are resolved (see section Operator Precedence).
Here is how %prec solves the problem of unary minus. First, declare a precedence for a fictitious terminal symbol named UMINUS. There are no tokens of this type, but the symbol serves to stand for its precedence:
...
%left '+' '-'
%left '*'
%left UMINUS
Now the precedence of UMINUS can be used in specific rules:
exp:
...
| exp '-' exp
...
| '-' exp %prec UMINUS
For example, here is an erroneous attempt to define a sequence of zero or more words:
%stype char *
%token WORD
%%
sequence:
// empty
{
cout << "empty sequence\n";
}
|
maybeword
|
sequence WORD
{
cout << "added word " << $2 << endl;
}
;
maybeword:
// empty
{
cout << "empty maybeword\n";
}
|
WORD
{
cout << "single word " << $1 << endl;
}
;
The error is an ambiguity: there is more than one way to parse a single word into a sequence. It could be reduced to a maybeword and then into a sequence via the second rule. Alternatively, nothing-at-all could be reduced into a sequence via the first rule, and this could be combined with the word using the third rule for sequence.
There is also more than one way to reduce nothing-at-all into a sequence. This can be done directly via the first rule, or indirectly via maybeword and then the second rule.
You might think that this is a distinction without a difference, because it does not change whether any particular input is valid or not. But it does affect which actions are run. One parsing order runs the second rule's action; the other runs the first rule's action and the third rule's action. In this example, the output of the program changes.
Bisonc++ resolves a reduce/reduce conflict by choosing to use the rule that appears first in the grammar, but it is very risky to rely on this. Every reduce/reduce conflict must be studied and usually eliminated. Here is the proper way to define sequence:
sequence:
// empty
{ printf ("empty sequence\n"); }
| sequence word
{ printf ("added word %s\n", $2); }
;
Here is another common error that yields a reduce/reduce conflict:
sequence:
// empty
| sequence words
| sequence redirects
;
words:
// empty
| words word
;
redirects:
// empty
| redirects redirect
;
The intention here is to define a sequence containing either word or
redirect nonterminals. The individual definitions of sequence, words and
redirects are error-free, but the three together make a subtle ambiguity: even
an empty input can be parsed in infinitely many ways!
Consider: nothing-at-all could be a words. Or it could be two words in a row, or three, or any number. It could equally well be a redirects, or two, or any number. Or it could be a words followed by three redirects and another words. And so on.
Here are two ways to correct these rules. First, to make it a single level of sequence:
sequence:
// empty
| sequence word
| sequence redirect
;
Second, to prevent either a words or a redirects from being empty:
sequence:
// empty
| sequence words
| sequence redirects
;
words:
word
| words word
;
redirects:
redirect
| redirects redirect
;
%token ID
%%
def:
param_spec return_spec ','
;
param_spec:
type
|
name_list ':' type
;
return_spec:
type
|
name ':' type
;
type:
ID
;
name:
ID
;
name_list:
name
|
name ',' name_list
;
It would seem that this grammar can be parsed with only a single
look-ahead token: when a param_spec is being read, an ID is a name if
a comma or colon follows, or a type if another ID follows. In other
words, this grammar is LR(1).
However, bisonc++, like most parser generators, cannot actually handle all LR(1)
grammars. In this grammar two contexts, one after an ID at the beginning
of a param_spec and another one at the beginning of a return_spec, are
similar enough for bisonc++ to assume that they are identical. They appear similar
because the same set of rules would be active--the rule for reducing to a name
and that for reducing to a type. Bisonc++ is unable to determine at that stage of
processing that the rules would require different look-ahead tokens in the two
contexts, so it makes a single parser state for them both. Combining the two
contexts causes a conflict later. In parser terminology, this occurrence means
that the grammar is not LALR(1).
In general, it is better to fix deficiencies than to document them. But this particular deficiency is intrinsically hard to fix; parser generators that can handle LR(1) grammars are hard to write and tend to produce parsers that are very large. In practice, bisonc++ is more useful the way it's currently operating.
When the problem arises, you can often fix it by identifying the two parser
states that are being confused, and adding something to make them look
distinct. In the above example, adding one rule to return_spec as follows
makes the problem go away:
%token BOGUS
...
%%
...
return_spec:
type
|
name ':' type
|
ID BOGUS // This rule is never used.
;
This corrects the problem because it introduces the possibility of an
additional active rule in the context after the ID at the beginning of
return_spec. This rule is not active in the corresponding context in a
param_spec, so the two contexts receive distinct parser states. As long as
the token BOGUS is never generated by the parser's member function
lex(), the added rule cannot alter the way actual input is parsed.
In this particular example, there is another way to solve the problem: rewrite
the rule for return_spec to use ID directly instead of via name. This
also causes the two confusing contexts to have different sets of active rules,
because the one for return_spec activates the altered rule for
return_spec rather than the one for name.
param_spec:
type
|
name_list ':' type
;
return_spec:
type
|
ID ':' type
;