Lexical analysis

@token(regex, types=None)

Decorator for token definitions in classes derived from LexerBase.

Parameters

• rx (str) – A regular expression defining the possible token values

• types (List of strings) – A list of token types that this method can recognize. If omitted, the token type is assumed to be the method’s name.

Basic usage:

from ptk.lexer import ReLexer, token

class MyLexer(ReLexer):
    @token(r'[a-zA-Z_][a-zA-Z0-9_]*')
    def identifier(self, tok):
        pass

This will define an identifier token type, which value is the recognized string. The tok parameter holds two attributes, type and value. You can modify the value in place:

from ptk.lexer import ReLexer, token

class MyLexer(ReLexer):
    @token(r'[1-9][0-9]*')
    def number(self, tok):
        tok.value = int(tok.value)

In some cases it may be necessary to change the token’s type as well; for instance to disambiguate between identifiers that are builtins and other ones. In order for the lexer to know which token types can be generated, you should pass a list of strings as the types parameter:

from ptk.lexer import ReLexer, token

class MyLexer(ReLexer):
    @token(r'[a-zA-Z_][a-zA-Z0-9_]*', types=['builtin', 'identifier'])
    def identifier_or_builtin(self, tok):
        tok.type = 'builtin' if tok.value in ['len', 'id'] else 'identifier'

In this case the default value of the type attribute is None and you must set it. Letting None as token type (or setting it to None) will cause the token to be ignored.

Note

The type of token values depends on the type of the strings used to define the regular expressions. Unicode expressions will hold Unicode values, and bytes expressions will hold bytes values.

Note

Disambiguation is done the regular way: if several regular expressions match the input, the longest match is choosen. If the matches are of equal length, the first (in source code order) declaration wins.

exception ptk.lexer.SkipToken

Raise this from your consumer to ignore the token.

exception ptk.lexer.LexerError(char, pos)

Unrecognized token in input

Variables

• lineno – Line in input

• colno – Column in input

class ptk.lexer.LexerBase

This defines the interface for lexer classes. For concrete implementations, see ProgressiveLexer and ReLexer.

class Token(type, value, position)

position

Alias for field number 2

type

Alias for field number 0

value

Alias for field number 1

position()

Returns

The current position in stream as a 2-tuple (column, line).

advanceColumn(count=1)

Advances the current position by count columns.

advanceLine(count=1)

Advances the current position by count lines.

static ignore(char)

Override this to ignore characters in input stream. The default is to ignore spaces and tabs.

Parameters

char – The character to test

Returns

True if char should be ignored

setConsumer(consumer)

Sets the current consumer. A consumer is an object with a feed method; all characters seen on the input stream after the consumer is set are passed directly to it. When the feed method returns a 2-tuple (type, value), the corresponding token is generated and the consumer reset to None. This may be handy to parse tokens that are not easily recognized by a regular expression but easily by code; for instance the following lexer recognizes C strings without having to use negative lookahead:

class MyLexer(ReLexer):
    @token('"')
    def cstring(self, tok):
        class CString(object):
            def __init__(self):
                self.state = 0
                self.value = StringIO.StringIO()
            def feed(self, char):
                if self.state == 0:
                    if char == '"':
                        return 'cstring', self.value.getvalue()
                    if char == '\\':
                        self.state = 1
                    else:
                        self.value.write(char)
                elif self.state == 1:
                    self.value.write(char)
                    self.state = 0
        self.setConsumer(CString())

You can also raise SkipToken instead of returning a token if it is to be ignored (comments).

parse(string)

Parses the whole string

newToken(tok)

This method will be invoked as soon as a token is recognized on input.

Parameters

tok – The token. This is a named tuple with type and value attributes.

classmethod tokenTypes()

Returns

the set of all token names, as strings.

class ptk.lexer.ReLexer

Concrete lexer based on Python regular expressions. this is way faster than ProgressiveLexer but it can only tokenize whole strings.

parse(string)

Parses the whole string

class ptk.lexer.ProgressiveLexer

Concrete lexer based on a simple pure-Python regular expression engine. This lexer is able to tokenize an input stream in a progressive fashion; just call the ProgressiveLexer.feed() method with whatever bytes are available when they’re available. Useful for asynchronous contexts. Starting with Python 3.5 there is also an asynchronous version, see AsyncLexer.

This is slow as hell.

parse(string)

Parses the whole string

feed(char)

Handle a single input character. When you’re finished, call this with EOF as argument.

class ptk.async_lexer.AsyncLexer

This class works like ProgressiveLexer but can be feed the input asynchronously via asyncFeed(). It works with AsyncLRParser.

ptk.lexer.EOF

This is a singleton used to indicate end of stream. It may be used as a token, a token type and a token value. In the first case it is its own type and value.