Method Search.Grammar.Lexer.tokenize()

Method tokenize

string|array(array(Token|string)) tokenize(string query)

Description

Tokenizes a query into tokens for later use by a parser.

Parameter query

The query to tokenize.

Returns

An array containing the tokens: ({ ({ TOKEN_WORD, "foo" }), ... }) Or, in case of an error, a string with the error message.