Method Search.Grammar.Lexer.tokenize()

Method tokenize: string|array(array(Token|string)) tokenize(string query)
Description: Tokenizes a query into tokens for later use by a parser.
Parameter query: The query to tokenize.
Returns: An array containing the tokens: ({ ({ TOKEN_WORD, "foo" }), ... }) Or, in case of an error, a string with the error message.