7.8.16. TokenTable¶
7.8.16.1. Summary¶
TokenTable is a tokenizer which treats only known keywords as a token.
The known keywords must be registered as a key in the table for TokenTable.
For example, query contains a known keyword and a unknown keyword, only a known keyword is used in search query (unknown keyword will be ignored). In other words, you can search contents with only known keywords. Because of this characteristic, you need to maintain keyword table for a new keyword continuously.
7.8.16.2. Syntax¶
TokenTable has a required parameter.
Specify the table:
TokenTable("table", TABLE)
TABLE must be created with --default_tokenizer 'TokenTable("table", TABLE).
7.8.16.3. Usage¶
Here is an example of TokenTable.
For example, let’s search Raspberry Pie from Pies table.
The table which is used for keyword is Keywords.
Here is the sample schema and data:
Execution example:
table_create Pies TABLE_NO_KEY
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Pies name COLUMN_SCALAR Text
# [[0, 1337566253.89858, 0.000355720520019531], true]
table_create Keywords TABLE_PAT_KEY ShortText --default_tokenizer 'TokenTable("table", "Keywords")'
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Keywords index COLUMN_INDEX Pies name
# [[0, 1337566253.89858, 0.000355720520019531], true]
load --table Keywords
[
{"_key": "Apple"}
{"_key": "Orange"}
{"_key": "Raspberry"}
]
# [[0, 1337566253.89858, 0.000355720520019531], 3]
load --table Pies
[
{"name": "Apple Pie"},
{"name": "Orange Pie"}
{"name": "Raspberry Pie"}
{"name": "Stargazy Pie"}
]
# [[0, 1337566253.89858, 0.000355720520019531], 4]
Then search Raspberry Pie with --query Raspberry.
Execution example:
select Pies --match_columns name --query 'Raspberry'
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# [
# [
# [
# 1
# ],
# [
# [
# "_id",
# "UInt32"
# ],
# [
# "name",
# "Text"
# ]
# ],
# [
# 3,
# "Raspberry Pie"
# ]
# ]
# ]
# ]
As you expected, the above query matches the Raspberry Pie record.
Next, search Stargazy Pie with --query Stargazy.
Execution example:
select Pies --match_columns name --query 'Stargazy'
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# [
# [
# [
# 0
# ],
# [
# [
# "_id",
# "UInt32"
# ],
# [
# "name",
# "Text"
# ]
# ]
# ]
# ]
# ]
In above example, as the keyword Stargazy is not registered in Keywords table yet, it doesn’t match anything.