7.8.7. TokenBigramSplitSymbol¶
7.8.7.1. Summary¶
TokenBigramSplitSymbol is similar to TokenBigram. The
difference between them is symbol handling.
7.8.7.2. Syntax¶
TokenBigramSplitSymbol hasn’t parameter:
TokenBigramSplitSymbol
7.8.7.3. Usage¶
TokenBigramSplitSymbol tokenizes symbols by bigram tokenize method:
Execution example:
tokenize TokenBigramSplitSymbol "100cents!!!" NormalizerAuto
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# [
# {
# "position": 0,
# "force_prefix": false,
# "value": "100"
# },
# {
# "position": 1,
# "force_prefix": false,
# "value": "cents"
# },
# {
# "position": 2,
# "force_prefix": false,
# "value": "!!"
# },
# {
# "position": 3,
# "force_prefix": false,
# "value": "!!"
# },
# {
# "position": 4,
# "force_prefix": false,
# "value": "!"
# }
# ]
# ]