7.7.2.3. NormalizerNFKC121¶
7.7.2.3.1. Summary¶
New in version 9.0.3.
NormalizerNFKC121 normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 12.1.
This normalizer can change behavior by specifying options.
7.7.2.3.2. Syntax¶
NormalizerNFKC121 has optional parameter.
No options:
NormalizerNFKC121
NormalizerNFKC121 normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 12.1.
Specify option:
NormalizerNFKC121("unify_kana", true)
NormalizerNFKC121("unify_kana_case", true)
NormalizerNFKC121("unify_kana_voiced_sound_mark", true)
NormalizerNFKC121("unify_hyphen", true)
NormalizerNFKC121("unify_prolonged_sound_mark", true)
NormalizerNFKC121("unify_hyphen_and_prolonged_sound_mark", true)
NormalizerNFKC121("unify_middle_dot", true)
NormalizerNFKC121("unify_katakana_v_sounds", true)
NormalizerNFKC121("unify_katakana_bu_sound", true)
NormalizerNFKC121("unify_to_romaji", true)
Specify multiple options:
NormalizerNFKC121("unify_to_romaji", true, "unify_kana_case", true, "unify_hyphen_and_prolonged_sound_mark", true)
NormalizerNFKC121 also specify multiple options as above. You can also specify mingle multiple options except above example.
7.7.2.3.3. Usage¶
7.7.2.3.3.1. Simple usage¶
Here is an example of NormalizerNFKC121. NormalizerNFKC121 normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 12.1.
Execution example:
normalize NormalizerNFKC121 "©" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "©",
# "types": [
# "emoji"
# ],
# "checks": []
# }
# ]
Here is an example of unify_kana option.
This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.
Execution example:
normalize 'NormalizerNFKC121("unify_kana", true)' "あイｳｪおヽヾ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "あいうぇおゝゞ",
# "types": [
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana"
# ],
# "checks": []
# }
# ]
Here is an example of unify_kana_case option.
This option enables that large and small versions of same letters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.
Execution example:
normalize 'NormalizerNFKC121("unify_kana_case", true)' "ぁあぃいぅうぇえぉおゃやゅゆょよゎわゕかゖけ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ああいいううええおおややゆゆよよわわかかけけ",
# "types": [
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana"
# ],
# "checks": []
# }
# ]
Execution example:
normalize 'NormalizerNFKC121("unify_kana_case", true)' "ァアィイゥウェエォオャヤュユョヨヮワヵカヶケ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "アアイイウウエエオオヤヤユユヨヨワワカカケケ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana"
# ],
# "checks": []
# }
# ]
Here is an example of unify_kana_voiced_sound_mark option.
This option enables that letters with/without voiced sound mark and semi voiced sound mark in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.
Execution example:
normalize 'NormalizerNFKC121("unify_kana_voiced_sound_mark", true)' "かがきぎくぐけげこごさざしじすずせぜそぞただちぢつづてでとどはばぱひびぴふぶぷへべぺほぼぽ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "かかききくくけけここささししすすせせそそたたちちつつててととはははひひひふふふへへへほほほ",
# "types": [
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana"
# ],
# "checks": []
# }
# ]
Execution example:
normalize 'NormalizerNFKC121("unify_kana_voiced_sound_mark", true)' "カガキギクグケゲコゴサザシジスズセゼソゾタダチヂツヅテデトドハバパヒビピフブプヘベペホボポ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "カカキキククケケココササシシススセセソソタタチチツツテテトトハハハヒヒヒフフフヘヘヘホホホ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana"
# ],
# "checks": []
# }
# ]
Here is an example of unify_hyphen option. This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS) as below.
Execution example:
normalize 'NormalizerNFKC121("unify_hyphen", true)' "-˗֊‐‑‒–⁃⁻₋−" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "-----------",
# "types": [
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol"
# ],
# "checks": []
# }
# ]
Here is an example of unify_prolonged_sound_mark option. This option enables normalize prolonged sound to “-” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK) as below.
Execution example:
normalize 'NormalizerNFKC121("unify_prolonged_sound_mark", true)' "ー—―─━ｰ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ーーーーーー",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana"
# ],
# "checks": []
# }
# ]
Here is an example of unify_hyphen_and_prolonged_sound_mark option. This option enables normalize hyphen and prolonged sound to “-” (U+002D HYPHEN-MINUS) as below.
Execution example:
normalize 'NormalizerNFKC121("unify_hyphen_and_prolonged_sound_mark", true)' "-˗֊‐‑‒–⁃⁻₋− ﹣－ ー—―─━ｰ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "----------- -- ------",
# "types": [
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "others",
# "symbol",
# "symbol",
# "others",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol"
# ],
# "checks": []
# }
# ]
Here is an example of unify_middle_dot option. This option enables normalize middle dot to “·” (U+00B7 MIDDLE DOT) as below.
Execution example:
normalize 'NormalizerNFKC121("unify_middle_dot", true)' "·ᐧ•∙⋅⸱・･" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "········",
# "types": [
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol"
# ],
# "checks": []
# }
# ]
Here is an example of unify_katakana_v_sounds option. This option enables normalize “ヴァヴィヴヴェヴォ” to “バビブベボ” as below.
Execution example:
normalize 'NormalizerNFKC121("unify_katakana_v_sounds", true)' "ヴァヴィヴヴェヴォヴ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "バビブベボブ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana"
# ],
# "checks": []
# }
# ]
Here is an example of unify_katakana_bu_sound option. This option enables normalize “ヴァヴィヴゥヴェヴォ” to “ブ” as below.
Execution example:
normalize 'NormalizerNFKC121("unify_katakana_bu_sound", true)' "ヴァヴィヴヴェヴォヴ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ブブブブブブ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana"
# ],
# "checks": []
# }
# ]
Here is an example of unify_to_romaji option. This option enables normalize hiragana and katakana to romaji as below.
Execution example:
normalize 'NormalizerNFKC121("unify_to_romaji", true)' "アァイィウゥエェオォ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "axaixiuxuexeoxo",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha"
# ],
# "checks": []
# }
# ]
7.7.2.3.3.2. Advanced usage¶
You can output romaji of specific a part of speech with using to combine
TokenMecab and NormalizerNFKC121 as below.
First of all, you extract reading of a noun with excluding non-independent word and suffix of person name with target_class option and include_reading option.
Next, you normalize reading of the noun that extracted with unify_to_romaji option of NormalizerNFKC121.
Execution example:
tokenize 'TokenMecab("target_class", "-名詞/非自立", "target_class", "-名詞/接尾/人名", "target_class", "名詞", "include_reading", true)' '彼の名前は山田さんのはずです。'
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# [
# {
# "force_prefix_search": false,
# "position": 0,
# "force_prefix": false,
# "value": "彼",
# "metadata": {
# "reading": "カレ"
# }
# },
# {
# "force_prefix_search": false,
# "position": 1,
# "force_prefix": false,
# "value": "名前",
# "metadata": {
# "reading": "ナマエ"
# }
# },
# {
# "force_prefix_search": false,
# "position": 2,
# "force_prefix": false,
# "value": "山田",
# "metadata": {
# "reading": "ヤマダ"
# }
# }
# ]
# ]
normalize 'NormalizerNFKC121("unify_to_romaji", true)' "カレ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "kare",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha"
# ],
# "checks": []
# }
# ]
normalize 'NormalizerNFKC121("unify_to_romaji", true)' "ナマエ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "namae",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha"
# ],
# "checks": []
# }
# ]
normalize 'NormalizerNFKC121("unify_to_romaji", true)' "ヤマダ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "yamada",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha"
# ],
# "checks": []
# }
# ]
7.7.2.3.4. Parameters¶
7.7.2.3.4.1. Optional parameter¶
There are optional parameters as below.
7.7.2.3.4.1.1. unify_kana¶
This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.
7.7.2.3.4.1.2. unify_kana_case¶
This option enables that large and small versions of same letters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.
7.7.2.3.4.1.3. unify_kana_voiced_sound_mark¶
This option enables that letters with/without voiced sound mark and semi voiced sound mark in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.
7.7.2.3.4.1.4. unify_hyphen¶
This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS).
Hyphen of the target of normalizing is as below.
“-” (U+002D HYPHEN-MINUS)
“֊” (U+058A ARMENIAN HYPHEN)
“˗” (U+02D7 MODIFIER LETTER MINUS SIGN)
“‐” (U+2010 HYPHEN)
“—” (U+2014 EM DASH)
“⁃” (U+2043 HYPHEN BULLET)
“⁻” (U+207B SUPERSCRIPT MINUS)
“₋” (U+208B SUBSCRIPT MINUS)
“−” (U+2212 MINUS SIGN)
7.7.2.3.4.1.5. unify_prolonged_sound_mark¶
This option enables normalize prolonged sound to “-” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK).
Prolonged sound of the target of normalizing is as below.
“—” (U+2014 EM DASH)
“―” (U+2015 HORIZONTAL BAR)
“─” (U+2500 BOX DRAWINGS LIGHT HORIZONTAL)
“━” (U+2501 BOX DRAWINGS HEAVY HORIZONTAL)
“ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK)
“ｰ” (U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK)
7.7.2.3.4.1.6. unify_hyphen_and_prolonged_sound_mark¶
This option enables normalize hyphen and prolonged sound to “-” (U+002D HYPHEN-MINUS).
Hyphen and prolonged sound of the target normalizing is below.
“-” (U+002D HYPHEN-MINUS)
“֊” (U+058A ARMENIAN HYPHEN)
“˗” (U+02D7 MODIFIER LETTER MINUS SIGN)
“‐” (U+2010 HYPHEN)
“—” (U+2014 EM DASH)
“⁃” (U+2043 HYPHEN BULLET)
“⁻” (U+207B SUPERSCRIPT MINUS)
“₋” (U+208B SUBSCRIPT MINUS)
“−” (U+2212 MINUS SIGN)
“—” (U+2014 EM DASH)
“―” (U+2015 HORIZONTAL BAR)
“─” (U+2500 BOX DRAWINGS LIGHT HORIZONTAL)
“━” (U+2501 BOX DRAWINGS HEAVY HORIZONTAL)
“ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK)
“ｰ” (U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK)
7.7.2.3.4.1.7. unify_middle_dot¶
This option enables normalize middle dot to “·” (U+00B7 MIDDLE DOT).
Middle dot of the target of normalizing is as below.
“·” (U+00B7 MIDDLE DOT)
“ᐧ” (U+1427 CANADIAN SYLLABICS FINAL MIDDLE DOT)
“•” (U+2022 BULLET)
“∙” (U+2219 BULLET OPERATOR)
“⋅” (U+22C5 DOT OPERATOR)
“⸱” (U+2E31 WORD SEPARATOR MIDDLE DOT)
“・” (U+30FB KATAKANA MIDDLE DOT)
“･” (U+FF65 HALFWIDTH KATAKANA MIDDLE DOT)
7.7.2.3.4.1.8. unify_katakana_v_sounds¶
This option enables normalize “ヴァヴィヴヴェヴォ” to “バビブベボ”.
7.7.2.3.4.1.9. unify_katakana_bu_sound¶
This option enables normalize “ヴァヴィヴゥヴェヴォ” to “ブ”.
7.7.2.3.4.1.10. unify_to_romaji¶
This option enables normalize hiragana and katakana to romaji.