7.7.2.5. NormalizerNFKC150#
7.7.2.5.1. Summary#
Added in version 13.0.0.
NormalizerNFKC150 normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 15.0.
This normalizer can change behavior by specifying options.
7.7.2.5.2. Syntax#
NormalizerNFKC150 has optional parameter.
No options:
NormalizerNFKC150
NormalizerNFKC150 normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 15.0.
Specify option:
NormalizerNFKC150("unify_kana", true)
NormalizerNFKC150("unify_kana_case", true)
NormalizerNFKC150("unify_kana_voiced_sound_mark", true)
NormalizerNFKC150("unify_hyphen", true)
NormalizerNFKC150("unify_prolonged_sound_mark", true)
NormalizerNFKC150("unify_hyphen_and_prolonged_sound_mark", true)
NormalizerNFKC150("unify_middle_dot", true)
NormalizerNFKC150("unify_katakana_v_sounds", true)
NormalizerNFKC150("unify_katakana_bu_sound", true)
NormalizerNFKC150("unify_to_katakana", true)
NormalizerNFKC150("unify_to_romaji", true)
NormalizerNFKC150("remove_symbol", true)
NormalizerNFKC150("unify_katakana_gu_small_sounds", true)
NormalizerNFKC150("unify_katakana_di_sound", true)
NormalizerNFKC150("unify_katakana_wo_sound", true)
NormalizerNFKC150("unify_katakana_zu_small_sounds", true)
NormalizerNFKC150("unify_katakana_du_sound", true)
NormalizerNFKC150("unify_katakana_trailing_o", true)
NormalizerNFKC150("unify_katakana_du_small_sounds", true)
NormalizerNFKC150("unify_kana_prolonged_sound_mark", true)
NormalizerNFKC150("unify_kana_hyphen", true)
NormalizerNFKC150("unify_latin_alphabet_with", true)
Added in version 14.0.7: unify_latin_alphabet_with is added.
Added in version 13.0.1: unify_kana_prolonged_sound_mark is added.
unify_kana_hyphen is added.
Specify multiple options:
NormalizerNFKC150("unify_to_romaji", true, "unify_kana_case", true, "unify_hyphen_and_prolonged_sound_mark", true)
NormalizerNFKC150 also specify multiple options as above. You can also specify mingle multiple options except above example.
7.7.2.5.3. Usage#
7.7.2.5.3.1. Simple usage#
Here is an example of NormalizerNFKC150. NormalizerNFKC150 normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 15.0.
Execution example:
normalize NormalizerNFKC150 "©" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "©",
# "types": [
# "emoji",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_kana option.
This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.
Execution example:
normalize 'NormalizerNFKC150("unify_kana", true)' "あイｳｪおヽヾ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "あいうぇおゝゞ",
# "types": [
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_kana_case option.
This option enables that large and small versions of same letters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.
Execution example:
normalize 'NormalizerNFKC150("unify_kana_case", true)' "ぁあぃいぅうぇえぉおゃやゅゆょよゎわゕかゖけ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ああいいううええおおややゆゆよよわわかかけけ",
# "types": [
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Execution example:
normalize 'NormalizerNFKC150("unify_kana_case", true)' "ァアィイゥウェエォオャヤュユョヨヮワヵカヶケ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "アアイイウウエエオオヤヤユユヨヨワワカカケケ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_kana_voiced_sound_mark option.
This option enables that letters with/without voiced sound mark and semi voiced sound mark in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.
Execution example:
normalize 'NormalizerNFKC150("unify_kana_voiced_sound_mark", true)' "かがきぎくぐけげこごさざしじすずせぜそぞただちぢつづてでとどはばぱひびぴふぶぷへべぺほぼぽ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "かかききくくけけここささししすすせせそそたたちちつつててととはははひひひふふふへへへほほほ",
# "types": [
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "hiragana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Execution example:
normalize 'NormalizerNFKC150("unify_kana_voiced_sound_mark", true)' "カガキギクグケゲコゴサザシジスズセゼソゾタダチヂツヅテデトドハバパヒビピフブプヘベペホボポ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "カカキキククケケココササシシススセセソソタタチチツツテテトトハハハヒヒヒフフフヘヘヘホホホ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_hyphen option. This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS) as below.
Execution example:
normalize 'NormalizerNFKC150("unify_hyphen", true)' "-˗֊‐‑‒–⁃⁻₋−" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "-----------",
# "types": [
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_prolonged_sound_mark option. This option enables normalize prolonged sound to “-” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK) as below.
Execution example:
normalize 'NormalizerNFKC150("unify_prolonged_sound_mark", true)' "ー—―─━ｰ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ーーーーーー",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_hyphen_and_prolonged_sound_mark option. This option enables normalize hyphen and prolonged sound to “-” (U+002D HYPHEN-MINUS) as below.
Execution example:
normalize 'NormalizerNFKC150("unify_hyphen_and_prolonged_sound_mark", true)' "-˗֊‐‑‒–⁃⁻₋− ﹣－ ー—―─━ｰ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "----------- -- ------",
# "types": [
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "others",
# "symbol",
# "symbol",
# "others",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_middle_dot option. This option enables normalize middle dot to “·” (U+00B7 MIDDLE DOT) as below.
You can use it with remove_symbol to remove all characters like middle dot.
Execution example:
normalize 'NormalizerNFKC150("unify_middle_dot", true)' "·ᐧ•∙⋅⸱・･" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "········",
# "types": [
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "symbol",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_v_sounds option. This option enables normalize “ヴァヴィヴヴェヴォ” to “バビブベボ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_v_sounds", true)' "ヴァヴィヴヴェヴォヴ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "バビブベボブ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_bu_sound option. This option enables normalize “ヴァヴィヴゥヴェヴォ” to “ブ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_bu_sound", true)' "ヴァヴィヴヴェヴォヴ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ブブブブブブ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_to_katakana option. This option normalizes hiragana to katakana.
Execution example:
normalize 'NormalizerNFKC150("unify_to_katakana", true)' "ゔぁゔぃゔゔぇゔぉ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ヴァヴィヴヴェヴォ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_to_romaji option. This option enables normalize hiragana and katakana to romaji as below.
Execution example:
normalize 'NormalizerNFKC150("unify_to_romaji", true)' "アァイィウゥエェオォ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "axaixiuxuexeoxo",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of remove_symbol option. This option removes symbols (e.g. #, !, “, &, %, …) as below.
Execution example:
normalize 'NormalizerNFKC150("remove_symbol", true)' "#This & is %% a pen." WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "this is a pen",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "others",
# "others",
# "alpha",
# "alpha",
# "others",
# "others",
# "alpha",
# "others",
# "alpha",
# "alpha",
# "alpha",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_gu_small_sounds option. This option enables to normalize “グァグィグェグォ” to “ガギゲゴ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_gu_small_sounds", true)' "グァグィグェグォ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ガギゲゴ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_di_sound option. This option enables to normalize “ヂ” to “ジ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_di_sound", true)' "ヂ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ジ",
# "types": [
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_wo_sound option. This option enables to normalize “ヲ” to “オ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_wo_sound", true)' "ヲ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "オ",
# "types": [
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_zu_small_sounds option. This option enables to normalize “ズァズィズェズォ” to “ザジゼゾ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_zu_small_sounds", true)' "ズァズィズェズォ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ザジゼゾ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_du_sound option. This option enables to normalize “ヅ” to “ズ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_du_sound", true)' "ヅ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ズ",
# "types": [
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_trailing_o option. This option enables to normalize “オ” to “ウ” when the vowel in the previous letter is “オ” as below.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_trailing_o", true)' "オオコオソオトオノオ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "オウコウソウトウノウ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_katakana_du_small_sounds option. This option enables to normalize “ヅァヅィヅェヅォ” to “ザジゼゾ”.
Execution example:
normalize 'NormalizerNFKC150("unify_katakana_du_small_sounds", true)' "ヅァヅィヅェヅォ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ザジゼゾ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_kana_prolonged_sound_mark option. This option enables to normalize “ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK) to a vowel of a previous kana letter.
If a previous kana letter is “ん” , “ー” is normalized to “ん”, And a previous kana letter is “ン” , “ー” is normalized to “ン”.
Execution example:
normalize 'NormalizerNFKC150("unify_kana_prolonged_sound_mark", true)' "カーキークーケーコー" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "カアキイクウケエコオ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_kana_hyphen option. This option enables to normalize “-” (U+002D HYPHEN-MINUS) to a vowel of a previous kana letter.
If a previous kana letter is “ん” , “-” is normalized to “ん”, And a previous kana letter is “ン” , “-” is normalized to “ン”.
Execution example:
normalize 'NormalizerNFKC150("unify_kana_hyphen", true)' "カ-キ-ク-ケ-コ-" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "カアキイクウケエコオ",
# "types": [
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "katakana",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
Here is an example of unify_latin_alphabet_with option. This option enables that alphabets with diacritical mark and alphabets without diacritical mark regarded as the same character as below.
However, this feature focus on only LATIN (SMALL|CAPITAL) LETTER X WITH XXX. It doesn’t support LATIN (SMALL|CAPITAL) LETTER X + COMBINING XXX characters.
Execution example:
normalize 'NormalizerNFKC150("unify_latin_alphabet_with", true)' "ngoằn" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "ngoan",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
7.7.2.5.3.2. Advanced usage#
7.7.2.5.3.2.1. With TokenMecab#
You can output romaji of specific a part of speech with using to combine
TokenMecab and NormalizerNFKC150 as below.
First of all, you extract reading of a noun with excluding non-independent word and suffix of person name with target_class option and include_reading option.
Next, you normalize reading of the noun that extracted with unify_to_romaji option of NormalizerNFKC150.
Execution example:
tokenize 'TokenMecab("target_class", "-名詞/非自立", "target_class", "-名詞/接尾/人名", "target_class", "名詞", "include_reading", true)' '彼の名前は山田さんのはずです。'
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# [
# {
# "value": "彼",
# "position": 0,
# "force_prefix": false,
# "force_prefix_search": false,
# "metadata": {
# "reading": "カレ"
# }
# },
# {
# "value": "名前",
# "position": 1,
# "force_prefix": false,
# "force_prefix_search": false,
# "metadata": {
# "reading": "ナマエ"
# }
# },
# {
# "value": "山田",
# "position": 2,
# "force_prefix": false,
# "force_prefix_search": false,
# "metadata": {
# "reading": "ヤマダ"
# }
# }
# ]
# ]
normalize 'NormalizerNFKC150("unify_to_romaji", true)' "カレ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "kare",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
normalize 'NormalizerNFKC150("unify_to_romaji", true)' "ナマエ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "namae",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
normalize 'NormalizerNFKC150("unify_to_romaji", true)' "ヤマダ" WITH_TYPES
# [
# [
# 0,
# 1337566253.89858,
# 0.000355720520019531
# ],
# {
# "normalized": "yamada",
# "types": [
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "alpha",
# "null"
# ],
# "checks": [
#
# ]
# }
# ]
7.7.2.5.3.2.2. Use unify_to_katakana with other options#
unify_to_katakana can be combined with the following options to equate special katakana with general katakana.
-
Equivalent: “ゔぁゔぃゔゔぇゔぉ”, “ばびぶべぼ”, “ヴァヴィヴヴェヴォ” and “バビブベボ”
unify_katakana_gu_small_sounds
Equivalent: “ぐぁぐぃぐぇぐぉ”, “がぎげご”, “グァグィグェグォ” and “ガギゲゴ”
unify_katakana_zu_small_sounds
Equivalent: “ずぁずぃずぇずぉ”, “ざじぜぞ”, “ズァズィズェズォ” and “ザジゼゾ”
-
Equivalent: “お”, “を”, “オ” and “ヲ”
-
Equivalent: “じ”, “ぢ”, “ジ” and “ヂ”
-
Equivalent: “ず”, “づ”, “ズ” and “ヅ”
For example, using unify_to_katakana and unify_katakana_v_sounds together, you can search “バイオリン”, “ヴァイオリン”, “ばいおりん” and “ゔぁいおりん” with “ばいおりん”.
7.7.2.5.4. Parameters#
7.7.2.5.4.1. Optional parameter#
There are optional parameters as below.
7.7.2.5.4.1.1. unify_kana#
This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.
7.7.2.5.4.1.2. unify_kana_case#
This option enables that large and small versions of same letters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.
7.7.2.5.4.1.3. unify_kana_voiced_sound_mark#
This option enables that letters with/without voiced sound mark and semi voiced sound mark in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.
7.7.2.5.4.1.4. unify_hyphen#
This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS).
Hyphen of the target of normalizing is as below.
“-” (U+002D HYPHEN-MINUS)
“֊” (U+058A ARMENIAN HYPHEN)
“˗” (U+02D7 MODIFIER LETTER MINUS SIGN)
“‐” (U+2010 HYPHEN)
“—” (U+2014 EM DASH)
“⁃” (U+2043 HYPHEN BULLET)
“⁻” (U+207B SUPERSCRIPT MINUS)
“₋” (U+208B SUBSCRIPT MINUS)
“−” (U+2212 MINUS SIGN)
7.7.2.5.4.1.5. unify_prolonged_sound_mark#
This option enables normalize prolonged sound to “-” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK).
Prolonged sound of the target of normalizing is as below.
“—” (U+2014 EM DASH)
“―” (U+2015 HORIZONTAL BAR)
“─” (U+2500 BOX DRAWINGS LIGHT HORIZONTAL)
“━” (U+2501 BOX DRAWINGS HEAVY HORIZONTAL)
“ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK)
“ｰ” (U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK)
7.7.2.5.4.1.6. unify_hyphen_and_prolonged_sound_mark#
This option enables normalize hyphen and prolonged sound to “-” (U+002D HYPHEN-MINUS).
Hyphen and prolonged sound of the target normalizing is below.
“-” (U+002D HYPHEN-MINUS)
“֊” (U+058A ARMENIAN HYPHEN)
“˗” (U+02D7 MODIFIER LETTER MINUS SIGN)
“‐” (U+2010 HYPHEN)
“—” (U+2014 EM DASH)
“⁃” (U+2043 HYPHEN BULLET)
“⁻” (U+207B SUPERSCRIPT MINUS)
“₋” (U+208B SUBSCRIPT MINUS)
“−” (U+2212 MINUS SIGN)
“—” (U+2014 EM DASH)
“―” (U+2015 HORIZONTAL BAR)
“─” (U+2500 BOX DRAWINGS LIGHT HORIZONTAL)
“━” (U+2501 BOX DRAWINGS HEAVY HORIZONTAL)
“ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK)
“ｰ” (U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK)
7.7.2.5.4.1.7. unify_middle_dot#
This option enables normalize middle dot to “·” (U+00B7 MIDDLE DOT).
Middle dot of the target of normalizing is as below.
“·” (U+00B7 MIDDLE DOT)
“ᐧ” (U+1427 CANADIAN SYLLABICS FINAL MIDDLE DOT)
“•” (U+2022 BULLET)
“∙” (U+2219 BULLET OPERATOR)
“⋅” (U+22C5 DOT OPERATOR)
“⸱” (U+2E31 WORD SEPARATOR MIDDLE DOT)
“・” (U+30FB KATAKANA MIDDLE DOT)
“･” (U+FF65 HALFWIDTH KATAKANA MIDDLE DOT)
You can use it with remove_symbol to remove all characters like middle dot.
7.7.2.5.4.1.8. unify_katakana_v_sounds#
This option enables normalize “ヴァヴィヴヴェヴォ” to “バビブベボ”.
7.7.2.5.4.1.9. unify_katakana_bu_sound#
This option enables normalize “ヴァヴィヴゥヴェヴォ” to “ブ”.
7.7.2.5.4.1.10. unify_to_katakana#
This option normalizes hiragana to katakana.
7.7.2.5.4.1.11. unify_to_romaji#
This option enables normalize hiragana and katakana to romaji.
7.7.2.5.4.1.12. remove_symbol#
This option removes symbols (e.g. #, !, “, &, %, …) from the string that the target of normalizing.
7.7.2.5.4.1.13. unify_katakana_gu_small_sounds#
Added in version 13.0.0.
This option enables to normalize “グァグィグェグォ” to “ガギゲゴ”.
7.7.2.5.4.1.14. unify_katakana_di_sound#
Added in version 13.0.0.
This option enables to normalize “ヂ” to “ジ”.
7.7.2.5.4.1.15. unify_katakana_wo_sound#
Added in version 13.0.0.
This option enables to normalize “ヲ” to “オ”.
7.7.2.5.4.1.16. unify_katakana_zu_small_sounds#
Added in version 13.0.0.
This option enables to normalize “ズァズィズェズォ” to “ザジゼゾ”.
7.7.2.5.4.1.17. unify_katakana_du_sound#
Added in version 13.0.0.
This option enables to normalize “ヅ” to “ズ”.
7.7.2.5.4.1.18. unify_katakana_trailing_o#
Added in version 13.0.0.
This option enables to normalize “オ” to “ウ” when the vowel in the previous letter is “オ”.
“ォオ” -> “ォウ”
“オオ” -> “オウ”
“コオ” -> “コウ”
“ソオ” -> “ソウ”
“トオ” -> “トウ”
“ノオ” -> “ノウ”
“ホオ” -> “ホウ”
“モオ” -> “モウ”
“ョオ” -> “ョオ”
“ヨオ” -> “ヨウ”
“ロオ” -> “ロウ”
“ゴオ” -> “ゴウ”
“ゾオ” -> “ゾウ”
“ドオ” -> “ドウ”
“ボオ” -> “ボウ”
“ポオ” -> “ポウ”
“ヺオ” -> “ヺウ”
7.7.2.5.4.1.19. unify_katakana_du_small_sounds#
Added in version 13.0.0.
This option enables to normalize “ヅァヅィヅェヅォ” to “ザジゼゾ”.
7.7.2.5.4.1.20. unify_kana_prolonged_sound_mark#
Added in version 13.0.1.
This option enables to normalize “ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK) to a vowel of a previous kana letter.
If a previous kana letter is “ん” , “ー” is normalized to “ん”, And a previous kana letter is “ン” , “ー” is normalized to “ン”.
ァー -> ァア, アー -> アア, ヵー -> ヵア, カー -> カア, ガー -> ガア, サー -> サア, ザー -> ザア,
ター -> タア, ダー -> ダア, ナー -> ナア, ハー -> ハア, バー -> バア, パー -> パア, マー -> マア,
ャー -> ャア, ヤー -> ヤア, ラー -> ラア, ヮー -> ヮア, ワー -> ワア, ヷー -> ヷア,
ィー -> ィイ, イー -> イイ, キー -> キイ, ギー -> ギイ, シー -> シイ, ジー -> ジイ, チー -> チイ,
ヂー -> ヂイ, ニー -> ニイ, ヒー -> ヒイ, ビー -> ビイ, ピー -> ピイ, ミー -> ミイ, リー -> リイ,
ヰー -> ヰイ, ヸー -> ヸイ,
ゥー -> ゥウ, ウー -> ウウ, クー -> クウ, グー -> グウ, スー -> スウ, ズー -> ズウ, ツー -> ツウ,
ヅー -> ヅウ, ヌー -> ヌウ, フー -> フウ, ブー -> ブウ, プー -> プウ, ムー -> ムウ, ュー -> ュウ,
ユー -> ユウ, ルー -> ルウ, ヱー -> ヱウ, ヴー -> ヴウ,
ェー -> ェエ, エー -> エエ, ヶー -> ヶエ, ケー -> ケエ, ゲー -> ゲエ, セー -> セエ, ゼー -> ゼエ,
テー -> テエ, デー -> デエ, ネー -> ネエ, ヘー -> ヘエ, ベー -> ベエ, ペー -> ペエ, メー -> メエ,
レー -> レエ, ヹー -> ヹエ,
ォー -> ォオ, オー -> オオ, コー -> コオ, ゴー -> ゴオ, ソー -> ソオ, ゾー -> ゾオ, トー -> トオ,
ドー -> ドオ, ノー -> ノオ, ホー -> ホオ, ボー -> ボオ, ポー -> ポオ, モー -> モオ, ョー -> ョオ,
ヨー -> ヨオ, ロー -> ロオ, ヲー -> ヲオ, ヺー -> ヺオ,
ンー -> ンン
ぁー -> ぁあ, あー -> ああ, ゕー -> ゕあ, かー -> かあ, がー -> があ, さー -> さあ, ざー -> ざあ,
たー -> たあ, だー -> だあ, なー -> なあ, はー -> はあ, ばー -> ばあ, ぱー -> ぱあ, まー -> まあ,
ゃー -> ゃあ, やー -> やあ, らー -> らあ, ゎー -> ゎあ, わー -> わあ
ぃー -> ぃい, いー -> いい, きー -> きい, ぎー -> ぎい, しー -> しい, じー -> じい, ちー -> ちい,
ぢー -> ぢい, にー -> にい, ひー -> ひい, びー -> びい, ぴー -> ぴい, みー -> みい, りー -> りい,
ゐー -> ゐい
ぅー -> ぅう, うー -> うう, くー -> くう, ぐー -> ぐう, すー -> すう, ずー -> ずう, つー -> つう,
づー -> づう, ぬー -> ぬう, ふー -> ふう, ぶー -> ぶう, ぷー -> ぷう, むー -> むう, ゅー -> ゅう,
ゆー -> ゆう, るー -> るう, ゑー -> ゑう, ゔー -> ゔう
ぇー -> ぇえ, えー -> ええ, ゖー -> ゖえ, けー -> けえ, げー -> げえ, せー -> せえ, ぜー -> ぜえ,
てー -> てえ, でー -> でえ, ねー -> ねえ, へー -> へえ, べー -> べえ, ぺー -> ぺえ, めー -> めえ,
れー -> れえ
ぉー -> ぉお, おー -> おお, こー -> こお, ごー -> ごお, そー -> そお, ぞー -> ぞお, とー -> とお,
どー -> どお, のー -> のお, ほー -> ほお, ぼー -> ぼお, ぽー -> ぽお, もー -> もお, ょー -> ょお,
よー -> よお, ろー -> ろお, をー -> をお
んー -> んん
7.7.2.5.4.1.21. unify_kana_hyphen#
Added in version 13.0.1.
This option enables to normalize “-” (U+002D HYPHEN-MINUS) to a vowel of a previous kana letter.
If a previous kana letter is “ん” , “-” is normalized to “ん”, And a previous kana letter is “ン” , “-” is normalized to “ン”.
ァ- -> ァア, ア- -> アア, ヵ- -> ヵア, カ- -> カア, ガ- -> ガア, サ- -> サア, ザ- -> ザア,
タ- -> タア, ダ- -> ダア, ナ- -> ナア, ハ- -> ハア, バ- -> バア, パ- -> パア, マ- -> マア,
ャ- -> ャア, ヤ- -> ヤア, ラ- -> ラア, ヮ- -> ヮア, ワ- -> ワア, ヷ- -> ヷア,
ィ- -> ィイ, イ- -> イイ, キ- -> キイ, ギ- -> ギイ, シ- -> シイ, ジ- -> ジイ, チ- -> チイ,
ヂ- -> ヂイ, ニ- -> ニイ, ヒ- -> ヒイ, ビ- -> ビイ, ピ- -> ピイ, ミ- -> ミイ, リ- -> リイ,
ヰ- -> ヰイ, ヸ- -> ヸイ,
ゥ- -> ゥウ, ウ- -> ウウ, ク- -> クウ, グ- -> グウ, ス- -> スウ, ズ- -> ズウ, ツ- -> ツウ,
ヅ- -> ヅウ, ヌ- -> ヌウ, フ- -> フウ, ブ- -> ブウ, プ- -> プウ, ム- -> ムウ, ュ- -> ュウ,
ユ- -> ユウ, ル- -> ルウ, ヱ- -> ヱウ, ヴ- -> ヴウ,
ェ- -> ェエ, エ- -> エエ, ヶ- -> ヶエ, ケ- -> ケエ, ゲ- -> ゲエ, セ- -> セエ, ゼ- -> ゼエ,
テ- -> テエ, デ- -> デエ, ネ- -> ネエ, ヘ- -> ヘエ, ベ- -> ベエ, ペ- -> ペエ, メ- -> メエ,
レ- -> レエ, ヹ- -> ヹエ,
ォ- -> ォオ, オ- -> オオ, コ- -> コオ, ゴ- -> ゴオ, ソ- -> ソオ, ゾ- -> ゾオ, ト- -> トオ,
ド- -> ドオ, ノ- -> ノオ, ホ- -> ホオ, ボ- -> ボオ, ポ- -> ポオ, モ- -> モオ, ョ- -> ョオ,
ヨ- -> ヨオ, ロ- -> ロオ, ヲ- -> ヲオ, ヺ- -> ヺオ,
ン- -> ンン
ぁ- -> ぁあ, あ- -> ああ, ゕ- -> ゕあ, か- -> かあ, が- -> があ, さ- -> さあ, ざ- -> ざあ,
た- -> たあ, だ- -> だあ, な- -> なあ, は- -> はあ, ば- -> ばあ, ぱ- -> ぱあ, ま- -> まあ,
ゃ- -> ゃあ, や- -> やあ, ら- -> らあ, ゎ- -> ゎあ, わ- -> わあ
ぃ- -> ぃい, い- -> いい, き- -> きい, ぎ- -> ぎい, し- -> しい, じ- -> じい, ち- -> ちい,
ぢ- -> ぢい, に- -> にい, ひ- -> ひい, び- -> びい, ぴ- -> ぴい, み- -> みい, り- -> りい,
ゐ- -> ゐい
ぅ- -> ぅう, う- -> うう, く- -> くう, ぐ- -> ぐう, す- -> すう, ず- -> ずう, つ- -> つう,
づ- -> づう, ぬ- -> ぬう, ふ- -> ふう, ぶ- -> ぶう, ぷ- -> ぷう, む- -> むう, ゅ- -> ゅう,
ゆ- -> ゆう, る- -> るう, ゑ- -> ゑう, ゔ- -> ゔう
ぇ- -> ぇえ, え- -> ええ, ゖ- -> ゖえ, け- -> けえ, げ- -> げえ, せ- -> せえ, ぜ- -> ぜえ,
て- -> てえ, で- -> でえ, ね- -> ねえ, へ- -> へえ, べ- -> べえ, ぺ- -> ぺえ, め- -> めえ,
れ- -> れえ
ぉ- -> ぉお, お- -> おお, こ- -> こお, ご- -> ごお, そ- -> そお, ぞ- -> ぞお, と- -> とお,
ど- -> どお, の- -> のお, ほ- -> ほお, ぼ- -> ぼお, ぽ- -> ぽお, も- -> もお, ょ- -> ょお,
よ- -> よお, ろ- -> ろお, を- -> をお
ん- -> んん
7.7.2.5.4.1.22. unify_latin_alphabet_with#
Added in version 14.0.7.
This option enables that alphabets with diacritical mark and alphabets without diacritical mark regarded as the same character as below.
However, this feature focus on only LATIN (SMALL|CAPITAL) LETTER X WITH XXX. It doesn’t support LATIN (SMALL|CAPITAL) LETTER X + COMBINING XXX characters.