7.8.13. `TokenNgram`#

7.8.13.1. Summary#

TokenNgram can change define its behavior dynamically via its options. For example, we can use it as unigram, bigram, trigram on with changing n option value as below.

Uni-gram:

Execution example:

tokenize --tokenizer 'TokenNgram("n", 1)' --string "Hello World"
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "H",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "e",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "l",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "l",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "o",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": " ",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "W",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "o",
#       "position": 7,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "r",
#       "position": 8,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "l",
#       "position": 9,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "d",
#       "position": 10,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

Bi-gram:

Execution example:

tokenize --tokenizer 'TokenNgram("n", 2)' --string "Hello World"
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "He",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "el",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ll",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "lo",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "o ",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": " W",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "Wo",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "or",
#       "position": 7,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "rl",
#       "position": 8,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ld",
#       "position": 9,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "d",
#       "position": 10,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

Tri-gram:

Execution example:

tokenize --tokenizer 'TokenNgram("n", 3)' --string "Hello World"
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "Hel",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ell",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "llo",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "lo ",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "o W",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": " Wo",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "Wor",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "orl",
#       "position": 7,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "rld",
#       "position": 8,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ld",
#       "position": 9,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "d",
#       "position": 10,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

This tokenizer also has options other than the above.

7.8.13.2. Syntax#

TokenNgram has optional parameter.

No options:

TokenNgram

If we don’t use options, TokenNgram is the same behavior for TokenBigram.

Specify option:

TokenNgram("n", true)

TokenNgram("loose_symbol", true)

TokenNgram("loose_blank", true)

TokenNgram("remove_blank", true)

TokenNgram("report_source_location", true)

TokenNgram("unify_alphabet", true)

TokenNgram("unify_symbol", true)

TokenNgram("unify_digit", true)

Specify multiple options:

TokenNgram("loose_symbol", true, "loose_blank", true)

We can also specify multiple options at the same time except the above examples.

7.8.13.3. Usage#

7.8.13.4. Simple usage#

Here is an example of TokenNgram.

If we use TokenNgram in the nothing of option. TokenNgram behavior is the same as TokenBigram as below:

If no normalizer is used.

Execution example:

tokenize --tokenizer TokenNgram --string "Hello World"
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "He",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "el",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ll",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "lo",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "o ",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": " W",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "Wo",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "or",
#       "position": 7,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "rl",
#       "position": 8,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ld",
#       "position": 9,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "d",
#       "position": 10,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

If normalizer is used.

Execution example:

tokenize --tokenizer TokenNgram --string "Hello World" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "hello",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "world",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

7.8.13.5. Advanced usage#

We can specify multiple options for TokenNgram.

For example, we can deal with variantsas as phone number by using loose_symbol and loose_blank as below.

We can search 0123(45)6789, 0123-45-6789 and, 0123 45 6789 by using 0123456789 as below example.

Execution example:

tokenize --tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)' --string "0123(45)6789" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "0123",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "(",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "45",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": ")",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "6789",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "￰",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "0123456789",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]
tokenize --tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)' --string "0123-45-6789" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "0123",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "-",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "45",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "-",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "6789",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "￰",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "0123456789",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]
tokenize --tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)' --string "0123 45 6789" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "0123",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "45",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "6789",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "￰",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "0123456789",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

7.8.13.6. Parameters#

7.8.13.6.1. Optional parameter#

There are eight optional parameters.

7.8.13.6.1.1. `n`#

This option shows N of Ngram. For example, n is 3 for trigram.

7.8.13.6.1.2. `ignore_blank`#

It ignores white-spaces in continuous symbols and non-ASCII characters. For example, the keyword 日本語 ! ! ! is tokenized as 日本語!!!.

7.8.13.6.1.3. `loose_symbol`#

Tokenize keywords including symbols, to be searched by both queries with/without symbols. For example, a keyword 090-1111-2222 will be found by any of 09011112222, 090, 1111, 2222 and 090-1111-2222.

7.8.13.6.1.4. `loose_blank`#

Tokenize keywords including blanks, to be searched by both queries with/without blanks. For example, a keyword 090 1111 2222 will be found by any of 09011112222, 090, 1111, 2222 and 090 1111 2222.

7.8.13.6.1.5. `remove_blank`#

Tokenize keywords including blanks, to be searched by queries without blanks. For example, a keyword 090 1111 2222 will be found by any of 09011112222, 090, 1111 or 2222.

Note that the keyword won’t be found by a query including blanks like 090 1111 2222.

7.8.13.6.1.6. `report_source_location`#

This option tells us a location of token of highlight target when we highlight token by highlight_html.

We only use this option when we want to highlight token by highlight_html.

When Groonga tokenize texts that target highlight, always used NormalizerAuto to normalizer until now. Therefore, if we use NormalizerNFKC100 to normalizer, sometimes it can’t find the position of the highlight as below.

Execution example:

table_create Entries TABLE_NO_KEY
# [[0,1337566253.89858,0.000355720520019531],true]
column_create Entries body COLUMN_SCALAR ShortText
# [[0,1337566253.89858,0.000355720520019531],true]
table_create Terms TABLE_PAT_KEY ShortText --normalizer 'NormalizerNFKC100'
# [[0,1337566253.89858,0.000355720520019531],true]
column_create Terms document_index COLUMN_INDEX|WITH_POSITION Entries body
# [[0,1337566253.89858,0.000355720520019531],true]
load --table Entries
[
{"body": "ア㌕Ａz"}
]
# [[0,1337566253.89858,0.000355720520019531],1]
select Entries \
  --match_columns body \
  --query 'グラム' \
  --output_columns 'highlight_html(body, Terms)'
# [[0,1337566253.89858,0.000355720520019531],[[[0],[["highlight_html",null]]]]]

Because we use different normalizer to normalize token.

This option is used to reduce the shift of the position of the highlight as below.

Execution example:

table_remove Terms
# [[0,1337566253.89858,0.000355720520019531],true]
table_create Terms TABLE_PAT_KEY ShortText \
  --default_tokenizer 'TokenNgram("report_source_location", true)' \
  --normalizer 'NormalizerNFKC100'
# [[0,1337566253.89858,0.000355720520019531],true]
column_create Terms document_index COLUMN_INDEX|WITH_POSITION Entries body
# [[0,1337566253.89858,0.000355720520019531],true]
select Entries \
  --match_columns body \
  --query 'グラム' \
  --output_columns 'highlight_html(body, Terms)'
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     [
#       [
#         1
#       ],
#       [
#         [
#           "highlight_html",
#           null
#         ]
#       ],
#       [
#         "ア<span class=\"keyword\">㌕</span>Ａz"
#       ]
#     ]
#   ]
# ]

7.8.13.6.1.7. `unify_alphabet`#

If we set false, TokenNgram uses bigram tokenize method for ASCII character.

Default value of this option is true.

Execution example:

tokenize --tokenizer 'TokenNgram("unify_alphabet", false)' --string "Hello World" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "he",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "el",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ll",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "lo",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "o",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "wo",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "or",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "rl",
#       "position": 7,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "ld",
#       "position": 8,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "d",
#       "position": 9,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

7.8.13.6.1.8. `unify_symbol`#

If we set false, TokenNgram uses bigram tokenize method for symbols. TokenNgram("unify_symbol", false) is same behavior of TokenBigramSplitSymbol.

Default value of this option is true.

Execution example:

tokenize --tokenizer 'TokenNgram("unify_symbol", false)' --string "___---" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "__",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "__",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "_-",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "--",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "--",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "-",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

7.8.13.6.1.9. `unify_digit`#

If we set false, TokenNgram uses bigram tokenize method for digits.

Default value of this option is true.

Execution example:

tokenize --tokenizer 'TokenNgram("unify_digit", false)' --string "012345 6789" --normalizer NormalizerAuto
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "value": "01",
#       "position": 0,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "12",
#       "position": 1,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "23",
#       "position": 2,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "34",
#       "position": 3,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "45",
#       "position": 4,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "5",
#       "position": 5,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "67",
#       "position": 6,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "78",
#       "position": 7,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "89",
#       "position": 8,
#       "force_prefix": false,
#       "force_prefix_search": false
#     },
#     {
#       "value": "9",
#       "position": 9,
#       "force_prefix": false,
#       "force_prefix_search": false
#     }
#   ]
# ]

7.8.13.7. See also#

tokenize

7.8.13. TokenNgram#

7.8.13.1. Summary#

7.8.13.2. Syntax#

7.8.13.3. Usage#

7.8.13.4. Simple usage#

7.8.13.5. Advanced usage#

7.8.13.6. Parameters#

7.8.13.6.1. Optional parameter#

7.8.13.6.1.1. n#

7.8.13.6.1.2. ignore_blank#

7.8.13.6.1.3. loose_symbol#

7.8.13.6.1.4. loose_blank#

7.8.13.6.1.5. remove_blank#

7.8.13.6.1.6. report_source_location#

7.8.13.6.1.7. unify_alphabet#

7.8.13.6.1.8. unify_symbol#

7.8.13.6.1.9. unify_digit#

7.8.13.7. See also#

7.8.13. `TokenNgram`#

7.8.13.6.1.1. `n`#

7.8.13.6.1.2. `ignore_blank`#

7.8.13.6.1.3. `loose_symbol`#

7.8.13.6.1.4. `loose_blank`#

7.8.13.6.1.5. `remove_blank`#

7.8.13.6.1.6. `report_source_location`#

7.8.13.6.1.7. `unify_alphabet`#

7.8.13.6.1.8. `unify_symbol`#

7.8.13.6.1.9. `unify_digit`#