Release 11.0.7 - 2021-09-29¶

Improvements¶

• [load] Added support for casting a string like as “[int, int,…]” to a vector of integer like as [int, int,…].

  For example, Groonga handle as a vector of integer like as [1, -2] even if we load vector of string like as “[1, -2]” as below.

  table_create Data TABLE_NO_KEY
  column_create Data numbers COLUMN_VECTOR Int16
  table_create Numbers TABLE_PAT_KEY Int16
  column_create Numbers data_numbers COLUMN_INDEX Data numbers
  
  load --table Data
  [
  {"numbers": "[1, -2]"},
  {"numbers": "[-3, 4]"}
  ]
  
  dump   --dump_plugins no   --dump_schema no
  load --table Data
  [
  ["_id","numbers"],
  [1,[1,-2]],
  [2,[-3,4]]
  ]
  
  column_create Numbers data_numbers COLUMN_INDEX Data numbers
  select Data --filter 'numbers @ -2'
  [[0,0.0,0.0],[[[1],[["_id","UInt32"],["numbers","Int16"]],[1,[1,-2]]]]]
  

  This feature supports for the floowings types.

  • Int8

  • UInt8

  • Int16

  • UInt16

  • Int32

  • UInt32

  • Int64

  • UInt64

• [load] Added support for loading a JSON array expressed as a text string as a vector of string.

  For example, Groonga handle as a vector that has two elements like as [“hello”, “world”] if we load JSON array expressed as a text string like as “["hello", "world"]” as below.

  table_create Data TABLE_NO_KEY
  [[0,0.0,0.0],true]
  column_create Data strings COLUMN_VECTOR ShortText
  [[0,0.0,0.0],true]
  table_create Terms TABLE_PAT_KEY ShortText   --normalizer NormalizerNFKC130   --default_tokenizer TokenNgram
  [[0,0.0,0.0],true]
  column_create Terms data_strings COLUMN_INDEX Data strings
  [[0,0.0,0.0],true]
  load --table Data
  [
  {"strings": "[\"Hello\", \"World\"]"},
  {"strings": "[\"Good-bye\", \"World\"]"}
  ]
  [[0,0.0,0.0],2]
  dump   --dump_plugins no   --dump_schema no
  load --table Data
  [
  ["_id","strings"],
  [1,["Hello","World"]],
  [2,["Good-bye","World"]]
  ]
  
  column_create Terms data_strings COLUMN_INDEX Data strings
  select Data --filter 'strings @ "bye"'
  [
    [
      0,
      0.0,
      0.0
    ],
    [
      [
        [
          1
        ],
        [
          [
            "_id",
            "UInt32"
          ],
          [
            "strings",
            "ShortText"
          ]
        ],
        [
          2,
          [
            "Good-bye",
            "World"
          ]
        ]
      ]
    ]
  ]
  

  In before version, Groonga handled as a vector that had one element like as [“["hello", "world"]”] if we loaded JSON array expressed as a text string like as “["hello", "world"]”.

• [Documentation] Added a documentation about the following items.

  • [column_create] Added a documentation about WEIGHT_FLOAT32 flag.

  • [NormalizerNFKC121] Added a documentation about NormalizerNFKC121.

  • [NormalizerNFKC130] Added a documentation about NormalizerNFKC130.

  • [NormalizerTable] Added a documentation about NormalizerTable.

• Updated to 3.0.0 that the version of Apache Arrow that Groonga requires. [GitHub#1265][Patched by Takashi Hashida]

Fixes¶

• Fixed a memory leak when we created a table with a tokenizer with invalid option.

• Fixed a bug that may not add a new entry in Hash table.

  This bug only occurs in Groonga 11.0.6, and it may occur if we quite a lot of add and delete data. If this bug occurs in your environment, you can resolve this problem by executing the following steps.

  1. We upgrade Groonga to 11.0.7 or later from 11.0.6.

  2. We make a new table that has the same schema as the original table.

  3. We copy data to the new table from the original table.

• [Windows] Fixed a resource leak when Groonga fail open a new file caused by out of memory.

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

• [The browser based administration tool] Currently, Groonga has a bug that a search query that is inputted to non-administration mode is sent even if we input checks to the checkbox for the administration mode of a record list.

• *< and *> only valid when we use query() the right side of filter condition. If we specify as below, *< and *> work as &&.

  • 'content @ "Groonga" *< content @ "Mroonga"'

• Groonga may not return records that should match caused by GRN_II_CURSOR_SET_MIN_ENABLE.

Thanks¶

• Takashi Hashida

Release 11.0.6 - 2021-08-29¶

Warning

Groonga 11.0.6 has had a bug that may not add a new entry in Hash table.

We fixed this bug on Groonga 11.0.7. This bug only occurs in Groonga 11.0.6. Therefore, if you were using Groonga 11.0.6, we highly recommended that you use Groonga 11.0.7 or later.

Improvements¶

• Added support for recovering on crash. (experimental)

  This is a experimental feature. Currently, this feature is still not stable.

  If Groonga crashes, it recovers the database automatically when it opens a database for the first time since the crash. However, This feature can’t recover the database automatically in all crash cases. We need to recover the database manually depending on timing even if this feature enables.

  Groonga execute WAL (write ahead log) when this feature is enable. We can dump WAL by the following tools, but currently, users doesn’t need to use them.

  • [grndb] dump-wal command.

  • dump-wal.rb scritp.

• [cache_limit] Groonga remove cache when we execute cache_limit 0. [GitHub#1224][Reported by higchi]

  Groonga stores query cache to internally table. The maximum total size of keys of this table is 4GiB. Because this table is hash table. Therefore, If we execute many huge queries, Groonga may be unable to store query cache, because the maximum total size of keys may be over 4GiB. In such cases, We can clear the table for query cache by using cache_limit 0, and Groonga can store query cache

Fixes¶

• Fixed a bug that Groonga doesn’t clear lock when some threads open the same object around the same time.

  If some threads open the same object around the same time, threads except for a thread that executes the opening object at first are waiting for opening the target object. At this time, threads that wait for an opening object take locks, but these locks are not released. Therefore, these locks remain until Groonga’s process is restarted in the above case, and a new thread can’t also open the object all the time until Groonga’s process is restarted.

  However, this bug rarely happens. Because a time of a thread open the object is a very short time.

• [query_parallel_or] Fixed a bug that result may be different from the query().

  For example, If we used query("tags || tags2", "beginner man"), the following record was a match, but if we used query_parallel_or("tags || tags2", "beginner man"), the following record wasn’t a match until now.

  • {"_key": "Bob",   "comment": "Hey!",       "tags": ["expert", "man"], "tags2": ["beginner"]}

  Even if we use query_parallel_or("tags || tags2", "beginner man"), the above record is match by this modification.

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

• [The browser based administration tool] Currently, Groonga has a bug that a search query that is inputted to non-administration mode is sent even if we input checks to the checkbox for the administration mode of a record list.

• *< and *> only valid when we use query() the right side of filter condition. If we specify as below, *< and *> work as &&.

  • 'content @ "Groonga" *< content @ "Mroonga"'

• Groonga may not return records that should match caused by GRN_II_CURSOR_SET_MIN_ENABLE.

Thanks¶

• higchi

Release 11.0.5 - 2021-07-29¶

Improvements¶

• [Normalizers] Added support for multiple normalizers.

  We can specify multiple normalizers by --notmalizers option when we create a table since this release. If we can also specify them by --normalizer existing option because of compatibility.

  We added NormalizerTable for customizing a normalizer in Groonga 11.0.4. We can more flexibly behavior of the normalizer by combining NormalizerTable with existing normalizer.

  For example, this feature is useful in the following case.

  • Search for a telephone number. However, we import data handwritten by OCR. If data is handwritten, OCR may misunderstand a number and string(e.g. 5 and S).

  The details are as follows.

  table_create Normalizations TABLE_PAT_KEY ShortText
  column_create Normalizations normalized COLUMN_SCALAR ShortText
  load --table Normalizations
  [
  {"_key": "s", "normalized": "5"}
  ]
  
  
  table_create Tels TABLE_NO_KEY
  column_create Tels tel COLUMN_SCALAR ShortText
  
  table_create TelsIndex TABLE_PAT_KEY ShortText \
    --normalizers 'NormalizerNFKC130("unify_hyphen_and_prolonged_sound_mark", true), \
                   NormalizerTable("column", "Normalizations.normalized")' \
    --default_tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)'
  column_create TelsIndex tel_index COLUMN_INDEX|WITH_SECTION Tels tel
  
  load --table Tels
  [
  {"tel": "03-4S-1234"}
  {"tel": "03-45-9876"}
  ]
  
  select --table Tels \
    --filter 'tel @ "03-45-1234"'
  [
    [
      0,
      1625227424.560146,
      0.0001730918884277344
    ],
    [
      [
        [
          1
        ],
        [
          [
            "_id",
            "UInt32"
          ],
          [
            "tel",
            "ShortText"
          ]
        ],
        [
          1,
          "03-4S-1234"
        ]
      ]
    ]
  ]
  

  Existing normalizers can’t meet in such case, but we can meet it by combining NormalizerTable with existing normalizer since this release.

• [query_parallel_or][query] Added support for customizing thresholds for sequential search.

  We can customize thresholds in each queries whether to use sequential search by the following options.

  • {"max_n_enough_filtered_records": xx}

    max_n_enough_filtered_records specify the number of records. query or query_parallel_or use sequential search when they seems to narrow down until under this number.

  • {"enough_filtered_ratio": x.x}

    enough_filtered_ratio specify percentage of total. query or query_parallel_or use sequential search when they seems to narrow down until under this percentage. For example, if we specify {"enough_filtered_ratio": 0.5}, query or query_parallel_or use sequential search when they seems to narrow down until half of the whole.

  The details are as follows.

  table_create Products TABLE_NO_KEY
  column_create Products name COLUMN_SCALAR ShortText
  
  table_create Terms TABLE_PAT_KEY ShortText --normalizer NormalizerAuto
  column_create Terms products_name COLUMN_INDEX Products name
  
  load --table Products
  [
  ["name"],
  ["Groonga"],
  ["Mroonga"],
  ["Rroonga"],
  ["PGroonga"],
  ["Ruby"],
  ["PostgreSQL"]
  ]
  
  select \
    --table Products \
    --filter 'query("name", "r name:Ruby", {"enough_filtered_ratio": 0.5})'
  

  table_create Products TABLE_NO_KEY
  column_create Products name COLUMN_SCALAR ShortText
  
  table_create Terms TABLE_PAT_KEY ShortText --normalizer NormalizerAuto
  column_create Terms products_name COLUMN_INDEX Products name
  
  load --table Products
  [
  ["name"],
  ["Groonga"],
  ["Mroonga"],
  ["Rroonga"],
  ["PGroonga"],
  ["Ruby"],
  ["PostgreSQL"]
  ]
  
  select \
    --table Products \
    --filter 'query("name", "r name:Ruby", {"max_n_enough_filtered_records": 10})'
  

• [between][in_values] Added support for customizing thresholds for sequential search.

  [between] and [in_values] have a feature that they switch to sequential search when the target of search records is narrowed down enough.

  The value of GRN_IN_VALUES_TOO_MANY_INDEX_MATCH_RATIO / GRN_BETWEEN_TOO_MANY_INDEX_MATCH_RATIO is used as threshold whether Groonga execute sequential search or search with indexes in such a case.

  This behavior is customized by only the following environment variable until now.

  in_values():

  # Don't use auto sequential search
  GRN_IN_VALUES_TOO_MANY_INDEX_MATCH_RATIO=-1
  # Set threshold to 0.02
  GRN_IN_VALUES_TOO_MANY_INDEX_MATCH_RATIO=0.02
  

  between():

  # Don't use auto sequential search
  GRN_BETWEEN_TOO_MANY_INDEX_MATCH_RATIO=-1
  # Set threshold to 0.02
  GRN_BETWEEN_TOO_MANY_INDEX_MATCH_RATIO=0.02
  

  if customize by the environment variable, this threshold applies to all queries, but we can specify it in each query by using this feature.

  The details are as follows. We can specify the threshold by using {"too_many_index_match_ratio": x.xx} option. The value type of this option is double.

  table_create Memos TABLE_HASH_KEY ShortText
  column_create Memos timestamp COLUMN_SCALAR Time
  
  table_create Times TABLE_PAT_KEY Time
  column_create Times memos_timestamp COLUMN_INDEX Memos timestamp
  
  load --table Memos
  [
  {"_key": "001", "timestamp": "2014-11-10 07:25:23"},
  {"_key": "002", "timestamp": "2014-11-10 07:25:24"},
  {"_key": "003", "timestamp": "2014-11-10 07:25:25"},
  {"_key": "004", "timestamp": "2014-11-10 07:25:26"},
  {"_key": "005", "timestamp": "2014-11-10 07:25:27"},
  {"_key": "006", "timestamp": "2014-11-10 07:25:28"},
  {"_key": "007", "timestamp": "2014-11-10 07:25:29"},
  {"_key": "008", "timestamp": "2014-11-10 07:25:30"},
  {"_key": "009", "timestamp": "2014-11-10 07:25:31"},
  {"_key": "010", "timestamp": "2014-11-10 07:25:32"},
  {"_key": "011", "timestamp": "2014-11-10 07:25:33"},
  {"_key": "012", "timestamp": "2014-11-10 07:25:34"},
  {"_key": "013", "timestamp": "2014-11-10 07:25:35"},
  {"_key": "014", "timestamp": "2014-11-10 07:25:36"},
  {"_key": "015", "timestamp": "2014-11-10 07:25:37"},
  {"_key": "016", "timestamp": "2014-11-10 07:25:38"},
  {"_key": "017", "timestamp": "2014-11-10 07:25:39"},
  {"_key": "018", "timestamp": "2014-11-10 07:25:40"},
  {"_key": "019", "timestamp": "2014-11-10 07:25:41"},
  {"_key": "020", "timestamp": "2014-11-10 07:25:42"},
  {"_key": "021", "timestamp": "2014-11-10 07:25:43"},
  {"_key": "022", "timestamp": "2014-11-10 07:25:44"},
  {"_key": "023", "timestamp": "2014-11-10 07:25:45"},
  {"_key": "024", "timestamp": "2014-11-10 07:25:46"},
  {"_key": "025", "timestamp": "2014-11-10 07:25:47"},
  {"_key": "026", "timestamp": "2014-11-10 07:25:48"},
  {"_key": "027", "timestamp": "2014-11-10 07:25:49"},
  {"_key": "028", "timestamp": "2014-11-10 07:25:50"},
  {"_key": "029", "timestamp": "2014-11-10 07:25:51"},
  {"_key": "030", "timestamp": "2014-11-10 07:25:52"},
  {"_key": "031", "timestamp": "2014-11-10 07:25:53"},
  {"_key": "032", "timestamp": "2014-11-10 07:25:54"},
  {"_key": "033", "timestamp": "2014-11-10 07:25:55"},
  {"_key": "034", "timestamp": "2014-11-10 07:25:56"},
  {"_key": "035", "timestamp": "2014-11-10 07:25:57"},
  {"_key": "036", "timestamp": "2014-11-10 07:25:58"},
  {"_key": "037", "timestamp": "2014-11-10 07:25:59"},
  {"_key": "038", "timestamp": "2014-11-10 07:26:00"},
  {"_key": "039", "timestamp": "2014-11-10 07:26:01"},
  {"_key": "040", "timestamp": "2014-11-10 07:26:02"},
  {"_key": "041", "timestamp": "2014-11-10 07:26:03"},
  {"_key": "042", "timestamp": "2014-11-10 07:26:04"},
  {"_key": "043", "timestamp": "2014-11-10 07:26:05"},
  {"_key": "044", "timestamp": "2014-11-10 07:26:06"},
  {"_key": "045", "timestamp": "2014-11-10 07:26:07"},
  {"_key": "046", "timestamp": "2014-11-10 07:26:08"},
  {"_key": "047", "timestamp": "2014-11-10 07:26:09"},
  {"_key": "048", "timestamp": "2014-11-10 07:26:10"},
  {"_key": "049", "timestamp": "2014-11-10 07:26:11"},
  {"_key": "050", "timestamp": "2014-11-10 07:26:12"}
  ]
  
  select Memos \
    --filter '_key == "003" && \
              between(timestamp, \
                      "2014-11-10 07:25:24", \
                      "include", \
                      "2014-11-10 07:27:26", \
                      "exclude", \
                      {"too_many_index_match_ratio": 0.03})'
  

  table_create Tags TABLE_HASH_KEY ShortText
  
  table_create Memos TABLE_HASH_KEY ShortText
  column_create Memos tag COLUMN_SCALAR Tags
  
  load --table Memos
  [
  {"_key": "Rroonga is fast!", "tag": "Rroonga"},
  {"_key": "Groonga is fast!", "tag": "Groonga"},
  {"_key": "Mroonga is fast!", "tag": "Mroonga"},
  {"_key": "Groonga sticker!", "tag": "Groonga"},
  {"_key": "Groonga is good!", "tag": "Groonga"}
  ]
  
  column_create Tags memos_tag COLUMN_INDEX Memos tag
  
  select \
    Memos \
    --filter '_id >= 3 && \
              in_values(tag, \
                       "Groonga", \
                       {"too_many_index_match_ratio": 0.7})' \
    --output_columns _id,_score,_key,tag
  

• [between] Added support for GRN_EXPR_OPTIMIZE=yes.

  between() supported for optimizing the order of evaluation of a conditional expression.

• [query_parallel_or][query] Added support for specifying group of match_columns as vector. [GitHub#1238][Patched by naoa]

  We can use vector in match_columns of query and query_parallel_or as below.

  table_create Users TABLE_NO_KEY
  column_create Users name COLUMN_SCALAR ShortText
  column_create Users memo COLUMN_SCALAR ShortText
  column_create Users tag COLUMN_SCALAR ShortText
  
  table_create Terms TABLE_PAT_KEY ShortText \
    --default_tokenizer TokenNgram \
    --normalizer NormalizerNFKC130
  column_create Terms name COLUMN_INDEX|WITH_POSITION Users name
  column_create Terms memo COLUMN_INDEX|WITH_POSITION Users memo
  column_create Terms tag COLUMN_INDEX|WITH_POSITION Users tag
  
  load --table Users
  [
  {"name": "Alice", "memo": "Groonga user", "tag": "Groonga"},
  {"name": "Bob",   "memo": "Rroonga user", "tag": "Rroonga"}
  ]
  
  select Users \
    --output_columns _score,name \
    --filter 'query(["name * 100", "memo", "tag * 10"], \
                    "Alice OR Groonga")'
  

• [select] Added support for section and weight in prefix search. [GitHub#1240][Patched by naoa]

  We can use multi column index and adjusting score in prefix search.

  table_create Memos TABLE_NO_KEY
  column_create Memos title COLUMN_SCALAR ShortText
  column_create Memos tags COLUMN_VECTOR ShortText
  
  table_create Terms TABLE_PAT_KEY ShortText
  column_create Terms index COLUMN_INDEX|WITH_SECTION Memos title,tags
  
  load --table Memos
  [
  {"title": "Groonga", "tags": ["Groonga"]},
  {"title": "Rroonga", "tags": ["Groonga", "Rroonga", "Ruby"]},
  {"title": "Mroonga", "tags": ["Groonga", "Mroonga", "MySQL"]}
  ]
  
  select Memos \
    --match_columns "Terms.index.title * 2" \
    --query 'G*' \
    --output_columns title,tags,_score
  [
    [
      0,
      0.0,
      0.0
    ],
    [
      [
        [
          1
        ],
        [
          [
            "title",
            "ShortText"
          ],
          [
            "tags",
            "ShortText"
          ],
          [
            "_score",
            "Int32"
          ]
        ],
        [
          "Groonga",
          [
            "Groonga"
          ],
          2
        ]
      ]
    ]
  ]
  

• [grndb] Added support for closing used object immediately in grndb recover.

  We can reduce memory usage by this. This may decrease performance but it will be acceptable.

  Note that grndb check doesn’t close used objects immediately yet.

• [query_parallel_or][query] Added support for specifying scorer_tf_idf in match_columns as below.

  table_create Tags TABLE_HASH_KEY ShortText
  
  table_create Users TABLE_HASH_KEY ShortText
  column_create Users tags COLUMN_VECTOR Tags
  
  load --table Users
  [
  {"_key": "Alice",
   "tags": ["beginner", "active"]},
  {"_key": "Bob",
   "tags": ["expert", "passive"]},
  {"_key": "Chris",
   "tags": ["beginner", "passive"]}
  ]
  
  column_create Tags users COLUMN_INDEX Users tags
  
  select Users \
    --output_columns _key,_score \
    --sort_keys _id \
    --command_version 3 \
    --filter 'query_parallel_or("scorer_tf_idf(tags)", \
                                "beginner active")'
  {
    "header": {
      "return_code": 0,
      "start_time": 0.0,
      "elapsed_time": 0.0
    },
    "body": {
      "n_hits": 1,
      "columns": [
        {
          "name": "_key",
          "type": "ShortText"
        },
        {
          "name": "_score",
          "type": "Float"
        }
      ],
      "records": [
        [
          "Alice",
          2.098612308502197
        ]
      ]
    }
  }
  

• [query_expand] Added support for weighted increment, decrement, and negative.

  We can specify weight against expanded words.

  If we want to increment score, we use >. If we want to decrement score, we use <.

  We can specify the quantity of scores as a number. We can also use a negative numbers in it.

  table_create TermExpansions TABLE_NO_KEY
  column_create TermExpansions term COLUMN_SCALAR ShortText
  column_create TermExpansions expansions COLUMN_VECTOR ShortText
  
  load --table TermExpansions
  [
  {"term": "Rroonga", "expansions": ["Rroonga", "Ruby Groonga"]}
  ]
  
  query_expand TermExpansions "Groonga <-0.2Rroonga Mroonga" \
    --term_column term \
    --expanded_term_column expansions
  [[0,0.0,0.0],"Groonga <-0.2((Rroonga) OR (Ruby Groonga)) Mroonga"]
  

• [httpd] Updated bundled nginx to 1.21.1.

• Updated bundled Apache Arrow to 5.0.0.

• [Ubuntu] Dropped Ubuntu 20.10 (Groovy Gorilla) support.

  • Because Ubuntu 20.10 reached EOL July 22, 2021.

Fixes¶

• [query_parallel_or][query] Fixed a bug that if we specify query_options and the other options, the other options are ignored.

  For example, "default_operator": "OR" option had been ignored in the following case.

  plugin_register token_filters/stop_word
  
  table_create Memos TABLE_NO_KEY
  column_create Memos content COLUMN_SCALAR ShortText
  
  table_create Terms TABLE_PAT_KEY ShortText \
    --default_tokenizer TokenBigram \
    --normalizer NormalizerAuto \
    --token_filters TokenFilterStopWord
  column_create Terms memos_content COLUMN_INDEX|WITH_POSITION Memos content
  column_create Terms is_stop_word COLUMN_SCALAR Bool
  
  load --table Terms
  [
  {"_key": "and", "is_stop_word": true}
  ]
  
  load --table Memos
  [
  {"content": "Hello"},
  {"content": "Hello and Good-bye"},
  {"content": "and"},
  {"content": "Good-bye"}
  ]
  
  select Memos \
    --filter 'query_parallel_or( \
                "content", \
                "Hello and", \
                {"default_operator": "OR", \
                 "options": {"TokenFilterStopWord.enable": false}})' \
    --match_escalation_threshold -1 \
    --sort_keys -_score
  [
    [
      0,
      0.0,
      0.0
    ],
    [
      [
        [
          1
        ],
        [
          [
            "_id",
            "UInt32"
          ],
          [
            "content",
            "ShortText"
          ]
        ],
        [
          2,
          "Hello and Good-bye"
        ]
      ]
    ]
  ]
  

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

• [The browser based administration tool] Currently, Groonga has a bug that a search query that is inputted to non-administration mode is sent even if we input checks to the checkbox for the administration mode of a record list.

• *< and *> only valid when we use query() the right side of filter condition. If we specify as below, *< and *> work as &&.

  • 'content @ "Groonga" *< content @ "Mroonga"'

• If we repeat that we remove any data and load them again, Groonga may not return records that should match.

Thanks¶

• naoa

Release 11.0.4 - 2021-06-29¶

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

• [The browser based administration tool] Currently, Groonga has a bug that a search query that is inputted to non-administration mode is sent even if we input checks to the checkbox for the administration mode of a record list.

• *< and *> only valid when we use query() the right side of filter condition. If we specify as below, *< and *> work as &&.

  • 'content @ "Groonga" *< content @ "Mroonga"'

Thanks¶

• Takashi Hashida

Improvements¶

• [query] Added support for ignoring TokenFilterStem by the query.

  • TokenFilterStem can search by using a stem. For example, all of develop, developing, developed and develops tokens are stemmed as develop. So we can find develop, developing and developed by develops query.

  • In this release, we are able to search without TokenFilterStem in only a specific query as below.

    plugin_register token_filters/stem
    
    table_create Memos TABLE_NO_KEY
    column_create Memos content COLUMN_SCALAR ShortText
    
    table_create Terms TABLE_PAT_KEY ShortText \
      --default_tokenizer TokenBigram \
      --normalizer NormalizerAuto \
      --token_filters 'TokenFilterStem("keep_original", true)'
    column_create Terms memos_content COLUMN_INDEX|WITH_POSITION Memos content
    
    load --table Memos
    [
    {"content": "I develop Groonga"},
    {"content": "I'm developing Groonga"},
    {"content": "I developed Groonga"}
    ]
    
    select Memos \
      --match_columns content \
      --query '"developed groonga"' \
      --query_options '{"TokenFilterStem.enable": false}'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            1
          ],
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "content",
              "ShortText"
            ]
          ],
          [
            3,
            "I developed Groonga"
          ]
        ]
      ]
    ]
    

  • This feature is useful when users want to search by a stemmed word generally but users sometimes want to search by a exact (not stemmed) word as below.

    • If Groonga returns many results when searching by a stemmed word.

    • If TokenFilterStem returns the wrong result of stemming.

    • If we want to find only records that have an exact (not stemmed) word.

• [query] Added support for ignoring TokenFilterStopWord by the query.

  • TokenFilterStopWord searched without stop word that we registered beforehand. It uses for reducing noise of search by ignoring frequently word (e.g., and, is, and so on.).

  • However, we sometimes want to search include these words only a specific query. In this release, we are able to search without TokenFilterStopWord in only a specific query as below.

    plugin_register token_filters/stop_word
    
    table_create Memos TABLE_NO_KEY
    column_create Memos content COLUMN_SCALAR ShortText
    
    table_create Terms TABLE_PAT_KEY ShortText \
      --default_tokenizer TokenBigram \
      --normalizer NormalizerAuto \
      --token_filters TokenFilterStopWord
    column_create Terms memos_content COLUMN_INDEX|WITH_POSITION Memos content
    column_create Terms is_stop_word COLUMN_SCALAR Bool
    
    load --table Terms
    [
    {"_key": "and", "is_stop_word": true}
    ]
    
    load --table Memos
    [
    {"content": "Hello"},
    {"content": "Hello and Good-bye"},
    {"content": "Good-bye"}
    ]
    
    select Memos \
      --match_columns content \
      --query "Hello and" \
      --query_options '{"TokenFilterStopWord.enable": false}' \
      --match_escalation_threshold -1 \
      --sort_keys -_score
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            1
          ],
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "content",
              "ShortText"
            ]
          ],
          [
            2,
            "Hello and Good-bye"
          ]
        ]
      ]
    ]
    

  • In the above example, we specify TokenFilterStopWord.enable by using --query-options, but we also specify it by using {"options": {"TokenFilterStopWord.enable": false}} as below.

    plugin_register token_filters/stop_word
    
    table_create Memos TABLE_NO_KEY
    column_create Memos content COLUMN_SCALAR ShortText
    
    table_create Terms TABLE_PAT_KEY ShortText \
      --default_tokenizer TokenBigram \
      --normalizer NormalizerAuto \
      --token_filters TokenFilterStopWord
    column_create Terms memos_content COLUMN_INDEX|WITH_POSITION Memos content
    column_create Terms is_stop_word COLUMN_SCALAR Bool
    
    load --table Terms
    [
    {"_key": "and", "is_stop_word": true}
    ]
    
    load --table Memos
    [
    {"content": "Hello"},
    {"content": "Hello and Good-bye"},
    {"content": "Good-bye"}
    ]
    
    select Memos \
      --filter 'query("content", \
                      "Hello and", \
                      {"options": {"TokenFilterStopWord.enable": false}})' \
      --match_escalation_threshold -1 \
      --sort_keys -_score
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            1
          ],
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "content",
              "ShortText"
            ]
          ],
          [
            2,
            "Hello and Good-bye"
          ]
        ]
      ]
    ]
    

  • This feature is useful if that Groonga can’t return results correctly if we don’t search by keywords include commonly used words (e.g., if a search for a song title, a shop name, and so on.).

• [Normalizers][NormalizerNFKC] Added a new option remove_new_line.

  • If we want to normalize the key of a table that stores data, we set a normalizer to it. However, normally, normalizers remove a new line.

  • Groonga can’t handle a key that is only a new line.

  • We can register data that is only a new line as key by this option.

• [string_slice] Added a new function string_slice(). [Github#1177][Patched by Takashi Hashida]

  • string_slice() extracts a substring of a string.

  • To enable this function, we need to register functions/string plugin.

  • We can use two different extraction methods depending on the arguments as below.

    • Extraction by position:

      plugin_register functions/string
      table_create Memos TABLE_HASH_KEY ShortText
      
      load --table Memos
      [
        {"_key": "Groonga"}
      ]
      select Memos --output_columns '_key, string_slice(_key, 2, 3)'
      [
        [
          0,
          1337566253.89858,
          0.000355720520019531
        ],
        [
          [
            [
              1
            ],
            [
              [
                "_key",
                "ShortText"
              ],
              [
                "string_slice",
                null
              ]
            ],
            [
              "Groonga",
              "oon"
            ]
          ]
        ]
      ]
      

    • Extraction by regular expression:

      plugin_register functions/string
      table_create Memos TABLE_HASH_KEY ShortText
      
      load --table Memos
      [
        {"_key": "Groonga"}
      ]
      select Memos --output_columns '_key, string_slice(_key, "(Gro+)(.*)", 2)'
      [
        [p
          0,
          1337566253.89858,
          0.000355720520019531
        ],
        [
          [
            [
              1
            ],
            [
              [
                "_key",
                "ShortText"
              ],
              [
                "string_slice",
                null
              ]
            ],
            [
              "Groonga",
              "nga"
            ]
          ]
        ]
      ]
      

• [Ubuntu] Dropped support for Ubuntu 16.04 LTS (Xenial Xerus).

• Added EditorConfig for Visual Studio. [GitHub#1191][Patched by Takashi Hashida]

  • Most settings are for Visual Studio only.

• [httpd] Updated bundled nginx to 1.20.1.

  • Contains security fix of CVE-2021-23017.

Fixes¶

• Fixed a bug that Groonga may not have returned a result of a search query if we sent many search queries when tokenizer, normalizer, or token_filters that support options were used.

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

• [The browser based administration tool] Currently, Groonga has a bug that a search query that is inputted to non-administration mode is sent even if we input checks to the checkbox for the administration mode of a record list.

Thanks¶

• Takashi Hashida

Improvements¶

• [Documentation] Removed a reference about ruby_load command. [GitHub#1172][Patched by Anthony M. Cook]

  • Because this command has already deleted.

• [Debian GNU/Linux] Added support for Debian 11(Bullseye).

• [select] Added support for --post_filter.

  • We can use post_filter to filter by filtered stage dynamic columns as below.

    table_create Items TABLE_NO_KEY
    column_create Items price COLUMN_SCALAR UInt32
    
    load --table Items
    [
    {"price": 100},
    {"price": 150},
    {"price": 200},
    {"price": 250},
    {"price": 300}
    ]
    
    select Items \
      --filter "price >= 150" \
      --columns[price_with_tax].stage filtered \
      --columns[price_with_tax].type UInt32 \
      --columns[price_with_tax].flags COLUMN_SCALAR \
      --columns[price_with_tax].value "price * 1.1" \
      --post_filter "price_with_tax <= 250"
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            2
          ],
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "price_with_tax",
              "UInt32"
            ],
            [
              "price",
              "UInt32"
            ]
          ],
          [
            2,
            165,
            150
          ],
          [
            3,
            220,
            200
          ]
        ]
      ]
    ]
    

• [select] Added support for --slices[].post_filter.

  • We can use post_filter to filter by --slices[].filter as below.

    table_create Items TABLE_NO_KEY
    column_create Items price COLUMN_SCALAR UInt32
    
    load --table Items
    [
    {"price": 100},
    {"price": 200},
    {"price": 300},
    {"price": 1000},
    {"price": 2000},
    {"price": 3000}
    ]
    
    select Items \
      --slices[expensive].filter 'price >= 1000' \
      --slices[expensive].post_filter 'price < 3000'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            6
          ],
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "price",
              "UInt32"
            ]
          ],
          [
            1,
            100
          ],
          [
            2,
            200
          ],
          [
            3,
            300
          ],
          [
            4,
            1000
          ],
          [
            5,
            2000
          ],
          [
            6,
            3000
          ]
        ],
        {
          "expensive": [
            [
              2
            ],
            [
              [
                "_id",
                "UInt32"
              ],
              [
                "price",
                "UInt32"
              ]
            ],
            [
              4,
              1000
            ],
            [
              5,
              2000
            ]
          ]
        }
      ]
    ]
    

• [select] Added support for describing expression into --sort_keys.

  • We can describe the expression into --sort_keys.

    • If nonexistent keys into expression as a --sort_keys, they are ignored and outputted warns into a log.

  • By this, for example, we can specify a value of an element of VECTOR COLUMN to --sort_keys. And we can sort a result with it.

  • We can sort a result with an element of VECTOR COLUMN even if the before version by using dynamic column. However, we can sort a result with an element of VECTOR COLUMN without using dynamic column by this feature.

    table_create Values TABLE_NO_KEY
    column_create Values numbers COLUMN_VECTOR Int32
    load --table Values
    [
    {"numbers": [127, 128, 129]},
    {"numbers": [126, 255]},
    {"numbers": [128, -254]}
    ]
    select Values --sort_keys 'numbers[1]' --output_columns numbers
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            3
          ],
          [
            [
              "numbers",
              "Int32"
            ]
          ],
          [
            [
              128,
              -254
            ]
          ],
          [
            [
              127,
              128,
              129
            ]
          ],
          [
            [
              126,
              255
            ]
          ]
        ]
      ]
    ]
    

• [Token filters] Added support for multiple token filters with options.

  • We can specify multiple token filters with options like --token_filters 'TokenFilterStopWord("column", "ignore"), TokenFilterNFKC130("unify_kana", true)'. [Github#mroonga/mroonga#399][Reported by MASUDA Kazuhiro]

• [query] Added support a dynamic column of result_set stage with complex expression.

  • Complex expression is that it needs temporary result sets internally like a following expression.

    '(true && query("name * 10", "ali", {"score_column": "ali_score"})) || \
     (true && query("name * 2", "li", {"score_column": "li_score"}))'
    

    • In the above expressions, the temporary result sets are used to store the result of evaluating the true.

    • Therefore, for example, in the following expression, we can use a value of dynamic column of result_set stage in expression. Because temporary result sets internally are needless as below expression.

      '(query("name * 10", "ali", {"score_column": "ali_score"})) || \
       (query("name * 2", "li", {"score_column": "li_score"}))'
      

  • In this release, for example, we can set a value to li_score as below. (The value of li_score had been 0 in before version. Because the second expression could not get dynamic column.)

    table_create Users TABLE_NO_KEY
    column_create Users name COLUMN_SCALAR ShortText
    
    table_create Lexicon TABLE_HASH_KEY ShortText \
      --default_tokenizer TokenBigramSplitSymbolAlphaDigit \
      --normalizer NormalizerAuto
    column_create Lexicon users_name COLUMN_INDEX|WITH_POSITION Users name
    
    load --table Users
    [
    {"name": "Alice"},
    {"name": "Alisa"},
    {"name": "Bob"}
    ]
    
    select Users \
      --columns[ali_score].stage result_set \
      --columns[ali_score].type Float \
      --columns[ali_score].flags COLUMN_SCALAR \
      --columns[li_score].stage result_set \
      --columns[li_score].type Float \
      --columns[li_score].flags COLUMN_SCALAR \
      --output_columns name,_score,ali_score,li_score \
      --filter '(true && query("name * 10", "ali", {"score_column": "ali_score"})) || \
                (true && query("name * 2", "li", {"score_column": "li_score"}))'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            2
          ],
          [
            [
              "name",
              "ShortText"
            ],
            [
              "_score",
              "Int32"
            ],
            [
              "ali_score",
              "Float"
            ],
            [
              "li_score",
              "Float"
            ]
          ],
          [
            "Alice",
            14,
            10.0,
            2.0
          ],
          [
            "Alisa",
            14,
            10.0,
            2.0
          ]
        ]
      ]
    ]
    

  • We also supported a dynamic vector column of result_set stage as below.

    table_create Users TABLE_NO_KEY
    column_create Users name COLUMN_SCALAR ShortText
    
    table_create Lexicon TABLE_HASH_KEY ShortText \
      --default_tokenizer TokenBigramSplitSymbolAlphaDigit \
      --normalizer NormalizerAuto
    column_create Lexicon users_name COLUMN_INDEX|WITH_POSITION Users name
    
    load --table Users
    [
    {"name": "Alice"},
    {"name": "Alisa"},
    {"name": "Bob"}
    ]
    
    select Users \
      --columns[tags].stage result_set \
      --columns[tags].type ShortText \
      --columns[tags].flags COLUMN_VECTOR \
      --output_columns name,tags \
      --filter '(true && query("name", "al", {"tags": ["al"], "tags_column": "tags"})) || \
                (true && query("name", "sa", {"tags": ["sa"], "tags_column": "tags"}))'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            2
          ],
          [
            [
              "name",
              "ShortText"
            ],
            [
              "tags",
              "ShortText"
            ]
          ],
          [
            "Alice",
            [
              "al"
            ]
          ],
          [
            "Alisa",
            [
              "al",
              "sa"
            ]
          ]
        ]
      ]
    ]
    

    • If we use a dynamic vector column, the storing values are appended values of each element.

• [Ubuntu] Added support for Ubuntu 21.04 (Hirsute Hippo).

• [httpd] Updated bundled nginx to 1.19.10.

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

• [The browser based administration tool] Currently, Groonga has a bug that a search query that is inputted to non-administration mode is sent even if we input checks to the checkbox for the administration mode of a record list. [Github#1186][Reported by poti]

Thanks¶

• Anthony M. Cook

• MASUDA Kazuhiro

• poti

Improvements¶

• [Debian GNU/Linux] Added support for a ARM64 package.

• [select] Added support for customizing adjust weight every key word.

  • We need to specify < or > to all keywords to adjust scores until now. Because the default adjustment of weight (6 or 4) is larger than the default score (1).

    • Therefore, for example, “A“‘s weight is 1 and “B“‘s weight is 4 in A <B. Decremented “B“‘s weight (4) is larger than not decremented “A“‘s weight (1). This is not works as expected. we need to specify >A <B to use smaller weight than “A” for “B”. “A“‘s weight is 6 and “B“‘s weight is 4 in >A <B.

  • We can customize adjustment of weight every key word by only specifying <${WEIGHT} or >${WEIGHT} to target keywords since this release. For example, “A“‘s weight is 1 and “B“‘s weight is 0.9 in A <0.1B (“B“‘s weight decrement 0.1).

  • However, note that these forms ( >${WEIGHT}..., <${WEIGHT}..., and ~${WEIGHT}... ) are incompatible.

• [select] Added support for outputting Float and Float32 value in Apache Arrow format.

  • For example, Groonga output as below.

  table_create Data TABLE_NO_KEY
  column_create Data float COLUMN_SCALAR Float
  
  load --table Data
  [
  {"float": 1.1}
  ]
  
  select Data \
    --command_version 3 \
    --output_type apache-arrow
  
    return_code: int32
    start_time: timestamp[ns]
    elapsed_time: double
    -- metadata --
    GROONGA:data_type: metadata
     return_code                    start_time       elapsed_time
    0                  0     1970-01-01T09:00:00+09:00           0.000000
    ========================================
    _id: uint32
    float: double
    -- metadata --
    GROONGA:n_hits: 1
     _id          float
    0          1       1.100000
  

• [select] Added support for getting a reference destination data via index column when we output a result.

  • Until now, Groonga had returned involuntary value when we specified output value like index_column.xxx. For example, A value of --columns[tags].value purchases.tag was ["apple",["many"]],["banana",["man"]],["cacao",["man"]] in the following example. In this case, the expected values was ["apple",["man","many"]],["banana",["man"]],["cacao",["woman"]]. In this release, we can get a correct reference destination data via index column as below.

    table_create Products TABLE_PAT_KEY ShortText
    
    table_create Purchases TABLE_NO_KEY
    column_create Purchases product COLUMN_SCALAR Products
    column_create Purchases tag COLUMN_SCALAR ShortText
    
    column_create Products purchases COLUMN_INDEX Purchases product
    
    load --table Products
    [
    {"_key": "apple"},
    {"_key": "banana"},
    {"_key": "cacao"}
    ]
    
    load --table Purchases
    [
    {"product": "apple",  "tag": "man"},
    {"product": "banana", "tag": "man"},
    {"product": "cacao",  "tag": "woman"},
    {"product": "apple",  "tag": "many"}
    ]
    
    select Products \
      --columns[tags].stage output \
      --columns[tags].flags COLUMN_VECTOR \
      --columns[tags].type ShortText \
      --columns[tags].value purchases.tag \
      --output_columns _key,tags
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            3
          ],
          [
            [
              "_key",
              "ShortText"
            ],
            [
              "tags",
              "ShortText"
            ]
          ],
          [
            "apple",
            [
              "man",
              "many"
            ]
          ],
          [
            "banana",
            [
              "man"
            ]
          ],
          [
            "cacao",
            [
              "woman"
            ]
          ]
        ]
      ]
    ]
    

• [select] Added support for specifying index column directly as a part of nested index.

  • We can search source table after filtering by using index_column.except_source_column. For example, we specify comments.content when searching in the following example. In this case, at first, this query execute full text search from content column of Commentts table, then fetch the records of Articles table which refers to already searched records of Comments table.

    table_create Articles TABLE_HASH_KEY ShortText
    
    table_create Comments TABLE_NO_KEY
    column_create Comments article COLUMN_SCALAR Articles
    column_create Comments content COLUMN_SCALAR ShortText
    
    column_create Articles content COLUMN_SCALAR Text
    column_create Articles comments COLUMN_INDEX Comments article
    
    table_create Terms TABLE_PAT_KEY ShortText \
      --default_tokenizer TokenBigram \
      --normalizer NormalizerNFKC130
    column_create Terms articles_content COLUMN_INDEX|WITH_POSITION \
      Articles content
    column_create Terms comments_content COLUMN_INDEX|WITH_POSITION \
      Comments content
    
    load --table Articles
    [
    {"_key": "article-1", "content": "Groonga is fast!"},
    {"_key": "article-2", "content": "Groonga is useful!"},
    {"_key": "article-3", "content": "Mroonga is fast!"}
    ]
    
    load --table Comments
    [
    {"article": "article-1", "content": "I'm using Groonga too!"},
    {"article": "article-3", "content": "I'm using Mroonga!"},
    {"article": "article-1", "content": "I'm using PGroonga!"}
    ]
    
    select Articles --match_columns comments.content --query groonga \
      --output_columns "_key, _score, comments.content
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            1
          ],
          [
            [
              "_key",
              "ShortText"
            ],
            [
              "_score",
              "Int32"
            ],
            [
              "comments.content",
              "ShortText"
            ]
          ],
          [
            "article-1",
            1,
            [
              "I'm using Groonga too!",
              "I'm using PGroonga!"
            ]
          ]
        ]
      ]
    ]
    

• [load] Added support for loading reference vector with inline object literal.

  • For example, we can load data like "key" : "[ { "key" : "value", ..., "key" : "value" } ]" as below.

    table_create Purchases TABLE_NO_KEY
    column_create Purchases item COLUMN_SCALAR ShortText
    column_create Purchases price COLUMN_SCALAR UInt32
    
    table_create Settlements TABLE_HASH_KEY ShortText
    column_create Settlements purchases COLUMN_VECTOR Purchases
    column_create Purchases settlements_purchases COLUMN_INDEX Settlements purchases
    
    load --table Settlements
    [
    {
      "_key": "super market",
      "purchases": [
         {"item": "apple", "price": 100},
         {"item": "milk",  "price": 200}
      ]
    },
    {
      "_key": "shoes shop",
      "purchases": [
         {"item": "sneakers", "price": 3000}
      ]
    }
    ]
    

  • It makes easier to add JSON data into reference columns by this feature.

  • Currently, this feature only support with JSON input.

• [load] Added support for loading reference vector from JSON text.

  • We can load data to reference vector from source table with JSON text as below.

    table_create Purchases TABLE_HASH_KEY ShortText
    column_create Purchases item COLUMN_SCALAR ShortText
    column_create Purchases price COLUMN_SCALAR UInt32
    
    table_create Settlements TABLE_HASH_KEY ShortText
    column_create Settlements purchases COLUMN_VECTOR Purchases
    
    column_create Purchases settlements_purchases COLUMN_INDEX Settlements purchases
    
    load --table Settlements
    [
    {
      "_key": "super market",
      "purchases": "[{\"_key\": \"super market-1\", \"item\": \"apple\", \"price\": 100}, {\"_key\": \"super market-2\", \"item\": \"milk\",  \"price\": 200}]"
    },
    {
      "_key": "shoes shop",
      "purchases": "[{\"_key\": \"shoes shop-1\", \"item\": \"sneakers\", \"price\": 3000}]"
    }
    ]
    
    dump \
      --dump_plugins no \
      --dump_schema no
    load --table Purchases
    [
    ["_key","item","price"],
    ["super market-1","apple",100],
    ["super market-2","milk",200],
    ["shoes shop-1","sneakers",3000]
    ]
    
    load --table Settlements
    [
    ["_key","purchases"],
    ["super market",["super market-1","super market-2"]],
    ["shoes shop",["shoes shop-1"]]
    ]
    
    column_create Purchases settlements_purchases COLUMN_INDEX Settlements purchases
    

  • Currently, this feature doesn’t support nested reference record.

• [Windows] Added support for UNIX epoch for time_classify_* functions.

  • Groonga handles timestamps on local time. Therefore, for example, if we input the UNIX epoch in Japan, inputting time is 9 hours behind the UNIX epoch.

  • The Windows API outputs an error when we input the time before the UNIX epoch.

  • We can use the UNIX epoch in time_classify_* functions as below in this release.

    plugin_register functions/time
    
    table_create Timestamps TABLE_PAT_KEY Time
    load --table Timestamps
    [
    {"_key": 0},
    {"_key": "2016-05-06 00:00:00.000001"},
    {"_key": "2016-05-06 23:59:59.999999"},
    {"_key": "2016-05-07 00:00:00.000000"},
    {"_key": "2016-05-07 00:00:00.000001"},
    {"_key": "2016-05-08 23:59:59.999999"},
    {"_key": "2016-05-08 00:00:00.000000"}
    ]
    
    select Timestamps \
      --sortby _id \
      --limit -1 \
      --output_columns '_key, time_classify_day_of_week(_key)'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            7
          ],
          [
            [
              "_key",
              "Time"
            ],
            [
              "time_classify_day_of_week",
              null
            ]
          ],
          [
            0.0,
            4
          ],
          [
            1462460400.000001,
            5
          ],
          [
            1462546799.999999,
            5
          ],
          [
            1462546800.0,
            6
          ],
          [
            1462546800.000001,
            6
          ],
          [
            1462719599.999999,
            0
          ],
          [
            1462633200.0,
            0
          ]
        ]
      ]
    ]
    

• [query_parallel_or] Added a new function for processing queries in parallel.

  • query_parallel_or requires Apache Arrow for processing queries in parallel. If it does not enable, query_parallel_or processes queries in sequence.

  • query_parallel_or processes combination of match_columns and query_string in parallel.

  • Syntax of query_parallel_or is as follow:

    query_parallel_or(match_columns, query_string1,
                                     query_string2,
                                     .
                                     .
                                     .
                                     query_stringN,
                                     {"option": "value", ...})
    

• [select] Added support for ignoring nonexistent sort keys.

  • Groonga had been outputted error when we specified nonexistent sort keys until now. However, Groonga ignore nonexistent sort keys since this release. (Groonga doesn’t output error.)

  • This feature implements for consistency. Because we just ignore invalid values in output_columns and most of invalid values in sort_keys.

• [select] Added support for ignoring nonexistent tables in drilldowns[].table. [GitHub#1169][Reported by naoa]

  • Groonga had been outputted error when we specified nonexistent tables in drilldowns[].table until now. However, Groonga ignore nonexistent tables in drilldowns[].table since this release. (Groonga doesn’t output error.)

  • This feature implements for consistency. Because we just ignore invalid values in output_columns and most of invalid values in sort_keys.

• [httpd] Updated bundled nginx to 1.19.8.

Fixes¶

• [reference_acquire] Fixed a bug that Groonga crash when a table’s reference is acquired and a column is added to the table before auto release is happened.

  • Because the added column’s reference isn’t acquired but it’s released on auto release.

• [Windows] Fixed a bug that one or more processes fail an output backtrace on SEGV when a new backtrace logging process starts when another backtrace logging process is running in another thread.

Known Issues¶

• Currently, Groonga has a bug that there is possible that data is corrupt when we execute many additions, delete, and update data to vector column.

Thanks¶

• naoa

Improvements¶

• [select] Added support for outputting values of scalar column and vector column via nested index.

  • The nested index is that has structure as below.

    table_create Products TABLE_PAT_KEY ShortText
    
    table_create Purchases TABLE_NO_KEY
    column_create Purchases product COLUMN_SCALAR Products
    column_create Purchases tag COLUMN_SCALAR ShortText
    
    column_create Products purchases COLUMN_INDEX Purchases product
    

  • The Products.purchases column is a index of Purchases.product column in the above example. Also, Purchases.product is a reference to Products table.

  • We had not got the correct search result when we search via nested index until now.

  • The result had been as follows until now. We can see that {"product": "apple",  "tag": "man"} is not output.

    table_create Products TABLE_PAT_KEY ShortText
    
    table_create Purchases TABLE_NO_KEY
    column_create Purchases product COLUMN_SCALAR Products
    column_create Purchases tag COLUMN_SCALAR ShortText
    
    column_create Products purchases COLUMN_INDEX Purchases product
    
    load --table Products
    [
    {"_key": "apple"},
    {"_key": "banana"},
    {"_key": "cacao"}
    ]
    
    load --table Purchases
    [
    {"product": "apple",  "tag": "man"},
    {"product": "banana", "tag": "man"},
    {"product": "cacao",  "tag": "woman"},
    {"product": "apple",  "tag": "many"}
    ]
    
    select Products \
      --output_columns _key,purchases.tag
    [
      [
        0,
        1612504193.380738,
        0.0002026557922363281
      ],
      [
        [
          [
            3
          ],
          [
            [
              "_key",
              "ShortText"
            ],
            [
              "purchases.tag",
              "ShortText"
            ]
          ],
          [
            "apple",
            "many"
          ],
          [
            "banana",
            "man"
          ],
          [
            "cacao",
            "man"
          ]
        ]
      ]
    ]
    

  • The result will be as follows from this release. We can see that {"product": "apple",  "tag": "man"} is output.

    select Products \
      --output_columns _key,purchases.tag
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            3
          ],
          [
            [
              "_key",
              "ShortText"
            ],
            [
              "purchases.tags",
              "Tags"
            ]
          ],
          [
            "apple",
            [
              [
                "man",
                "one"
              ],
              [
                "child",
                "many"
              ]
            ]
          ],
          [
            "banana",
            [
              [
                "man",
                "many"
              ]
            ]
          ],
          [
            "cacao",
            [
              [
                "woman"
              ]
            ]
          ]
        ]
      ]
    ]
    

• [Windows] Dropped support for packages of Windows version that we had cross compiled by using MinGW on Linux.

  • Because there aren’t probably many people who use that.

  • These above packages are that We had provided as below name until now.

    • groonga-x.x.x-x86.exe

    • groonga-x.x.x-x86.zip

    • groonga-x.x.x-x64.exe

    • groonga-x.x.x-x86.zip

  • From now on, we use the following packages for Windows.

    • groonga-latest-x86-vs2019-with-vcruntime.zip

    • groonga-latest-x64-vs2019-with-vcruntime.zip

  • If a system already has installed Microsoft Visual C++ Runtime Library, we suggest that we use the following packages.

    • groonga-latest-x86-vs2019.zip

    • groonga-latest-x64-vs2019.zip

Fixes¶

• Fixed a bug that there is possible that index is corrupt when Groonga executes many additions, delete, and update information in it.

  • This bug occurs when we only execute many delete information from index. However, it doesn’t occur when we only execute many additions information into index.

  • We can repair the index that is corrupt by this bug using reconstruction of it.

  • This bug doesn’t detect unless we reference the broken index. Therefore, the index in our indexes may has already broken.

  • We can use [index_column_diff] command to confirm whether the index has already been broken or not.

Improvements¶

• [select] Added support for outputting UInt64 value in Apache Arrow format.

• [select] Added support for outputting a number of hits in Apache Arrow format as below.

  -- metadata --
  GROONGA:n-hits: 10
  

• [query] Added support for optimization of “order by estimated size”.

  • Normally, we can search at high speed when we first execute a condition which number of hits is a little.

    • The “B (There are few) && A (There are many)” faster than “A (There are many) && B (There are few)”.

  • This is a known optimization. However we need reorder conditions by ourselves.

  • Groonga executes this reorder automatically by using optimization of “order by estimated size”.

  • This optimization is valid by GRN_ORDER_BY_ESTIMATED_SIZE_ENABLE=yes.

• [between] Improved performance by the following improvements.

  • Improved the accuracy of a decision whether the between() use sequential search or not.

  • Improved that we set a result of between() into a result set in bulk.

• [select] Improved performance for a prefix search.

  • For example, the performance of the following prefix search by using “*” improved.

    table_create Memos TABLE_PAT_KEY ShortText
    table_create Contents TABLE_PAT_KEY ShortText   --normalizer NormalizerAuto
    column_create Contents entries_key_index COLUMN_INDEX Memos _key
    
    load --table Memos
    [
    {"_key": "(groonga) Start to try!"},
    {"_key": "(mroonga) Installed"},
    {"_key": "(groonga) Upgraded!"}
    ]
    
    select \
      --table Memos \
      --match_columns _key \
      --query '\\(groonga\\)*'
    

• [Tokenizers][TokenMecab] Improved performance for parallel construction fo token column. [GitHub#1158][Patched by naoa]

Fixes¶

• [sub_filter] Fixed a bug that sub_filter doesn’t work in slices[].filter.

  • For example, the result of sub_filter was 0 records by the following query by this bug.

    table_create Users TABLE_HASH_KEY ShortText
    column_create Users age COLUMN_SCALAR UInt8
    
    table_create Memos TABLE_NO_KEY
    column_create Memos user COLUMN_SCALAR Users
    column_create Memos content COLUMN_SCALAR Text
    
    load --table Users
    [
    {"_key": "alice", "age": 9},
    {"_key": "bob",   "age": 29}
    ]
    
    load --table Memos
    [
    {"user": "alice", "content": "hello"},
    {"user": "bob",   "content": "world"},
    {"user": "alice", "content": "bye"},
    {"user": "bob",   "content": "yay"}
    ]
    
    select \
      --table Memos \
      --slices[adult].filter '_id > 1 && sub_filter(user, "age >= 18")'
    

• Fixed a bug that it is possible that we can’t add data and Groonga crash when we repeat much addition of data and deletion of data against a hash table.

Thanks¶

• naoa

Improvements¶

• [highlight_html] Added support for removing leading full width spaces from highlight target. [PGroonga#GitHub#155][Reported by Hung Nguyen V]

  • Until now, leading full width spaces had also included in the target of highlight as below.

    table_create Entries TABLE_NO_KEY
    column_create Entries body COLUMN_SCALAR ShortText
    
    table_create Terms TABLE_PAT_KEY ShortText --default_tokenizer TokenBigram --normalizer NormalizerAuto
    column_create Terms document_index COLUMN_INDEX|WITH_POSITION Entries body
    
    load --table Entries
    [
    {"body": "Groonga　高速！"}
    ]
    
    select Entries --output_columns \
      --match_columns body --query '高速' \
      --output_columns 'highlight_html(body)'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            1
          ],
          [
            [
              "highlight_html",
              null
            ]
          ],
          [
            "Groonga<span class=\"keyword\">　高速</span>！"
          ]
        ]
      ]
    ]
    

  • However, this is needless as the target of highlight. Therefore, in this release, highlight_html() removes leading full width spaces.

• [status] Added a new item features.

  • We can display which Groonga’s features are enabled as below.

    status --output_pretty yes
    [
      [
        0,
        0.0,
        0.0
      ],
      {
        "alloc_count": 361,
        "starttime": 1608087311,
        "start_time": 1608087311,
        "uptime": 35,
        "version": "10.1.0",
        "n_queries": 0,
        "cache_hit_rate": 0.0,
        "command_version": 1,
        "default_command_version": 1,
        "max_command_version": 3,
        "n_jobs": 0,
        "features": {
          "nfkc": true,
          "mecab": true,
          "message_pack": true,
          "mruby": true,
          "onigmo": true,
          "zlib": true,
          "lz4": false,
          "zstandard": false,
          "kqueue": false,
          "epoll": true,
          "poll": false,
          "rapidjson": false,
          "apache_arrow": false,
          "xxhash": false
        }
      }
    ]
    

• [status] Added a new item apache_arrow.

  • We can display Apache Arrow version that Groonga use as below.

  [
    [
      0,
      1608088628.440753,
      0.0006628036499023438
    ],
    {
      "alloc_count": 360,
      "starttime": 1608088617,
      "start_time": 1608088617,
      "uptime": 11,
      "version": "10.0.9-39-g5a4c6f3",
      "n_queries": 0,
      "cache_hit_rate": 0.0,
      "command_version": 1,
      "default_command_version": 1,
      "max_command_version": 3,
      "n_jobs": 0,
      "features": {
        "nfkc": true,
        "mecab": true,
        "message_pack": true,
        "mruby": true,
        "onigmo": true,
        "zlib": true,
        "lz4": true,
        "zstandard": false,
        "kqueue": false,
        "epoll": true,
        "poll": false,
        "rapidjson": false,
        "apache_arrow": true,
        "xxhash": false
      },
      "apache_arrow": {
        "version_major": 2,
        "version_minor": 0,
        "version_patch": 0,
        "version": "2.0.0"
      }
    }
  ]
  

  • This item only display when Apache Arrow is valid in Groonga.

• [Window function] Added support for processing all tables at once even if target tables straddle a shard. (experimental)

  • If the target tables straddle a shard, the window function had processed each shard until now.

    • Therefore, if we used multiple group keys in windows functions, the value of the group keys from the second had to be one kind of value.

    • However, we can use multiple kind of values for it as below by this improvement.

      plugin_register sharding
      plugin_register functions/time
      
      table_create Logs_20170315 TABLE_NO_KEY
      column_create Logs_20170315 timestamp COLUMN_SCALAR Time
      column_create Logs_20170315 price COLUMN_SCALAR UInt32
      column_create Logs_20170315 item COLUMN_SCALAR ShortText
      
      table_create Logs_20170316 TABLE_NO_KEY
      column_create Logs_20170316 timestamp COLUMN_SCALAR Time
      column_create Logs_20170316 price COLUMN_SCALAR UInt32
      column_create Logs_20170316 item COLUMN_SCALAR ShortText
      
      table_create Logs_20170317 TABLE_NO_KEY
      column_create Logs_20170317 timestamp COLUMN_SCALAR Time
      column_create Logs_20170317 price COLUMN_SCALAR UInt32
      column_create Logs_20170317 item COLUMN_SCALAR ShortText
      
      load --table Logs_20170315
      [
      {"timestamp": "2017/03/15 10:00:00", "price": 1000, "item": "A"},
      {"timestamp": "2017/03/15 11:00:00", "price":  900, "item": "A"},
      {"timestamp": "2017/03/15 12:00:00", "price":  300, "item": "B"},
      {"timestamp": "2017/03/15 13:00:00", "price":  200, "item": "B"}
      ]
      
      load --table Logs_20170316
      [
      {"timestamp": "2017/03/16 10:00:00", "price":  530, "item": "A"},
      {"timestamp": "2017/03/16 11:00:00", "price":  520, "item": "B"},
      {"timestamp": "2017/03/16 12:00:00", "price":  110, "item": "A"},
      {"timestamp": "2017/03/16 13:00:00", "price":  410, "item": "A"},
      {"timestamp": "2017/03/16 14:00:00", "price":  710, "item": "B"}
      ]
      
      load --table Logs_20170317
      [
      {"timestamp": "2017/03/17 10:00:00", "price":  800, "item": "A"},
      {"timestamp": "2017/03/17 11:00:00", "price":  400, "item": "B"},
      {"timestamp": "2017/03/17 12:00:00", "price":  500, "item": "B"},
      {"timestamp": "2017/03/17 13:00:00", "price":  300, "item": "A"}
      ]
      
      table_create Times TABLE_PAT_KEY Time
      column_create Times logs_20170315 COLUMN_INDEX Logs_20170315 timestamp
      column_create Times logs_20170316 COLUMN_INDEX Logs_20170316 timestamp
      column_create Times logs_20170317 COLUMN_INDEX Logs_20170317 timestamp
      
      logical_range_filter Logs \
        --shard_key timestamp \
        --filter 'price >= 300' \
        --limit -1 \
        --columns[offsetted_timestamp].stage filtered \
        --columns[offsetted_timestamp].type Time \
        --columns[offsetted_timestamp].flags COLUMN_SCALAR \
        --columns[offsetted_timestamp].value 'timestamp - 39600000000' \
        --columns[offsetted_day].stage filtered \
        --columns[offsetted_day].type Time \
        --columns[offsetted_day].flags COLUMN_SCALAR \
        --columns[offsetted_day].value 'time_classify_day(offsetted_timestamp)' \
        --columns[n_records_per_day_and_item].stage filtered \
        --columns[n_records_per_day_and_item].type UInt32 \
        --columns[n_records_per_day_and_item].flags COLUMN_SCALAR \
        --columns[n_records_per_day_and_item].value 'window_count()' \
        --columns[n_records_per_day_and_item].window.group_keys 'offsetted_day,item' \
        --output_columns "_id,time_format_iso8601(offsetted_day),item,n_records_per_day_and_item"
      [
        [
          0,
          0.0,
          0.0
        ],
        [
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "time_format_iso8601",
              null
            ],
            [
              "item",
              "ShortText"
            ],
            [
              "n_records_per_day_and_item",
              "UInt32"
            ]
          ],
          [
            1,
            "2017-03-14T00:00:00.000000+09:00",
            "A",
            1
          ],
          [
            2,
            "2017-03-15T00:00:00.000000+09:00",
            "A",
            2
          ],
          [
            3,
            "2017-03-15T00:00:00.000000+09:00",
            "B",
            1
          ],
          [
            1,
            "2017-03-15T00:00:00.000000+09:00",
            "A",
            2
          ],
          [
            2,
            "2017-03-16T00:00:00.000000+09:00",
            "B",
            2
          ],
          [
            4,
            "2017-03-16T00:00:00.000000+09:00",
            "A",
            2
          ],
          [
            5,
            "2017-03-16T00:00:00.000000+09:00",
            "B",
            2
          ],
          [
            1,
            "2017-03-16T00:00:00.000000+09:00",
            "A",
            2
          ],
          [
            2,
            "2017-03-17T00:00:00.000000+09:00",
            "B",
            2
          ],
          [
            3,
            "2017-03-17T00:00:00.000000+09:00",
            "B",
            2
          ],
          [
            4,
            "2017-03-17T00:00:00.000000+09:00",
            "A",
            1
          ]
        ]
      ]
      

  • This feature requires Apache Arrow 3.0.0 that is not released yet.

• Added support for sequential search against reference column.

  • This feature is only used if an index search will match many records and the current result set is enough small.

    • Because the sequential search is faster than the index search in the above case.

  • It is invalid by default.

  • It is valid if we set GRN_II_SELECT_TOO_MANY_INDEX_MATCH_RATIO_REFERENCE environment variable.

  • GRN_II_SELECT_TOO_MANY_INDEX_MATCH_RATIO_REFERENCE environment variable is a threshold to switch the sequential search from the index search.

    • For example, if we set GRN_II_SELECT_TOO_MANY_INDEX_MATCH_RATIO_REFERENCE=0.7 as below, if the number of records for the result set less than 70 % of total records, a search is executed by a sequential search.

      $ export GRN_II_SELECT_TOO_MANY_INDEX_MATCH_RATIO_REFERENCE=0.7
      
      table_create Tags TABLE_HASH_KEY ShortText
      table_create Memos TABLE_HASH_KEY ShortText
      column_create Memos tag COLUMN_SCALAR Tags
      
      load --table Memos
      [
      {"_key": "Rroonga is fast!", "tag": "Rroonga"},
      {"_key": "Groonga is fast!", "tag": "Groonga"},
      {"_key": "Mroonga is fast!", "tag": "Mroonga"},
      {"_key": "Groonga sticker!", "tag": "Groonga"},
      {"_key": "Groonga is good!", "tag": "Groonga"}
      ]
      
      column_create Tags memos_tag COLUMN_INDEX Memos tag
      
      select Memos --query '_id:>=3 tag:@Groonga' --output_columns _id,_score,_key,tag
      [
        [
          0,
          0.0,
          0.0
        ],
        [
          [
            [
              2
            ],
            [
              [
                "_id",
                "UInt32"
              ],
              [
                "_score",
                "Int32"
              ],
              [
                "_key",
                "ShortText"
              ],
              [
                "tag",
                "Tags"
              ]
            ],
            [
              4,
              2,
              "Groonga sticker!",
              "Groonga"
            ],
            [
              5,
              2,
              "Groonga is good!",
              "Groonga"
            ]
          ]
        ]
      ]
      

• [tokenizers] Added support for the token column into TokenDocumentVectorTFIDF and TokenDocumentVectorBM25.

  • If there is the token column that has the same source as the index column, these tokenizer use the token id of the token column.

    • The token column has already had data that has already finished tokenize.

    • Therefore, these tokenizer are improved performance by using a token column.

  • For example, we can use this feature by making a token column named content_tokens as below.

    table_create Memos TABLE_NO_KEY
    column_create Memos content COLUMN_SCALAR Text
    
    load --table Memos
    [
    {"content": "a b c a"},
    {"content": "c b c"},
    {"content": "b b a"},
    {"content": "a c c"},
    {"content": "a"}
    ]
    
    table_create Tokens TABLE_PAT_KEY ShortText \
      --normalizer NormalizerNFKC121 \
      --default_tokenizer TokenNgram
    column_create Tokens memos_content COLUMN_INDEX|WITH_POSITION Memos content
    
    column_create Memos content_tokens COLUMN_VECTOR Tokens content
    
    table_create DocumentVectorBM25 TABLE_HASH_KEY Tokens \
      --default_tokenizer \
        'TokenDocumentVectorBM25("index_column", "memos_content", \
                                 "df_column", "df")'
    column_create DocumentVectorBM25 df COLUMN_SCALAR UInt32
    
    column_create Memos content_feature COLUMN_VECTOR|WITH_WEIGHT|WEIGHT_FLOAT32 \
      DocumentVectorBM25 content
    
    select Memos
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            5
          ],
          [
            [
              "_id",
              "UInt32"
            ],
            [
              "content",
              "Text"
            ],
            [
              "content_feature",
              "DocumentVectorBM25"
            ],
            [
              "content_tokens",
              "Tokens"
            ]
          ],
          [
            1,
            "a b c a",
            {
              "a": 0.5095787,
              "b": 0.6084117,
              "c": 0.6084117
            },
            [
              "a",
              "b",
              "c",
              "a"
            ]
          ],
          [
            2,
            "c b c",
            {
              "c": 0.8342565,
              "b": 0.5513765
            },
            [
              "c",
              "b",
              "c"
            ]
          ],
          [
            3,
            "b b a",
            {
              "b": 0.9430448,
              "a": 0.3326656
            },
            [
              "b",
              "b",
              "a"
            ]
          ],
          [
            4,
            "a c c",
            {
              "a": 0.3326656,
              "c": 0.9430448
            },
            [
              "a",
              "c",
              "c"
            ]
          ],
          [
            5,
            "a",
            {
              "a": 1.0
            },
            [
              "a"
            ]
          ]
        ]
      ]
    ]
    

  • TokenDocumentVectorTFIDF and TokenDocumentVectorBM25 give a weight against each tokens.

    • TokenDocumentVectorTFIDF calculate a weight for token by using TF-IDF.

      • Please refer to https://radimrehurek.com/gensim/models/tfidfmodel.html about TF-IDF.

    • TokenDocumentVectorBM25 calculate a weight for token by using Okapi BM25.

      • Please refer to https://en.wikipedia.org/wiki/Okapi_BM25 about Okapi BM25.

• Improved performance when below case.

  • (column @ "value") && (column @ "value")

• [Ubuntu] Added support for Ubuntu 20.10 (Groovy Gorilla).

• [Debian GNU/Linux] Dropped stretch support.

  • It reached EOL.

• [CentOS] Dropped CentOS 6.

  • It reached EOL.

• [httpd] Updated bundled nginx to 1.19.6.

Fixes¶

• Fixed a bug that Groonga crash when we use multiple keys drilldown and use multiple accessor as below. [GitHub#1153][Patched by naoa]

  table_create Tags TABLE_PAT_KEY ShortText
  
  table_create Memos TABLE_HASH_KEY ShortText
  column_create Memos tags COLUMN_VECTOR Tags
  column_create Memos year COLUMN_SCALAR Int32
  
  load --table Memos
  [
  {"_key": "Groonga is fast!", "tags": ["full-text-search"], "year": 2019},
  {"_key": "Mroonga is fast!", "tags": ["mysql", "full-text-search"], "year": 2019},
  {"_key": "Groonga sticker!", "tags": ["full-text-search", "sticker"], "year": 2020},
  {"_key": "Rroonga is fast!", "tags": ["full-text-search", "ruby"], "year": 2020},
  {"_key": "Groonga is good!", "tags": ["full-text-search"], "year": 2020}
  ]
  
  select Memos \
    --filter '_id > 0' \
    --drilldowns[tags].keys 'tags,year >= 2020' \
    --drilldowns[tags].output_columns _key[0],_key[1],_nsubrecs
  
  select Memos \
    --filter '_id > 0' \
    --drilldowns[tags].keys 'tags,year >= 2020' \
    --drilldowns[tags].output_columns _key[1],_nsubrecs
  

• Fixed a bug that the near phrase search did not match when the same phrase occurs multiple times as below.

  table_create Entries TABLE_NO_KEY
  column_create Entries content COLUMN_SCALAR Text
  
  table_create Terms TABLE_PAT_KEY ShortText \
    --default_tokenizer TokenNgram \
    --normalizer NormalizerNFKC121
  column_create Terms entries_content COLUMN_INDEX|WITH_POSITION Entries content
  
  load --table Entries
  [
  {"content": "a x a x b x x"},
  {"content": "a x x b x"}
  ]
  
  select Entries \
    --match_columns content \
    --query '*NP2"a b"' \
    --output_columns '_score, content'
  

Thanks¶

• Hung Nguyen V

• naoa

• timgates42 [Provided the patch at GitHub#1155]

Improvements¶

• Improved performance when we specified -1 to limit.

• [reference_acquire] Added a new option --auto_release_count.

  • Groonga reduces a reference count automatically when a request reaching the number of value that is specified in --auto_release_count.

  • For example, the acquired reference of Users is released automatically after the second status is processed as below.

    reference_acquire --target_name Users --auto_release_count 2
    status # Users is still referred.
    status # Users' reference is released after this command is processed.
    

  • We can prevent a leak of the release of acquired reference by this option.

• Modify behavior when Groonga evaluated empty vector and uvector.

  • Empty vector and uvector are evaluated to false in command version 3.

    • This behavior is valid for only command version 3.

    • Note that the different behavior until now.

• [Normalizers] Added a new Normalizer NormalizerNFKC130 based on Unicode NFKC (Normalization Form Compatibility Composition) for Unicode 13.0.

• [Token filters] Added a new TokenFilter TokenFilterNFKC130 based on Unicode NFKC (Normalization Form Compatibility Composition) for Unicode 13.0.

• [select] Improved performance for "_score = column - X".

• [reference_acquire] Improved that --reference_acquire doesn’t get unnecessary reference of index column when we specify the --recursive dependent option.

  • From this release, the targets of --recursive dependent are only the target table’s key and the index column that is set to data column.

• [select] Add support for ordered near phrase search.

  • Until now, the near phrase search have only looked for records that the distance of between specified phrases near.

  • This feature look for satisfy the following conditions records.

    • If the distance of between specified phrases is near.

    • If the specified phrases are in line with specified order.

  • This feature use *ONP as an operator. (Note that the operator isn’t *NP.)

  • $ is handled as itself in the query syntax. Note that it isn’t a special character in the query syntax.

  • If we use script syntax, this feature use as below.

    table_create Entries TABLE_NO_KEY
    column_create Entries content COLUMN_SCALAR Text
    
    table_create Terms TABLE_PAT_KEY ShortText \
      --default_tokenizer 'TokenNgram("unify_alphabet", false, \
                                      "unify_digit", false)' \
      --normalizer NormalizerNFKC121
    column_create Terms entries_content COLUMN_INDEX|WITH_POSITION Entries content
    
    load --table Entries
    [
    {"content": "abcXYZdef"},
    {"content": "abebcdXYZdef"},
    {"content": "abcdef"},
    {"content": "defXYZabc"},
    {"content": "XYZabc"},
    {"content": "abc123456789def"},
    {"content": "abc12345678def"},
    {"content": "abc1de2def"}
    ]
    
    select Entries --filter 'content *ONP "abc def"' --output_columns '_score, content'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            4
          ],
          [
            [
              "_score",
              "Int32"
            ],
            [
              "content",
              "Text"
            ]
          ],
          [
            1,
            "abcXYZdef"
          ],
          [
            1,
            "abcdef"
          ],
          [
            1,
            "abc12345678def"
          ],
          [
            1,
            "abc1de2def"
          ]
        ]
      ]
    ]
    

  • If we use query syntax, this feature use as below.

    select Entries --query 'content:*ONP "abc def"' --output_columns '_score, content'
    [
      [
        0,
        0.0,
        0.0
      ],
      [
        [
          [
            4
          ],
          [
            [
              "_score",
              "Int32"
            ],
            [
              "content",
              "Text"
            ]
          ],
          [
            1,
            "abcXYZdef"
          ],
          [
            1,
            "abcdef"
          ],
          [
            1,
            "abc12345678def"
          ],
          [
            1,
            "abc1de2def"
          ]
        ]
      ]
    ]
    

• [httpd] Updated bundled nginx to 1.19.5.

Fixes¶

• [Groonga HTTP server] Fixed that Groonga HTTP server finished without waiting all woker threads finished completely.

  • Until now, Groonga HTTP server had started the process of finish after worker threads finished itself process. However, worker threads may not finish completely at this timing. Therefore, Groonga HTTP server may crashed according to timing. Because Groonga HTTP server may free area of memory that worker threads are using them yet.

Improvements¶

• [select] Added support for large drilldown keys.

  • The maximum on the key size of Groonga’s tables are 4KiB. However, if we specify multiple keys in drilldown, the size of drilldown keys may be larger than 4KiB.

    • For example, if the total size for tag key and n_like key is lager than 4KiB in the following case, the drilldown had failed.

    select Entries \
      --limit -1 \
      --output_columns tag,n_likes \
      --drilldowns[tag.n_likes].keys tag,n_likes \
      --drilldowns[tag.n_likes].output_columns _value.tag,_value.n_likes,_nsubrecs
    

    • Because the drilldown packes specifying all keys for drilldown. So, if each the size of drilldown key is large, the size of the packed drilldown keys is larger than 4KiB.

  • This feature requires xxHash .

    • However, if we install Groonga from package, we can use this feature without doing anything special. Because Groonga’s package already include xxHash .

• [select] Added support for handling as the same dynamic column even if columns refer to different tables.

  • We can’t have handled the same dynamic column if columns refer to different tables until now. Because the type of columns is different.

  • However, from this version, we can handle the same dynamic column even if columns refer to different tables by casting to built-in types as below.

    table_create Store_A TABLE_HASH_KEY ShortText
    table_create Store_B TABLE_HASH_KEY ShortText
    
    table_create Customers TABLE_HASH_KEY Int32
    column_create Customers customers_A COLUMN_VECTOR Store_A
    column_create Customers customers_B COLUMN_VECTOR Store_B
    
    load --table Customers
    [
      {"_key": 1604020694, "customers_A": ["A", "B", "C"]},
      {"_key": 1602724694, "customers_B": ["Z", "V", "Y", "T"]},
    ]
    
    select Customers \
      --filter '_key == 1604020694' \
      --columns[customers].stage output \
      --columns[customers].flags COLUMN_VECTOR \
      --columns[customers].type ShortText \
      --columns[customers].value 'customers_A' \
      --output_columns '_key, customers'
    

  • We have needed to set Store_A or Store_B in the type of customers column until now.

  • The type of customers_A column cast to ShortText in the above example.

  • By this, we can also set the value of customers_B in the value of customers column. Because both the key of customers_A and customers_B are ShortText type.

• [select] Improved performance when the number of records for search result are huge.

  • This optimization works when below cases.

    • --filter 'column <= "value"' or --filter 'column >= "value"'

    • --filter 'column == "value"'

    • --filter 'between(...)' or --filter 'between(_key, ...)'

    • --filter 'sub_filter(reference_column, ...)'

    • Comparing against _key such as --filter '_key > "value"'.

    • --filter 'geo_in_circle(...)'

• Updated bundled LZ4 to 1.9.2 from 1.8.2.

• Added support xxHash 0.8

• [httpd] Updated bundled nginx to 1.19.4.