Groonga 9.0.2 has been released

BloGroonga

2019-04-29

Groonga 9.0.2 has been released

Groonga 9.0.2 has been released!

We provide a package for Windows made from VC++ from this release.

We also provide a package for Windows made form MinGW as in the past.

However, we will provide it made from VC++ instead of making from MinGW sooner or later.

How to install: Install

Changes

Here are important changes in this release:

column_create Added a new flag INDEX_LARGE for index column.
object_inspect Added a new statistics next_physical_segment_id and max_n_physical_segments for physical segment information.
logical_select Added support for window function over shard.
logical_range_filter Added support for window function over shard.
logical_count Added support for window function over shard.
io_flush Added a new option --recursive dependent
Fixed "unknown type name 'bool'" compilation error in some environments.
Fixed a bug that incorrect output number over Int32 by command of execute via mruby (e.g. logical_select, logical_range_filter, logical_count, etc.).

column_create Added a new flag `INDEX_LARGE` for index column.

We can make an index column has space that two times of default by this flag. However, note that it also uses two times of memory usage.

This flag useful when index target data are large. Large data must have many records (normally at least 10 millions records) and at least one of the following features.

Index targets are multiple columns
Index table has tokenizer

Here is an example to create a large index column:

  column_create \
  --table Terms \
  --name people_roles_large_index \
  --flags COLUMN_INDEX|WITH_POSITION|WITH_SECTION|INDEX_LARGE \
  --type People \
  --source roles
  [[0, 1337566253.89858, 0.000355720520019531], true]

object_inspect Added a new statistics `next_physical_segment_id` and `max_n_physical_segments` for physical segment information.

next_physical_segment_id is the ID of the segment to the inspected index column use next. That is this number shows currently the usage of the segment.

max_n_physical_segments is the max number of the segments to the inspected index column.

The max number of these statistics depend on index column size:

Index column size	The max number of segments
`INDEX_SMALL`	`2**9` (512)
`INDEX_MEDIUM`	`2**16` (65536)
`INDEX_LARGE`	`2*17 2` (262144)
Default	`2**17` (131072)

logical_select Added support for window function over shard.

We can apply the window function to over multiple tables. However, we need to align the same order for shard key and leading group key or sort key.

For example, we can apply the window function to over multiple tables as below case. Because the below example aligns the same order for shard key and leading group key.

The leading group key is price and shard key is timestamp in the below example:

  plugin_register sharding
  
  table_create Logs_20170415 TABLE_NO_KEY
  column_create Logs_20170415 timestamp COLUMN_SCALAR Time
  column_create Logs_20170415 price COLUMN_SCALAR UInt32
  column_create Logs_20170415 n_likes COLUMN_SCALAR UInt32
  
  table_create Logs_20170416 TABLE_NO_KEY
  column_create Logs_20170416 timestamp COLUMN_SCALAR Time
  column_create Logs_20170416 price COLUMN_SCALAR UInt32
  column_create Logs_20170416 n_likes COLUMN_SCALAR UInt32
  
  load --table Logs_20170415
  [
  {"timestamp": "2017/04/15 00:00:00", "n_likes": 2, "price": 100},
  {"timestamp": "2017/04/15 01:00:00", "n_likes": 1, "price": 100},
  {"timestamp": "2017/04/15 01:00:00", "n_likes": 2, "price": 200}
  ]
  
  load --table Logs_20170416
  [
  {"timestamp": "2017/04/16 10:00:00", "n_likes": 1, "price": 200},
  {"timestamp": "2017/04/16 11:00:00", "n_likes": 2, "price": 300},
  {"timestamp": "2017/04/16 11:00:00", "n_likes": 1, "price": 300}
  ]
  
  logical_select Logs \
    --shard_key timestamp \
    --columns[count].stage initial \
    --columns[count].type UInt32 \
    --columns[count].flags COLUMN_SCALAR \
    --columns[count].value 'window_count()' \
    --columns[count].window.group_keys price \
    --output_columns price,count
  [
    [
      0,
      0.0,
      0.0
    ],
    [
      [
        [
          6
        ],
        [
          [
            "price",
            "UInt32"
          ],
          [
            "count",
            "UInt32"
          ]
        ],
        [
          100,
          2
        ],
        [
          100,
          2
        ],
        [
          200,
          2
        ],
        [
          200,
          2
        ],
        [
          300,
          2
        ],
        [
          300,
          2
        ]
      ]
    ]
  ]

logical_range_filter Added support for window function over shard.

We can apply the window function to over multiple tables. However, we need to align the same order for shard key and leading group key or sort key as with logical_select.

Here is an example to apply the window function to over multiple tables by logical_range_filter:

  plugin_register sharding
  
  table_create Logs_20170415 TABLE_NO_KEY
  column_create Logs_20170415 timestamp COLUMN_SCALAR Time
  column_create Logs_20170415 price COLUMN_SCALAR UInt32
  column_create Logs_20170415 n_likes COLUMN_SCALAR UInt32
  
  table_create Logs_20170416 TABLE_NO_KEY
  column_create Logs_20170416 timestamp COLUMN_SCALAR Time
  column_create Logs_20170416 price COLUMN_SCALAR UInt32
  column_create Logs_20170416 n_likes COLUMN_SCALAR UInt32
  
  load --table Logs_20170415
  [
  {"timestamp": "2017/04/15 00:00:00", "n_likes": 2, "price": 100},
  {"timestamp": "2017/04/15 01:00:00", "n_likes": 1, "price": 100},
  {"timestamp": "2017/04/15 01:00:00", "n_likes": 2, "price": 200}
  ]
  
  load --table Logs_20170416
  [
  {"timestamp": "2017/04/16 10:00:00", "n_likes": 1, "price": 200},
  {"timestamp": "2017/04/16 11:00:00", "n_likes": 2, "price": 300},
  {"timestamp": "2017/04/16 11:00:00", "n_likes": 1, "price": 300}
  ]
  
  logical_range_filter Logs \
    --shard_key timestamp \
    --columns[count].stage initial \
    --columns[count].type UInt32 \
    --columns[count].flags COLUMN_SCALAR \
    --columns[count].value 'window_count()' \
    --columns[count].window.group_keys price \
    --output_columns price,count
  [
    [
      0,
      0.0,
      0.0
    ],
    [
      [
        [
          6
        ],
        [
          [
            "price",
            "UInt32"
          ],
          [
            "count",
            "UInt32"
          ]
        ],
        [
          100,
          2
        ],
        [
          100,
          2
        ],
        [
          200,
          2
        ],
        [
          200,
          2
        ],
        [
          300,
          2
        ],
        [
          300,
          2
        ]
      ]
    ]
  ]

logical_count Added support for window function over shard.

We can apply the window function to over multiple tables. However, we need to align the same order for shard key and leading group key or sort key as with logical_select.

Here is an example to apply the window function to over multiple tables by logical_count:

  plugin_register sharding
  
  table_create Logs_20170415 TABLE_NO_KEY
  column_create Logs_20170415 timestamp COLUMN_SCALAR Time
  column_create Logs_20170415 price COLUMN_SCALAR UInt32
  column_create Logs_20170415 n_likes COLUMN_SCALAR UInt32
  
  table_create Logs_20170416 TABLE_NO_KEY
  column_create Logs_20170416 timestamp COLUMN_SCALAR Time
  column_create Logs_20170416 price COLUMN_SCALAR UInt32
  column_create Logs_20170416 n_likes COLUMN_SCALAR UInt32
  
  load --table Logs_20170415
  [
  {"timestamp": "2017/04/15 00:00:00", "n_likes": 2, "price": 100},
  {"timestamp": "2017/04/15 01:00:00", "n_likes": 1, "price": 100},
  {"timestamp": "2017/04/15 01:00:00", "n_likes": 2, "price": 200}
  ]
  
  load --table Logs_20170416
  [
  {"timestamp": "2017/04/16 10:00:00", "n_likes": 1, "price": 200},
  {"timestamp": "2017/04/16 11:00:00", "n_likes": 2, "price": 300},
  {"timestamp": "2017/04/16 11:00:00", "n_likes": 1, "price": 300}
  ]
  
  logical_count Logs \
    --shard_key timestamp \
    --columns[count].stage initial \
    --columns[count].type UInt32 \
    --columns[count].flags COLUMN_SCALAR \
    --columns[count].value 'window_count()' \
    --columns[count].window.group_keys price \
    --filter 'count >= 1'
  [
    [
      0,
      0.0,
      0.0
    ],
    [
      4
    ]
  ]

io_flush Added a new option `--recursive dependent`

We can flush not only target object and child objects, but also related objects by this option.

The related objects are:

A referenced table
A related index column (There is source column in target TABLE_NAME)
A table of related index column (There is source column in target TABLE_NAME)

Here is an example to use this option:

  io_flush --recursive "dependent" --target_name "Users"

Conclusion

See Release 9.0.2 2019-04-29 about detailed changes since 9.0.1

Let's search by Groonga!

BloGroonga