BloGroonga

2018-02-09

Groonga 8.0.0 has been released

Groonga 8.0.0 has been released!

This is a major version up! But It keeps backward compatibility. You can upgrade to 8.0.0 without rebuilding database.

How to install: Install

Changes

Here are important changes in this release:

  • select added --drilldown_adjuster and --drilldowns[label].adjuster.

  • between Accept between() without borders.

  • Fixed a memory leak for normal hash table.

select added --drilldown_adjuster and --drilldowns[label].adjuster.

Added --drilldown_adjuster and --drilldowns[LABEL].adjuster in select arguments. You can adjust score against result of drilldown.

Specifically, below.

table_create Categories TABLE_PAT_KEY ShortText

table_create Tags TABLE_PAT_KEY ShortText
column_create Tags categories COLUMN_VECTOR|WITH_WEIGHT Categories

table_create Memos TABLE_HASH_KEY ShortText
column_create Memos tags COLUMN_VECTOR Tags

column_create Categories tags_categories COLUMN_INDEX|WITH_WEIGHT \
  Tags categories

load --table Tags
[
{"_key": "groonga", "categories": {"full-text-search": 100}},
{"_key": "mroonga", "categories": {"mysql": 100, "full-text-search": 80}},
{"_key": "ruby", "categories": {"language": 100}}
]

load --table Memos
[
{
  "_key": "Groonga is fast",
  "tags": ["groonga"]
},
{
  "_key": "Mroonga is also fast",
  "tags": ["mroonga", "groonga"]
},
{
  "_key": "Ruby is an object oriented script language",
  "tags": ["ruby"]
}
]

select Memos \
  --limit 0 \
  --output_columns _id \
  --drilldown tags \
  --drilldown_adjuster 'categories @ "full-text-search" * 2 + categories @ "mysql"' \
  --drilldown_output_columns _key,_nsubrecs,_score
[
  [
    0,
    0.0,
    0.0
  ],
  [
    [
      [
        3
      ],
      [
        [
          "_id",
          "UInt32"
        ]
      ]
    ],
    [
      [
        3
      ],
      [
        [
          "_key",
          "ShortText"
        ],
        [
          "_nsubrecs",
          "Int32"
        ],
        [
          "_score",
          "Int32"
        ]
      ],
      [
        "groonga",
        2,
        203
      ],
      [
        "mroonga",
        1,
        265
      ],
      [
        "ruby",
        1,
        0
      ]
    ]
  ]
]

In the above example, we adjust the score of records that have full-text-search or mysql in categories .

between Accept between() without borders.

From this release, max_border and min_border are now optional. If the number of arguments passed to between() is 3, the 2nd and 3rd arguments are handled as the inclusive edges.

Specifically, below.

table_create Users TABLE_HASH_KEY ShortText
column_create Users age COLUMN_SCALAR Int32

table_create Ages TABLE_PAT_KEY Int32
column_create Ages users_age COLUMN_INDEX Users age

load --table Users
[
{"_key": "alice",  "age": 17},
{"_key": "bob",    "age": 18},
{"_key": "calros", "age": 19},
{"_key": "dave",   "age": 20},
{"_key": "eric",   "age": 21}
]

select Users --filter 'between(age, 18, 20)'
[
  [
    0,
    0.0,
    0.0
  ],
  [
    [
      [
        3
      ],
      [
        [
          "_id",
          "UInt32"
        ],
        [
          "_key",
          "ShortText"
        ],
        [
          "age",
          "Int32"
        ]
      ],
      [
        2,
        "bob",
        18
      ],
      [
        3,
        "calros",
        19
      ],
      [
        4,
        "dave",
        20
      ]
    ]
  ]
]

Fixed a memory leak for normal hash table.

Fixed a bug that you sometimes can not connect to groonga just by continuing to send queries.

Conclusion

See Release 8.0.0 2018-02-09 about detailed changes since 7.1.1

Let's search by Groonga!

2018-01-29

Groonga 7.1.1 has been released

Groonga 7.1.1 has been released!

How to install: Install

Changes

Here are important changes in this release:

  • Added quorum match support.

  • filter Added custom similarity threshold support in script syntax.

  • grndb recover Added --force-lock-clear option.

  • load added surrogate pairs support in escape syntax.

  • Added environment variable to disable reducing expire.

  • logical_range_filter Added a new filter timing.

Added quorum match support.

You can use quorum match in both script syntax and query syntax. Quorum match use when fuzzy search. Matches records with tokens that exceed the setted threshold. For example, if "I have a pen" is splitted four token, matches records with any three or more of these tokens will match.

Specifically, below.

--filter column *Q${THRESHOLD} "I have a pen"

--query *Q${THRESHOLD}"I have a pen"

filter Added custom similarity threshold support in script syntax.

You can similarity retrieval with use custom similarity threshold as below. Similarity retrieval is feature for search a similar "document" as below.

--filter column *S${SIMILARITY_THRESHOLD} "document"

grndb recover Added --force-lock-clear option.

This option, grndb forces to clear locks of database, tables and data columns. You can use your database again even if locks are remained in database, tables and data columns.

If your database is broken, your database is still broken. This option just ignores locks.

Specifically, below.

 % grndb recover --force-lock-clear DB_PATH

load added surrogate pairs support in escape syntax.

You can use surrogate pairs in escape syntax in load. For example, \\uD83C\\uDF7A is processed as 🍺.

Added environment variable to disable reducing expire.

GRN_II_REDUCE_EXPIRE_ENABLE=no disables. It's enabled by default. ``

logical_range_filter Added a new filter timing.

You can executed filter again after filtered stage generated columns are generated.

Conclusion

See Release 7.1.1 2018-01-29 about detailed changes since 7.1.0

Let's search by Groonga!

2017-12-29

Groonga 7.1.0 has been released

Groonga 7.1.0 has been released!

How to install: Install

Changes

Here are important changes in this release:

  • load Improved the load's query-log format.

  • logical_count Improved the logical_count's query-log format.

  • logical_select Improve the logical_select's query-log format.

  • delete Improved the delete's query-log format.

  • Supported vector for drilldown calc target.

  • [bulk] Reduced the number of realloc().

  • Added new function index_column_source_records.

load Improved the load's query-log format

Added detail below items in the load's query-log.

  • outputs number of loaded records.
  • outputs number of error records and columns.
  • outputs number of total records.

Specifically, below.

2017-12-29 15:23:47.049299|0x7ffe8af29a50|:000000001209848 load(3): [1][2][3]
2017-12-29 15:23:47.049311|0x7ffe8af29a50|<000000001221494 rc=-22

The number in () of after the load is number of loaded records. The number in first [] is number of error columns. The number in second [] is number of error records. The number in third [] is number of total records.

logical_countImproved the logical_count's query-log format.

Added detail below items in the logical_count's query-log.

  • outputs number of count.

Specifically, below.

2017-12-29 15:25:06.068077|0x7fffedde8460|:000000001276405 count(2)
2017-12-29 15:25:06.068107|0x7fffedde8460|<000000001305264 rc=0

The number in () of after the count is number of count.

logical_select Improve the logical_select's query-log format.

Added detail below items in the logical_select's query-log.

  • log N outputs.
  • outputs plain drilldown.
  • outputs labeled drilldown.

Specifically, below.

2017-12-29 15:19:53.703472|0x7ffe0ce4e650|:000000001372833 filter(1)
2017-12-29 15:19:53.703499|0x7ffe0ce4e650|:000000001397623 select(1)[Logs_20170315]
2017-12-29 15:19:53.703796|0x7ffe0ce4e650|:000000001695440 filter(2)
2017-12-29 15:19:53.703813|0x7ffe0ce4e650|:000000001711123 select(2)[Logs_20170316]
2017-12-29 15:19:53.704024|0x7ffe0ce4e650|:000000001923225 filter(2)
2017-12-29 15:19:53.704040|0x7ffe0ce4e650|:000000001937931 select(2)[Logs_20170317]
2017-12-29 15:19:53.704198|0x7ffe0ce4e650|:000000002096788 output(5)
2017-12-29 15:19:53.704354|0x7ffe0ce4e650|<000000002253133 rc=0

The number in () of after the select is number of matched records or result of plain drilldown, result of labeled drilldown. These number's meaning is differernt by executed query. These numbers is displayed each shard. The above example have three shard. So, Three selects are displyed. At the trailing [], the table name searched is displayed.

deleteImproved the delete's query-log format.

Added detail below items in the delete's query-log.

  • outputs number of deleted and error records.
  • outputs number of rest number of records.

The number in () of after the delete is number of deleted records. The number in first [] is number of error records. The number in second [] is rest number of records.

Supported vector for drilldown calc target

You can drilldown against vector columns. As below, you can specify vecotr column in drilldown_calc_target. So, you can get min and max and sum, average with elements of vector columns.

table_create Tags TABLE_PAT_KEY ShortText

table_create Memos TABLE_HASH_KEY ShortText
column_create Memos tag COLUMN_SCALAR Tags
column_create Memos scores COLUMN_VECTOR Int64

load --table Memos
[
{"_key": "Groonga1", "tag": "Groonga", "scores": [10, 29]},
{"_key": "Groonga2", "tag": "Groonga", "scores": [20]},
{"_key": "Groonga3", "tag": "Groonga", "scores": [60, 71]},
{"_key": "Mroonga1", "tag": "Mroonga", "scores": [61, 62, 63]},
{"_key": "Mroonga2", "tag": "Mroonga", "scores": [24, 20, 16]},
{"_key": "Mroonga3", "tag": "Mroonga", "scores": [8, 5, 2]},
{"_key": "Rroonga1", "tag": "Rroonga", "scores": [3]},
{"_key": "Rroonga2", "tag": "Rroonga", "scores": [-9, 0, 9]},
{"_key": "Rroonga3", "tag": "Rroonga", "scores": [0]}
]

When you execute below query against the above table, you can get min and max and sum, average in the below groups.

  • Group with Groonga in tag.
  • Group with Mroonga in tag.
  • Group with Rroonga in tag.
select Memos \
  --limit 0 \
  --drilldowns[tag].keys tag \
  --drilldowns[tag].calc_types 'MAX, MIN, SUM, AVG' \
  --drilldowns[tag].calc_target scores \
  --drilldowns[tag].output_columns _key,_max,_min,_sum,_avg

It is the value of _key, _max, _min, _sum, _avg from left.

["Groonga", 71, 10, 190, 38.0],
["Mroonga", 63, 2, 261, 29.0],
["Rroonga", 9, -1, 3, 0.6],

[bulk] Reduced the number of realloc()

It improves performance for large output case on Windows. For example, it causes 100x faster for 100MB over output.

Added new function index_column_source_records

As below, this function gets source records of index column.

plugin_register functions/index_column

table_create Memos TABLE_HASH_KEY ShortText

table_create Terms TABLE_PAT_KEY ShortText \
  --default_tokenizer TokenBigram \
  --normalizer NormalizerAuto
column_create Terms index COLUMN_INDEX|WITH_POSITION Memos _key

load --table Memos
[
{"_key": "Groonga is a fast full text search engine."},
{"_key": "Mroonga is a MySQL storage engine based on Groonga."},
{"_key": "Rroonga is a Ruby bindings for Groonga."}
]

When you execute query in the below against the above table, you can get record which has token in registered in the Terms table.

select Terms \
  --limit -1 \
  --sort_keys _id \
  --columns[index_records].stage output \
  --columns[index_records].type Memos \
  --columns[index_records].flags COLUMN_VECTOR \
  --columns[index_records].value 'index_column_source_records("index")' \
  --output_columns '_id, _key, index_records'

The rightmost value is the result of index_column_source_records.

[ 1, "groonga", [ "Groonga is a fast full text search engine.", "Mroonga is a MySQL storage engine based on Groonga.", "Rroonga is a Ruby bindings for Groonga." ] ],
[ 2, "is", [ "Groonga is a fast full text search engine.", "Mroonga is a MySQL storage engine based on Groonga.", "Rroonga is a Ruby bindings for Groonga." ] ],
(Abbreviation)

Conclusion

See Release 7.1.0 2017-12-29 about detailed changes since 7.0.9

Let's search by Groonga!

2017-11-29

Groonga 7.0.9 has been released

Groonga 7.0.9 has been released!

How to install: Install

Changes

Here are important changes in this release:

More than 126 arguments has been supported in in_values function

In the previous version, there is a limitation about the max number of arguments in in_values.

This fact disturbs you to simplify the query by in_values which uses too many OR and == in a query.

In this release, this limitation about the max number of arguments is removed.

Dynamic columns has been supported for logical_range_filter and logical_count

Not only select and logical_select commands, but also logical_range_filter and logical_count command, dynamic columns has been supported.

logical_range_filter command is fast in contrast to logical_select command if many records are matched and requested records are small enough.

Conclusion

See Release 7.0.9 2017-11-29 about detailed changes since 7.0.8

Let's search by Groonga!

2017-10-29

Groonga 7.0.8 has been released

Groonga 7.0.8 has been released!

How to install: Install

Changes

Here are important changes in this release:

  • [Windows] Supported backtrace on crash
  • Fixed the some cases doesn't work for QUERY_NO_SYNTAX_ERROR flag
  • Supported Ubuntu 17.10 (Artful Aardvark)

[Windows] Supported backtrace on crash

This feature not only function call history but also source filename and number of lines can be displayed as much as possible. This feature makes problem solving easier.

Example of bcktrace

2017-10-29 16:27:02.371000|C| db.c:12352:0: 00000000657E9BD9: grn_table_sort(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.402000|C| window_function.c:374:0: 00000000659D08FB: grn_table_apply_window_function(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.434000|C| mrb_table.c:437:0: 00000000659EEFE0: mrb_grn_table_apply_window_function_raw(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.449000|C| ..\mruby-source\src\vm.c:1266:0: 0000000065A573BF: mrb_vm_exec(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.465000|C| ..\mruby-source\src\vm.c:821:19: 0000000065A59CA7: mrb_vm_run(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.481000|C| ..\mruby-source\src\vm.c:2619:0: 0000000065A5347C: mrb_run(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.481000|C| (unknown):-1:-1: 0000000065A5389D: mrb_funcall_with_block(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.496000|C| ..\mruby-source\src\vm.c:360:0: 0000000065A53CA9: mrb_funcall_with_block(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.512000|C| ..\mruby-source\src\vm.c:345:0: 0000000065A53E9F: mrb_funcall(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.543000|C| mrb_command.c:90:0: 00000000659E11FA: mrb_grn_command_run_wrapper(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.559000|C| command.c:199:0: 00000000657BA61C: grn_command_run(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.574000|C| expr.c:2635:0: 0000000065807B5B: grn_expr_exec(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.590000|C| ctx.c:1255:14: 00000000657BE8AC: grn_ctx_qe_exec(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.590000|C| ctx.c:1361:14: 00000000657BF36D: grn_ctx_send(): <libgroonga-0>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\libgroonga-0.dll>
2017-10-29 16:27:02.606000|C| groonga.c:402:0: 0000000000410A61: main(): <groonga>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\groonga.exe>
2017-10-29 16:27:02.606000|C| .\mingw-w64-crt\crt\crtexe.c:336:0: 00000000004013F8: __tmainCRTStartup(): <groonga>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\groonga.exe>
2017-10-29 16:27:02.606000|C| .\mingw-w64-crt\crt\crtexe.c:214:0: 000000000040151B: mainCRTStartup(): <groonga>: <c:\Users\groonga\groonga-7.0.8-x64-2017-10-29\bin\groonga.exe>
2017-10-29 16:27:02.606000|C| (unknown):-1:-1: 000000003B44168D: (unknown)(): <(unknown)>: <(unknown)>

Fixed the some cases doesn't work for QUERY_NO_SYNTAX_ERROR flag

In the previous version, QUERY_NO_SYNTAX_ERROR flag is introduced. If this flag is set, query never causes syntax error.

But there are cases that it causes an error when this flag is used with --query '( )' , --query '(+)' , and --query '~foo' . In this release, this bug was fixed.

Supported Ubuntu 17.10 (Artful Aardvark)

In this release, Ubuntu 17.10 (Artful Aardvark) is supported!

Conclusion

See Release 7.0.8 2017-10-29 about detailed changes since 7.0.7.

Let's search by Groonga!