7.3.31. logical_count

7.3.31.1. Summary

New in version 5.0.0.

logical_count is a command to count matched records even though actual records are stored into parted tables. It is useful for users because there is less need to care about maximum records of table Limitations.

Note that this feature is not matured yet, so there are some limitations.

  • Create parted tables which contains "_YYYYMMDD" postfix. It is hardcoded, so you must create tables by each day.
  • Load proper data into parted tables on your own.

7.3.31.2. Syntax

This command takes many parameters.

The required parameters are logical_table and shard_key:

logical_count logical_table
              shard_key
              [min]
              [min_border]
              [max]
              [max_border]
              [filter]

7.3.31.3. Usage

Register sharding plugin to use logical_count command in advance.

Note that logical_count is implemented as an experimental plugin, and the specification may be changed in the future.

Here is the simple example which shows how to use this feature. Let's consider to count specified logs which are stored into multiple tables.

Here is the schema and data.

Execution example:

table_create Logs_20150203 TABLE_NO_KEY
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Logs_20150203 timestamp COLUMN_SCALAR Time
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Logs_20150203 message COLUMN_SCALAR Text
# [[0, 1337566253.89858, 0.000355720520019531], true]
table_create Logs_20150204 TABLE_NO_KEY
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Logs_20150204 timestamp COLUMN_SCALAR Time
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Logs_20150204 message COLUMN_SCALAR Text
# [[0, 1337566253.89858, 0.000355720520019531], true]
table_create Logs_20150205 TABLE_NO_KEY
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Logs_20150205 timestamp COLUMN_SCALAR Time
# [[0, 1337566253.89858, 0.000355720520019531], true]
column_create Logs_20150205 message COLUMN_SCALAR Text
# [[0, 1337566253.89858, 0.000355720520019531], true]

Execution example:

load --table Logs_20150203
[
{"timestamp": "2015-02-03 23:59:58", "message": "Start"},
{"timestamp": "2015-02-03 23:59:58", "message": "Shutdown"},
{"timestamp": "2015-02-03 23:59:59", "message": "Start"},
{"timestamp": "2015-02-03 23:59:59", "message": "Shutdown"}
]
# [[0, 1337566253.89858, 0.000355720520019531], 4]
load --table Logs_20150204
[
{"timestamp": "2015-02-04 00:00:00", "message": "Start"},
{"timestamp": "2015-02-04 00:00:00", "message": "Shutdown"},
{"timestamp": "2015-02-04 00:00:01", "message": "Start"},
{"timestamp": "2015-02-04 00:00:01", "message": "Shutdown"},
{"timestamp": "2015-02-04 23:59:59", "message": "Start"},
{"timestamp": "2015-02-04 23:59:59", "message": "Shutdown"}
]
# [[0, 1337566253.89858, 0.000355720520019531], 6]
load --table Logs_20150205
[
{"timestamp": "2015-02-05 00:00:00", "message": "Start"},
{"timestamp": "2015-02-05 00:00:00", "message": "Shutdown"},
{"timestamp": "2015-02-05 00:00:01", "message": "Start"},
{"timestamp": "2015-02-05 00:00:01", "message": "Shutdown"}
]
# [[0, 1337566253.89858, 0.000355720520019531], 4]

There are three tables which are mapped each day from 2015 Feb 03 to 2015 Feb 05.

  • Logs_20150203
  • Logs_20150204
  • Logs_20150205

Then, it loads data into each table which correspond to.

Let's count logs which contains "Shutdown" in message column and the value of timestamp is "2015-02-04 00:00:00" or later.

Here is the query to achieve above purpose.

Execution example:

logical_count Logs timestamp --filter 'message == "Shutdown"' --min "2015-02-04 00:00:00" --min_border "include"
# [[0, 1337566253.89858, 0.000355720520019531], 5]

There is a well known limitation about the number of records. By sharding feature, you can overcome such limitations because such a limitation is applied per table.

Note

There is no convenient query such as PARTITIONING BY in SQL. Thus, you must create table by table_create for each tables which contains "_YYYYMMDD" postfix in table name.

7.3.31.4. Parameters

This section describes parameters of logical_count.

7.3.31.4.1. Required parameters

There are required parameters, logical_table and shard_key.

7.3.31.4.1.1. logical_table

Specifies logical table name. It means table name without "_YYYYMMDD" postfix. If you use actual table such as "Logs_20150203", "Logs_20150203" and so on, logical table name is "Logs".

7.3.31.4.1.2. shard_key

Specifies column name which is treated as shared key in each parted table.

7.3.31.4.2. Optional parameters

There are optional parameters.

7.3.31.4.2.1. min

Specifies the min value of shard_key

7.3.31.4.2.2. min_border

Specifies whether the min value of borderline must be include or not. Specify include or exclude as the value of this parameter.

7.3.31.4.2.3. max

Specifies the max value of shard_key.

7.3.31.4.2.4. max_border

Specifies whether the max value of borderline must be include or not. Specify include or exclude as the value of this parameter.

7.3.31.4.2.5. filter

7.3.31.5. Return value

TODO

[HEADER, LOGICAL_COUNT]