Tablet metadata management tool

Tablet metadata management tool

Background

In the latest version of the code, we introduced RocksDB in BE to store meta-information of tablet, in order to solve various functional and performance problems caused by storing meta-information through header file. Currently, each data directory (root path) has a corresponding RocksDB instance, in which all tablets on the corresponding root path are stored in the key-value manner.

To facilitate the maintenance of these metadata, we provide an online HTTP interface and an offline meta tool to complete related management operations.

The HTTP interface is only used to view tablet metadata online, and can be used when the BE process is running.

However, meta tool is only used for off-line metadata management operations. BE must be stopped before it can be used.

The meta tool tool is stored in the Lib / directory of BE.

Operation

View Tablet Meta

Viewing Tablet Meta information can be divided into online and offline methods

Online

Access BE's HTTP interface to obtain the corresponding Tablet Meta information:

api:

http://{host}:{port}/api/meta/header/{tablet_id}/{schema_hash}

Host: be Hostname

port: BE's HTTP port

tablet id: tablet id

schema hash: tablet schema hash

Give an example:

http://be_host:8040/api/meta/header/14156/2458238340

If the final query is successful, the Tablet Meta will be returned as json.

Offline

Get Tablet Meta on a disk based on the meta\ tool tool.

Command:

./lib/meta_tool --root_path=/path/to/root_path --operation=get_meta --tablet_id=xxx --schema_hash=xxx

root_path: The corresponding root_path path path configured in be.conf.

The result is also a presentation of Tablet Meta in JSON format.

Load header

The function of loading header is provided to realize manual migration of tablet. This function is based on Tablet Meta in JSON format, so if changes in the shard field and version information are involved, they can be changed directly in the JSON content of Tablet Meta. Then use the following commands to load.

Command:

./lib/meta_tool --operation=load_meta --root_path=/path/to/root_path --json_meta_path=path

Delete header

In order to realize the function of deleting a tablet meta from a disk of a BE. Support single delete and batch delete.

Single delete:

./lib/meta_tool --operation=delete_meta --root_path=/path/to/root_path --tablet_id=xxx --schema_hash=xxx`

Batch delete:

./lib/meta_tool --operation=batch_delete_meta --tablet_file=/path/to/tablet_file.txt

Each line in tablet_file.txt represents the information of a tablet. The format is:

root_path,tablet_id,schema_hash

Each column are separated by comma.

tablet_file example:

/output/be/data/,14217,352781111
/output/be/data/,14219,352781111
/output/be/data/,14223,352781111
/output/be/data/,14227,352781111
/output/be/data/,14233,352781111
/output/be/data/,14239,352781111

Batch delete will skip the line with incorrect tablet information format in tablet_file. And after the execution is completed, the number of successful deletions and the number of errors are displayed.

TabletMeta in Pb format

This command is to view the old file-based management PB format Tablet Meta, and to display Tablet Meta in JSON format.

Command:

./lib/meta_tool --operation=show_meta --root_path=/path/to/root_path --pb_header_path=path

Segment meta in Pb format

This command is to view the PB format segment meta, and to display segment meta in JSON format.

Command:

./meta_tool --operation=show_segment_footer --file=/path/to/segment/file