Elastic Scaling
Doris supports online elastic scaling, allowing users to dynamically add or remove nodes without interrupting services. This capability ensures businesses can meet growing demands or reduce idle resource waste. Scaling up or down BE nodes does not affect cluster availability but involves data migration, so it is recommended to perform scaling operations during periods of low business activity.
Scale In/Out the FE Clustersβ
Doris FE nodes are divided into the following three roles, with each FE node containing a full set of metadata:
-
Master Node: Responsible for reading and writing metadata. When metadata changes occur on the Master node, they are synchronized to non-Master nodes via the BDB JE protocol. There can be only one Master FE node in the cluster.
-
Follower Node: Responsible for reading metadata. In the event of a Master node failure, Follower nodes initiate a leader election to select a new Master node. Within the cluster, the total number of Master and Follower nodes is recommended to be an odd number.
-
Observer Node: Responsible for reading metadata but does not participate in leader election. It is used to extend the read service capacity of FE nodes.
Typically, each FE node can handle the load operations of 10-20 BE nodes. A configuration of 3 FE nodes is sufficient to meet the requirements of most business scenarios.
Scale the FE outβ
When adding a new FE node, please pay attention to the following:
-
The
http_port
of the new FE node must match thehttp_port
of all existing FE nodes in the cluster. -
If adding a Follower node, it is recommended that the total number of Master and Follower nodes in the cluster be an odd number.
-
You can view the ports and roles of the current cluster nodes using the
show frontends
command.
- Start FE Node:
fe/bin/start_fe.sh --helper <leader_fe_host>:<edit_log_port> --daemon
-
Register FE Node:
-
Register the node as a Follower FE:
ALTER SYSTEM ADD FOLLOWER "<follower_host>:<edit_log_port>";
-
Register the node as a Observer FE:
ALTER SYSTEM ADD OBSERVER "<observer_host>:<edit_log_port>";
-
-
Check the status of the newly added FE node
show frontends;
Scale In the FE Clusterβ
When scaling in FE nodes, ensure that the total number of Master and Follower nodes in the cluster remains an odd number. Use the following commands to remove nodes:
ALTER SYSTEM DROP FOLLOWER[OBSERVER] "<fe_host>:<edit_log_port>";
After scaling in, you need to manually delete the FE directory.
Scale In/Out the BE Clusterβ
Scale Out the BE Clusterβ
-
Start the BE process:
be/bin/start_be.sh
-
Register the BE node:
ALTER SYSTEM ADD backend '<be_host>:<be_heartbeat_service_port>';
Scale In the BE Clusterβ
When scaling in BE nodes, you can choose between the DROP or DECOMMISSION methods:
DROP | DECOMMISSION | |
---|---|---|
Principle | Directly remove the node, deleting the BE node. | Initiates a command to migrate data on the BE node to other nodes. Once migration is complete, the BE node is automatically removed. |
Effective Time | Takes effect immediately after execution. | Takes effect after data migration is completed. Depending on the cluster's existing data volume, this can take hours to up to a day. |
Single Replica Table Handling | May result in data loss. | Does not result in data loss. |
Removing Multiple Nodes Simultaneously | May result in data loss. | Does not result in data loss. |
Production Recommendation | Not recommended for production environments. | Recommended for production environments. |
-
Use the following command to remove a BE node using the DROP method:
ALTER SYSTEM DROP backend "<be_host>:<be_heartbeat_service_port>";
-
Use the following command to remove a BE node using the DECOMMISSION method:
ALTER SYSTEM DECOMMISSION backend "<be_host>:<be_heartbeat_service_port>";
DECOMMISSION Command Description:β
-
DECOMMISSION is an asynchronous operation. After execution, you can see the BE node's
SystemDecommissioned
status set totrue
viaSHOW backends;
. This indicates the node is being removed. -
The DECOMMISSION command may fail. For instance, if there is insufficient storage space on the remaining BE nodes to accommodate the data from the BE being removed, or if the remaining nodes do not meet the minimum replication requirements, the command will not complete, and the BE will remain in a
SystemDecommissioned
state set totrue
. -
The progress of DECOMMISSION can be monitored using
SHOW PROC '/backends';
. If the operation is in progress, theTabletNum
value will decrease continuously. -
You can cancel the operation using the command
CANCEL DECOMMISSION BACKEND "be_host:be_heartbeat_service_port";
. After cancellation, the BE node will retain its current remaining data, and Doris will re-balance the load. -
The data migration rate can be adjusted by modifying the
balance_slot_num_per_path
parameter.