In order to save the compute and storage resources in the Doris cluster, Doris needs to reference to some other external resources to do the related work. such as spark/GPU for query, HDFS/S3 for external storage, spark/MapReduce for ETL, connect to external storage by ODBC driver. Therefore, Doris need a resource management mechanism to manage these external resources.
A resource contains basic information such as name and type. The name is globally unique. Different types of resources contain different attributes. Please refer to the introduction of each resource for details.
The creation and deletion of resources can only be performed by users own
admin permission. One resource belongs to the entire Doris cluster. Users with
admin permission can assign permission of resource to other users. Please refer to
HELP GRANT or doris document.
Operation Of Resource
There are three main commands for resource management:
drop resource and
show resources. They are to create, delete and check resources. The specific syntax of these three commands can be viewed by executing
help CMD after MySQL client connects to Doris.
This statement is used to create a resource. For details, please refer to CREATE RESOURCE.
This command can delete an existing resource. For details, see DROP RESOURCE.
This command can view the resources that the user has permission to use. For details, see SHOW RESOURCES.
Currently, Doris can support
- Spark resource: do ETL work
- ODBC resource: query and import data from external tables
The following shows how the two resources are used.
spark.master: required, currently supported yarn, spark://host:port.
spark.submit.deployMode: The deployment mode of spark. required. It supports cluster and client.
spark.hadoop.yarn.resourcemanager.address: required when master is yarn.
spark.hadoop.fs.defaultFS: required when master is yarn.
Other parameters are optional, refer to: http://spark.apache.org/docs/latest/configuration.html
If spark is used for ETL, also need to specify the following parameters:
working_dir: Directory used by ETL. Spark is required when used as an ETL resource. For example: hdfs://host:port/tmp/doris.
broker: The name of broker. Is required when spark be used as ETL resource. You need to use the
ALTER SYSTEM ADD BROKER command to complete the configuration in advance.
broker.property_key: When the broker reads the intermediate file generated by ETL, it needs the specified authentication information.
Create a spark resource named
spark0 in the yarn cluster mode.
CREATE EXTERNAL RESOURCE "spark0"
"type" = "spark",
"spark.master" = "yarn",
"spark.submit.deployMode" = "cluster",
"spark.jars" = "xxx.jar,yyy.jar",
"spark.files" = "/tmp/aaa,/tmp/bbb",
"spark.executor.memory" = "1g",
"spark.yarn.queue" = "queue0",
"spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999",
"spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000",
"working_dir" = "hdfs://127.0.0.1:10000/tmp/doris",
"broker" = "broker0",
"broker.username" = "user0",
"broker.password" = "password0"
type: Required, must be
odbc_catalog. As the type identifier of resource.
user: The user name of the external table, required.
password: The user password of the external table, required.
host: The ip address of the external table, required.
port: The port of the external table, required.
odbc_type: Indicates the type of external table. Currently, Doris supports
Oracle. In the future, it may support more databases. The ODBC external table referring to the resource is required. The old MySQL external table referring to the resource is optional.
driver: Indicates the driver dynamic library used by the ODBC external table.
The ODBC external table referring to the resource is required. The old MySQL external table referring to the resource is optional.
Create the ODBC resource of Oracle, named
CREATE EXTERNAL RESOURCE `oracle_odbc`
"type" = "odbc_catalog",
"host" = "192.168.0.1",
"port" = "8086",
"user" = "test",
"password" = "test",
"database" = "test",
"odbc_type" = "oracle",
"driver" = "Oracle 19 ODBC driver"