S3
This document describes the parameters required for accessing AWS S3. These parameters apply to:
- Catalog properties.
- Table Valued Function properties.
- Broker Load properties.
- Export properties.
- Outfile properties.
Parameter Overview
| Property Name | Legacy Name | Description | Default Value | Required |
|---|---|---|---|---|
| s3.endpoint | S3 service access endpoint, e.g., s3.us-east-1.amazonaws.com | None | No | |
| s3.access_key | AWS Access Key for authentication | None | No | |
| s3.secret_key | AWS Secret Key for authentication | None | No | |
| s3.region | S3 region, e.g., us-east-1. Highly recommended to configure | None | Yes | |
| s3.use_path_style | Whether to use path-style access | FALSE | No | |
| s3.connection.maximum | Maximum number of connections for high concurrency scenarios | 50 | No | |
| s3.connection.request.timeout | Request timeout in milliseconds for connection acquisition | 3000 | No | |
| s3.connection.timeout | Connection establishment timeout in milliseconds | 1000 | No | |
| s3.role_arn | Role ARN when using Assume Role mode | None | No | |
| s3.external_id | External ID used with s3.role_arn | None | No |
Authentication Configuration
Doris supports the following two methods to access S3:
- Direct Access Key and Secret Key
"s3.access_key"="your-access-key",
"s3.secret_key"="your-secret-key",
"s3.endpoint"="s3.us-east-1.amazonaws.com",
"s3.region"="us-east-1"
- Assume Role
Suitable for cross-account and temporary authorization access. Automatically obtains temporary credentials through role authorization.
"s3.role_arn"="arn:aws:iam::123456789012:role/demo-role",
"s3.external_id"="external-identifier",
"s3.endpoint"="s3.us-east-1.amazonaws.com",
"s3.region"="us-east-1"
If both Access Key and Role ARN are configured, Access Key mode takes priority.
Accessing S3 Directory Bucket
This feature is supported since version 3.1.0.
Amazon S3 Express One Zone (also known as Directory Bucket) provides higher performance but has a different endpoint format.
- Regular bucket: s3.us-east-1.amazonaws.com
- Directory Bucket: s3express-usw2-az1.us-west-2.amazonaws.com
For more available regions, refer to: AWS Official Documentation
Example:
"s3.access_key"="ak",
"s3.secret_key"="sk",
"s3.endpoint"="s3express-usw2-az1.us-west-2.amazonaws.com",
"s3.region"="us-west
