FILE

Description

The File table-valued-function (tvf) is a wrapper around table functions like S3, HDFS, and LOCAL, providing a unified interface to access file contents on different storage systems.

This function is supported since version 3.1.0.

Syntax

FILE(
    {StorageProperties},
    {FileFormatProperties}
)

• {StorageProperties}

  The StorageProperties section is used to fill in connection and authentication information related to the storage system. For details, please refer to the [Supported Storage Systems] section.

• {FileFormatProperties}

  The FileFormatProperties section is used to fill in properties related to file formats, such as CSV delimiters. For details, please refer to the [Supported File Formats] section.

Supported Storage Systems

• hdfs

• aws s3

• google cloud storage

• Alibaba Cloud OSS

• Tencent Cloud COS

• Huawei Cloud OBS

• MINIO

Supported File Formats

• Parquet

• ORC

• Text/CSV/JSON

Examples

Accessing S3 Storage

select * from file(
    "fs.s3.support" = "true",
    "uri" = "s3://bucket/file.csv",
    "s3.access_key" = "ak",
    "s3.secret_key" = "sk",
    "s3.endpoint" = "endpoint",
    "s3.region" = "region",
    "format" = "csv"
);

Accessing HDFS Storage

select * from file(
    "fs.hdfs.support" = "true",
    "uri" = "hdfs://path/to/file.csv",
    "fs.defaultFS" = "hdfs://localhost:9000",
    "hadoop.username" = "doris",
    "format" = "csv"
);

Accessing Local Storage

select * from file(
    "fs.local.support" = "true",
    "file_path" = "student.csv",
    "backend_id" = "10003",
    "format" = "csv"
);

Using desc function to View Table Structure

desc function file(
    "fs.s3.support" = "true",
    "uri" = "s3://bucket/file.csv",
    "s3.access_key" = "ak",
    "s3.secret_key" = "sk",
    "s3.endpoint" = "endpoint",
    "s3.region" = "region",
    "format" = "csv"
);