Parquet

This document introduces the support for reading and writing Parquet file formats in Doris. It applies to the following features:

• Reading and writing data in the Catalog.
• Reading data using Table Valued Functions.
• Reading data with Broker Load.
• Writing data during Export.
• Writing data with Outfile.

Supported Compression Formats

• uncompressed
• snappy
• lz4
• zstd
• gzip
• lzo
• brotli

Parameters

Timestamp and Date Compatibility (Since 4.1.5)

Since 4.1.5, Parquet reads and writes preserve the distinction between wall-clock times and instants. A logical timestamp without a time zone maps to DATETIME; a logical timestamp with isAdjustedToUTC=true maps to TIMESTAMPTZ. Neither the Catalog compatibility property nor enable_mapping_timestamp_tz=false in a file TVF can restore the DATETIME mapping for the latter.

Parquet INT96 has no time zone annotation and is inferred as DATETIME(6). Hive Catalogs and file TVFs accept the optional hive.parquet.time-zone property to specify the INT96 writer time zone. Its default is an empty string, which preserves wall-clock time. Use an IANA time zone or a UTC offset from -12:00 to +14:00. UTC and GMT are accepted; short aliases such as CST and PRC are rejected. Set it only when the writer time zone is known, for example, "hive.parquet.time-zone" = "Asia/Shanghai".

Since 4.1.5, enable_int96_timestamps defaults to false for Outfile and Export, which write logical INT64 timestamps. Set it to true only for consumers that require INT96. Hive table writes use a different compatibility choice; see Hive Catalog.

DATE files written with the old Doris encoding and containing 0000-01-01 through 0000-02-28 require migration based on their writer. Upgrading the reader alone cannot repair them. See Year-Zero DATE File Migration.

Session Variables

• enable_parquet_lazy_mat (2.1+, 3.0+)

  Controls whether the Parquet Reader enables lazy materialization. Default is true.

• hive_parquet_use_column_names (2.1.6+, 3.0.3+)

  When reading Parquet data types from Hive tables, Doris will, by default, read data from columns in the Parquet file that have the same name as the columns in the Hive table. When this variable is set to false, Doris will read data from the Parquet file based on the column order in the Hive table, regardless of column names. This is similar to the parquet.column.index.access variable in Hive. This parameter only applies to top-level column names and is ineffective for columns inside Structs.

BE Configuration

• parquet_metadata_size_limit (since 4.1.5)

  The maximum serialized footer metadata size for the native Parquet V2 Reader. Defaults to 268435456 bytes (256 MiB), must be greater than 0, and supports dynamic updates. This is separate from the Page Header limit. Exceeding it returns Parquet v2 footer size ... exceeds metadata limit .... See BE Configuration.

• enable_parquet_page_index (2.1.5+, 3.0+)

  Determines whether the Parquet Reader uses the Page Index to filter data. This is only for debugging purposes, in case the page index sometimes filters incorrect data. Default value is false.

• parquet_header_max_size_mb (2.1+, 3.0+)

  The maximum buffer size allocated when reading the Parquet Page header. Default is 1M.

• parquet_rowgroup_max_buffer_mb (2.1+, 3.0+)

  The maximum buffer size allocated when reading a Parquet Row Group. Default is 128M.

• parquet_column_max_buffer_mb (2.1+, 3.0+)

  The maximum buffer size allocated when reading a Column within a Parquet Row Group. Default is 8M.