Compaction

Doris writes data through a structure similar to LSM-Tree, and continuously merges small files into large ordered files through compaction in the background. Compaction handles operations such as deletion and updating.

Appropriately adjusting the compaction strategy can greatly improve load and query efficiency. Doris provides the following compaction strategies for tuning:

Vertical compaction

Vertical compaction is a new compaction algorithm implemented in Doris 1.2.2, which is used to optimize compaction execution efficiency and resource overhead in large-scale and wide table scenarios. It can effectively reduce the memory overhead of compaction and improve the execution speed of compaction. The test results show that the memory consumption by vertical compaction is only 1/10 of the original compaction algorithm, and the compaction rate is increased by 15%.

In vertical compaction, merging by row is changed to merging by column group. The granularity of each merge is changed to column group, which reduces the amount of data involved in single compaction and reduces the memory usage during compaction.

BE configuration:

• enable_vertical_compaction = true will turn on vertical compaction
• vertical_compaction_num_columns_per_group = 5 The number of columns contained in each column group, by testing, the efficiency and memory usage of a group of 5 columns by default is more friendly
• vertical_compaction_max_segment_size is used to configure the size of the disk file after vertical compaction, the default value is 268435456 (bytes)

Segment compaction

Segment compaction mainly deals with the large-scale data load. Segment compaction operates during the load process and compact segments inside the job, which is different from normal compaction and vertical compaction. This mechanism can effectively reduce the number of generated segments and avoid the -238 (OLAP_ERR_TOO_MANY_SEGMENTS) errors.

The following features are provided by segment compaction:

• reduce the number of segments generated by load
• the compacting process is parallel to the load process, which will not increase the load time
• memory consumption and computing resources will increase during loading, but the increase is relatively low because it is evenly distributed throughout the long load process.
• data after segment compaction will have resource and performance advantages in subsequent queries and normal compaction.

BE configuration:

• enable_segcompaction=true turn it on.
• segcompaction_batch_size is used to configure the interval for merging. The default value 10 means that every 10 segment files will trigger a segment compaction. It is recommended to set between 10 - 30. The larger value will increase the memory usage of segment compaction.

Situations where segment compaction is recommended:

• Loading large amounts of data fails at OLAP_ ERR_ TOO_ MANY_ SEGMENTS (errcode - 238) error. Then it is recommended to turn on segment compaction to reduce the quantity of segments during the load process.
• Too many small files are generated during the load process: although the amount of loading data is reasonable, the generation of a large number of small segment files may also fail the load job because of low cardinality or memory constraints that trigger memtable to be flushed in advance. Then it is recommended to turn on this function.
• Query immediately after loading. When the load is just finished and the standard compaction has not finished, large number of segment files will affect the efficiency of subsequent queries. If the user needs to query immediately after loading, it is recommended to turn on this function.
• The pressure of normal compaction is high after loading: segment compaction evenly puts part of the pressure of normal compaction on the loading process. At this time, it is recommended to enable this function.

Situations where segment compaction is not recommended:

• When the load operation itself has exhausted memory resources, it is not recommended to use the segment compaction to avoid further increasing memory pressure and causing the load job to fail.

Refer to this link for more information about implementation and test results.

Compaction strategy

The compaction strategy determines when and which small files are merged into larger files. Doris currently offers two compaction strategies, specified by the compaction_policy parameter in the table properties.

Size-based compaction strategy

The size-based compaction strategy is the default strategy and is suitable for most scenarios.

"compaction_policy" = "size_based"

Time series compaction strategy

The time series compaction strategy is optimized for scenarios like logs and time-series data. It leverages the time locality of time-series data, merging small files written in adjacent times into larger files. Each file participates in compaction only once, reducing write amplification from repeated compaction.

"compaction_policy" = "time_series"

The time series compaction strategy is triggered when any of the following conditions are met:

• The size of unmerged files exceeds time_series_compaction_goal_size_mbytes (default 1 GB).
• The number of unmerged files exceeds time_series_compaction_file_count_threshold (default 2000).
• The time since the last compaction exceeds time_series_compaction_time_threshold_seconds (default 1 hour).

These parameters are set in the table's PROPERTIES and can be specified when creating the table or modified later using:

ALTER TABLE table_name SET("name" = "value");

Compaction concurrency control

Compaction runs in the background and consumes CPU and IO resources. The resource consumption can be controlled by adjusting the number of concurrent compaction threads.

The number of concurrent compaction threads is configured in the BE configuration file, including the following parameters:

• max_base_compaction_threads: Number of base compaction threads, default is 4.
• max_cumu_compaction_threads: Number of cumulative compaction threads, default is -1, which mean that 1 thread per disk.

Compaction memory overhead on large MOW tablets

Unique Key tables that use merge-on-write (MOW) maintain an in-memory rowid conversion map during compaction. This map associates row locations in the input rowsets with their new locations in the output rowset so that the delete bitmap can be updated. For tablets with many rows, especially Base Compaction involving large amounts of historical data, the map can consume substantial memory and cause compaction to fail when memory is insufficient.

How memory usage grows with row count

The map preallocates one entry for every physical row in the input segments being merged. Each entry records the destination segment position and row ID as two 32-bit integers, so its main storage uses approximately 8 bytes per row. You can estimate this memory as follows:

Main mapping storage ≈ Total physical input rows participating in this compaction × 8 bytes

Physical input rows include historical versions that compaction has not yet removed. Even if an old row is marked as deleted and will not be written to the output rowset, its preallocated entry still occupies memory. Therefore, the number of valid rows returned by a query cannot directly substitute for the physical input row count in this estimate. The map is retained until the compaction task finishes and is not released after each read batch.

This estimate covers only the main storage of the map. Actual compaction memory also includes container overhead, read buffers, write buffers, and indexes. There is no fixed ratio between a tablet's compressed data size and its physical row count. A narrow table or a table with a high compression ratio can therefore require a large map even when its on-disk size is relatively small.

Calculation example

Suppose the base rowset of a MOW tablet contains 1 billion physical rows, and this Base Compaction also merges incremental rowsets containing 100 million physical rows. Even if all 100 million incremental rows update existing keys and the output contains only 1 billion valid rows, the map is still allocated for the 1.1 billion physical input rows:

Physical input rows = 1 billion + 100 million = 1.1 billion
Main mapping storage ≈ 1.1 billion × 8 bytes = 8.8 × 10^9 bytes ≈ 8.2 GiB

If the same BE runs four compaction tasks with this input size concurrently, the main storage of these maps alone uses approximately 32.8 GiB. This excludes other compaction memory and memory used by queries, loads, and other tasks. This is a calculated example; actual usage depends on the input size of each task and the concurrency.

Troubleshooting and mitigation

• When creating a table or adding partitions, choose partition and bucket counts based on the expected row count as well as the compressed data size. Select bucket keys that distribute data evenly to prevent a single tablet from accumulating too many rows. See Data Bucketing. The bucket count of an existing partition cannot be changed directly; for an existing oversized tablet, consider recreating the table and migrating the data.
• When memory is limited, consider reducing Base Compaction concurrency to leave more memory available for each task. This only reduces the combined memory peak of multiple tasks on the same BE; it does not reduce the map of an individual task. If a single map already exceeds the available memory, reducing concurrency alone may still be insufficient.
• Vertical compaction, a smaller compaction_batch_size, or fewer columns in vertical_compaction_num_columns_per_group can reduce memory used for reading and processing column groups, but do not reduce the size of the complete rowid map. compaction_memory_bytes_limit is used to estimate the read batch size and does not impose a hard cap on the rowid map or the entire compaction task.