Skip to main content
Last updated on

Data Cache

Data Cache accelerates subsequent queries of the same data by caching recently accessed data files from remote storage systems (HDFS or object storage) to local disks. In scenarios where the same data is frequently accessed, Data Cache can avoid the overhead of repeated remote data access, improving the performance and stability of query analysis on hot data.

Applicable Scenarios​

The data cache function only works for queries on Hive, Iceberg, Hudi, and Paimon tables. It has no effect on internal table queries or non-file external table queries (such as JDBC, Elasticsearch).

Whether data caching can improve query efficiency depends on multiple factors. Below are the applicable scenarios for data caching:

  • High-speed local disk

    It is recommended to use high-speed local disks, such as SSD or NVME media local disks, as the data cache directory. It is not recommended to use mechanical hard drives as the data cache directory. Essentially, the local disk's IO bandwidth and IOPS must be significantly higher than the network bandwidth and the source storage system's IO bandwidth and IOPS to bring noticeable performance improvements.

  • Sufficient cache space size

    Data caching uses the LRU strategy as the cache eviction policy. If the queried data does not have a clear distinction between hot and cold, the cached data may be frequently updated and replaced, which may reduce query performance. It is recommended to enable data caching in scenarios where the query pattern has a clear distinction between hot and cold (e.g., most queries only access today's data and rarely access historical data), and the cache space is sufficient to store hot data.

  • Unstable IO latency of remote storage

    This situation usually occurs on HDFS storage. In most enterprises, different business departments share the same HDFS, which may lead to very unstable IO latency during peak periods. In this case, if you need to ensure stable IO latency, it is recommended to enable data caching. However, the first two conditions should still be considered.

Enabling Data Cache​

The data cache function is disabled by default and needs to be enabled by setting relevant parameters in FE and BE.

BE Configuration​

First, configure the cache path information in be.conf and restart the BE node to make the configuration effective.

ParameterRequiredDescription
enable_file_cacheYesWhether to enable Data Cache, default is false
file_cache_pathYesConfiguration related to the cache directory, in JSON format.
clear_file_cacheNoDefault is false. If true, the cache directory will be cleared when the BE node restarts.

Example configuration of file_cache_path:

file_cache_path=[{"path": "/path/to/file_cache1", "total_size":53687091200},{"path": "/path/to/file_cache2", "total_size":53687091200},{"path": "/path/to/file_cache3", "total_size":53687091200}]

path is the path where the cache is stored, and one or more paths can be configured. It is recommended to configure only one path per disk.

total_size is the upper limit of the cache space size, in bytes. When the cache space is exceeded, the LRU strategy will be used to evict cached data.

FE Configuration​

Enable Data Cache in a single session:

SET enable_file_cache = true;

Enable Data Cache globally:

SET GLOBAL enable_file_cache = true;

Note that if enable_file_cache is not enabled, the cache will not be used even if the BE is configured with a cache directory. Similarly, if the BE is not configured with a cache directory, the cache will not be used even if enable_file_cache is enabled.

Cache Observability​

View Cache Hit Rate​

Execute set enable_profile=true to open the session variable, and you can view the job's Profile on the Queries tab of the FE web page. The data cache-related metrics are as follows:

-  FileCache:  0ns
    -  BytesScannedFromCache:  2.02  GB
    -  BytesScannedFromRemote:  0.00  
    -  BytesWriteIntoCache:  0.00  
    -  LocalIOUseTimer:  2s723ms
    -  NumLocalIOTotal:  444
    -  NumRemoteIOTotal:  0
    -  NumSkipCacheIOTotal:  0
    -  RemoteIOUseTimer:  0ns
    -  WriteCacheIOUseTimer:  0ns

  • BytesScannedFromCache: The amount of data read from the local cache.

  • BytesScannedFromRemote: The amount of data read from the remote.

  • BytesWriteIntoCache: The amount of data written into the cache.

  • LocalIOUseTimer: The IO time of the local cache.

  • RemoteIOUseTimer: The IO time of remote reading.

  • NumLocalIOTotal: The number of IO operations on the local cache.

  • NumRemoteIOTotal: The number of remote IO operations.

  • WriteCacheIOUseTimer: The IO time of writing into the cache.

If BytesScannedFromRemote is 0, it means the cache is fully hit.

Monitoring Metrics​

Users can view cache statistics for each Backend node through the system table file_cache_statistics.

Cache Query Limit​

Doris provides two independent query-level Data Cache controls:

ControlPrimary parameterBehavior after the limit
Limit by cache footprint percentagefile_cache_query_limit_percentCache-miss fills remain allowed, while BE releases space through the query-level LRU record and other cache queues
Stop remote-scan cache writes by byte thresholdfile_cache_query_limit_bytesWhen the next cache block would make admitted bytes exceed the threshold, later misses are read from remote storage without filling the local cache

Limit by Cache Footprint Percentage​

This feature is supported since version 4.0.3.

The cache-footprint percentage limit controls the maximum percentage of each File Cache instance that a single query can use. When multiple users or complex queries share cache resources, it reduces the risk that one large query retains too much cache and evicts other hot data.

This feature involves configuration on BE and FE, as well as session variable settings.

1. BE Configuration

  • enable_file_cache_query_limit:
    • Type: Boolean
    • Default: false
    • Description: The master switch for the file cache query limit feature on the BE side. Only when enabled will the BE process the query limit parameters passed from the FE.

2. FE Configuration

  • file_cache_query_limit_max_percent:
    • Type: Integer
    • Default: 100
    • Description: The maximum query limit constraint used to validate the upper bound of the session variable.

3. Session Variables

  • file_cache_query_limit_percent:
    • Type: Integer (1-100)
    • Default: -1
    • Description: The maximum percentage of cache that a query can use. This value is constrained by file_cache_query_limit_max_percent. The calculated cache quota should be at least 256 MB; otherwise, BE writes a warning to the log.

Usage Example

-- Set the session variable to limit a query to at most 50% of the cache
SET file_cache_query_limit_percent = 50;

-- Execute the query
SELECT * FROM large_table;

Notes:

  1. The value must be in [1, file_cache_query_limit_max_percent].
  2. Before using this control, enable both enable_file_cache and enable_file_cache_query_limit on the BE, and ensure that enable_file_cache is true in the query session.
  3. Cache misses remain eligible for Data Cache writes after the percentage quota is reached. This control does not switch the query into a no-more-write mode.

Stop Remote-Scan Cache Writes by Byte Threshold​

This feature is supported only in Doris 4.1.x. It is not supported in Doris 4.0.x.

In compute-storage decoupled mode, the file_cache_query_limit_bytes session variable limits cumulative bytes admitted for Data Cache fills caused by Hive, Iceberg, Hudi, or Paimon data-file cache misses for one SELECT query on each BE. It requires enable_file_cache=true on the BE but does not depend on enable_file_cache_query_limit or file_cache_query_limit_max_percent.

ParameterTypeDefaultDescription
file_cache_query_limit_bytesBigInt-1Remote-scan cache-fill threshold for one query on each BE, in bytes. A value below 0 disables the control; 0 disables cache fills from the start of the query; a positive value accumulates admitted bytes by cache block

Parallel scanners for the same query share one threshold on a BE. The threshold is not a query-wide or cluster-wide total. When the next cache block would make admitted bytes exceed the threshold, the query enters remote-only-on-miss mode on that BE: a request range fully covered by local cache can still be read locally; a range that is not fully covered is read directly from remote storage without filling Data Cache. The query result is unchanged.

The following example sets a 1 GiB remote-scan cache-fill threshold on each BE:

SET enable_profile = true;
SET profile_level = 2;
SET file_cache_query_limit_bytes = 1073741824;

SELECT COUNT(*) FROM hive_catalog.sales.orders;

To prevent a one-time scan from writing Data Cache from the start, set the threshold to 0, and restore the default after the query:

SET file_cache_query_limit_bytes = 0;
SELECT COUNT(*) FROM hive_catalog.sales.orders;

SET file_cache_query_limit_bytes = -1;

After enabling Query Profile, inspect RemoteOnlyOnMissTriggered and RemoteOnlyOnMissThresholdBytes in the Scanner's FileCache metric group. When RemoteOnlyOnMissTriggered=1, the Scanner observed the query entering remote-only-on-miss mode. Use BytesWriteIntoCache and NumSkipCacheIOTotal to inspect actual writes and skipped-cache I/O. Because NumSkipCacheIOTotal can also include I/O skipped by other cache policies, it is not sufficient by itself to prove that the threshold was triggered.

note
  • This control applies only to Data Cache fills after SELECT query misses in compute-storage decoupled mode. It does not limit remote bytes read or affect explicit cache warmup. Remote-only-on-miss mode may increase remote I/O and query latency.
  • Admission is evaluated by cache block. If the remaining budget is smaller than the next block, the entire block is skipped. RemoteOnlyOnMissTriggered changes to 1 only when a block would exceed the threshold.
  • The BE parameter enable_file_cache_query_limit_segment_meta controls whether Doris internal-table Segment footer and Segment metadata writes count toward the same threshold. Cache fills for Hive, Iceberg, Hudi, and Paimon data files are always subject to file_cache_query_limit_bytes. For the complete parameter scope and internal-table behavior, see File Cache Configuration and Usage Guide.

Cache Warmup​

Data Cache provides a cache "warmup" feature that allows preloading external data into the local cache of BE nodes, thereby improving cache hit rates and query performance for subsequent first-time queries.

This feature is supported since version 4.0.2.

Syntax​

WARM UP SELECT <select_expr_list>
FROM <table_reference>
[WHERE <boolean_expression>]

Usage restrictions:

  • Supported:

    • Single table queries (only one table_reference allowed)
    • Simple SELECT for specified columns
    • WHERE filtering (supports regular predicates)

  • Not supported:

    • JOIN, UNION, subqueries, CTE
    • GROUP BY, HAVING, ORDER BY
    • LIMIT
    • INTO OUTFILE
    • Multi-table / complex query plans
    • Other complex syntax

Examples​

  1. Warm up the entire table
WARM UP SELECT * FROM hive_db.tpch100_parquet.lineitem;
  1. Warm up partial columns by partition
WARM UP SELECT l_orderkey, l_shipmode
FROM hive_db.tpch100_parquet.lineitem
WHERE dt = '2025-01-01';
  1. Warm up partial columns by filter conditions
WARM UP SELECT l_shipmode, l_linestatus
FROM hive_db.tpch100_parquet.lineitem
WHERE l_orderkey = 123456;

Execution Results​

After executing WARM UP SELECT, the FE dispatches tasks to each BE. The BE scans remote data and writes it to Data Cache.

The system directly returns scan and cache write statistics for each BE (Note: Statistics are generally accurate but may have some margin of error). For example:

+---------------+-----------+-------------+---------------------------+----------------------------+---------------------+
| BackendId     | ScanRows  | ScanBytes   | ScanBytesFromLocalStorage | ScanBytesFromRemoteStorage | BytesWriteIntoCache |
+---------------+-----------+-------------+---------------------------+----------------------------+---------------------+
| 1755134092928 | 294744184 | 11821864798 | 538154009                 | 11283717130                | 11899799492         |
| 1755134092929 | 305293718 | 12244439301 | 560970435                 | 11683475207                | 12332861380         |
| TOTAL         | 600037902 | 24066304099 | 1099124444                | 22967192337                | 24232660872         |
+---------------+-----------+-------------+---------------------------+----------------------------+---------------------+

Field explanations:

  • ScanRows: Number of rows scanned and read.
  • ScanBytes: Amount of data scanned and read.
  • ScanBytesFromLocalStorage: Amount of data scanned and read from local cache.
  • ScanBytesFromRemoteStorage: Amount of data scanned and read from remote storage.
  • BytesWriteIntoCache: Amount of data written to Data Cache during this warmup.

Cache Admission Control​

This is an experimental feature and is supported since version 4.1.0.

The cache admission control feature provides a mechanism that allows users to control whether data read by a query is allowed to enter the File Cache (Data Cache) based on dimensions such as User, Catalog, Database, and Table. In scenarios with massive cold data reads (e.g., large-scale ETL jobs or heavy ad-hoc queries), if all read data is allowed to enter the cache, it may cause existing hot data to be frequently evicted (i.e., "cache pollution"), leading to a drop in cache hit rates and overall query performance. When enabled, data denied admission will be pulled directly from remote underlying storage (e.g., HDFS, S3), effectively protecting core hot data from being swapped out.

The cache admission control feature is disabled by default and needs to be enabled by configuring relevant parameters in the FE.

FE Configuration​

You need to enable this feature and specify the rule configuration file path in fe.conf, then restart the FE node for it to take effect. Modifications to the rule files themselves can be loaded dynamically.

ParameterRequiredDescription
enable_file_cache_admission_controlYesWhether to enable cache admission control. Default is false.
file_cache_admission_control_json_dirYesThe directory path for storing admission rules JSON files. All .json files in this directory will be automatically loaded, and any rule additions, deletions, or modifications will take effect dynamically.

Admission Rules Configuration Format​

Rule configurations are placed in .json files under the file_cache_admission_control_json_dir directory. The file content must be in a JSON array format.

Field Description​

Field NameTypeDescriptionExample
idLongRule ID.1
user_identityStringUser identity (format: user@host, e.g., % matches all IPs). Leaving it empty or omitting it matches all users globally."root@%"
catalog_nameStringCatalog name. Leaving it empty or omitting it matches all catalogs."hive_cat"
database_nameStringDatabase name. Leaving it empty or omitting it matches all databases."db1"
table_nameStringTable name. Leaving it empty or omitting it matches all tables."tbl1"
partition_patternString(Not implemented yet) Partition regular expression. Empty means matching all partitions.""
rule_typeIntegerRule type: 0 means deny cache (blacklist); 1 means allow cache (whitelist).0
enabledIntegerWhether the current rule is enabled: 0 means disabled; 1 means enabled.1
created_timeLongCreation time (UNIX timestamp, seconds).1766557246
updated_timeLongUpdate time (UNIX timestamp, seconds).1766557246

JSON File Example​

[
  {
    "id": 1,
    "user_identity": "root@%",
    "catalog_name": "hive_cat",
    "database_name": "db1",
    "table_name": "table1",
    "partition_pattern": "",
    "rule_type": 0,
    "enabled": 1,
    "created_time": 1766557246,
    "updated_time": 1766557246
  },
  {
    "id": 2,
    "user_identity": "",
    "catalog_name": "hive_cat",
    "database_name": "",
    "table_name": "",
    "partition_pattern": "",
    "rule_type": 1,
    "enabled": 1,
    "created_time": 1766557246,
    "updated_time": 1766557246
  }
]

Import Rules from MySQL​

For users with automated system integration needs, an auxiliary script is provided at tools/export_mysql_rule_to_json.sh in the Doris source code repository. This script can be used to export cache admission rules pre-stored in a MySQL database table into JSON configuration files that comply with the above format.

Rule Matching Principles​

Rule Scope Categories​

By combining different fields (user_identity, catalog_name, database_name, table_name) as either empty or specific values, the system supports 7 dimensions of valid rules. Any rule configuration that does not comply with hierarchical dependencies (for example, skipping Database to specify Table directly) will be considered invalid.

user_identitycatalog_namedatabase_nametable_nameLevel and Scope
SpecifiedSpecifiedSpecifiedSpecifiedTable-level rule for specified user
Empty or OmittedSpecifiedSpecifiedSpecifiedTable-level rule for all users
SpecifiedSpecifiedSpecifiedEmpty or OmittedDatabase-level rule for specified user
Empty or OmittedSpecifiedSpecifiedEmpty or OmittedDatabase-level rule for all users
SpecifiedSpecifiedEmpty or OmittedEmpty or OmittedCatalog-level rule for specified user
Empty or OmittedSpecifiedEmpty or OmittedEmpty or OmittedCatalog-level rule for all users
SpecifiedEmpty or OmittedEmpty or OmittedEmpty or OmittedGlobal rule for specified user

Matching Priority and Order​

When a query accesses a table's data, the system comprehensively evaluates all rules to make an admission decision. The judgment process follows these principles:

  1. Exact Match First: Matching is conducted in order from specific to broad hierarchy (Table → Database → Catalog → Global). Once successfully matched at the most precise level (e.g., Table level), the judgment terminates immediately.
  2. Blacklist First (Security Principle): Within the same rule level, deny cache rules always take precedence over allow cache rules. If both a blacklist and a whitelist are matched simultaneously, the blacklist operation executes first, ensuring that access denial decisions at the same level take effect first.

The complete decision derivation sequence is as follows:

1. Table-level rule matching
   a) Hit Blacklist (rule_type=0) -> Deny
   b) Hit Whitelist (rule_type=1) -> Allow
2. Database-level rule matching
   ...
3. Catalog-level rule matching
   ...
4. Global rule matching (only user_identity matched)
   ...
5. Default fallback decision: If no rules at any level above match, caching is [Denied] by default (equivalent to a global blacklist).

Tip: Because the system's fallback strategy is default-deny, best practice when deploying this feature is generally to establish a broad global allowance rule (e.g., a whitelist for all users, or for an important business Catalog), and then configure targeted Table-level blacklists for large tables known to undergo offline full-table scans. This achieves refined separation of cold and hot data.

Cache Decision Observability​

After successfully enabling and applying the configuration, users can view detailed cache admission decisions at the file data level for a single table via the EXPLAIN command (refer to the file cache request output below).

|   0:VHIVE_SCAN_NODE(74)                                                                                          |
|      table: test_file_cache_features.tpch1_parquet.lineitem                                                      |
|      inputSplitNum=10, totalFileSize=205792918, scanRanges=10                                                    |
|      partition=1/1                                                                                               |
|      cardinality=1469949, numNodes=1                                                                             |
|      pushdown agg=NONE                                                                                           |
|      file cache request ADMITTED: user_identity:root@%, reason:user table-level whitelist rule, cost:0.058 ms    |
|      limit: 1                                                                                                    |

Key fields and decision descriptions:

  • ADMITTED / DENIED: Represents whether the request is allowed (ADMITTED) or rejected (DENIED) from entering the cache.
  • user_identity: The user identity verified during the execution of this query.
  • reason: The specific decision reason (the matched rule) that triggered the result. Common outputs include: user table-level whitelist rule (Current example: Table-level whitelist for a specified user); common table-level blacklist rule (Table-level blacklist for all users). The format is generally [Scope] [Rule Level] [Rule Type] rule.
  • cost: The time cost to complete the entire admission matching calculation (in milliseconds). If the overhead is too large, it can be optimized by adjusting the number of rule hierarchies.

Appendix​

Principle​

Data caching caches accessed remote data to the local BE node. The original data file is split into Blocks based on the accessed IO size, and Blocks are stored in the local file cache_path/hash(filepath).substr(0, 3)/hash(filepath)/offset, with Block metadata saved in the BE node. When accessing the same remote file, Doris checks whether the cache data of the file exists in the local cache and determines which data to read from the local Block and which data to pull from the remote based on the Block's offset and size, caching the newly pulled remote data. When the BE node restarts, it scans the cache_path directory to restore Block metadata. When the cache size reaches the upper limit, it cleans up long-unused Blocks according to the LRU principle.

Report issue
ASF
Resources
Community
Join the community
Copyright © 2026 The Apache Software Foundation,Licensed under the Apache License, Version 2.0. Apache, Doris, Apache Doris, the Apache feather logo and the Apache Doris logo are trademarks of The Apache Software Foundation.