A campaign's conversion rate has dropped. The team needs to compare it with similar past campaigns before calling it abnormal. In a road test, a rise in human interventions means little without today's test-run count, each vehicle's current state, and records from previous weeks. Both investigations need new events and historical data in the same query.

Fluss keeps event logs and current state in tables. With its Fluss Catalog, Apache Doris 5.0 can query those tables alongside history tiered to Paimon. The read path depends on which portion of the data a query needs.

1. How Fluss unifies streams and lakes​

Keep event streams and current state in tables​

Many real-time pipelines send events through a message queue and then copy them into a state store or analytics engine. Until the copy finishes, analysts cannot query the latest events. For a query that also needs history or current state, they must keep track of what each system holds.

Apache Fluss stores events in tables with schemas and feeds downstream stream consumers. In a suitable pipeline, it can handle the ingestion and distribution previously done by a separate messaging system. Analysts can query recent events without a copy job built solely for analytics. Fluss's columnar layout can also reduce I/O when a query selects only a few fields from a wide table.

Fluss has two table types:

Each table has a Log layer for recent writes and a Lake layer for older data. A tiering service writes the older records to an open lake format. The readable lake snapshot records how far the data has been tiered. Doris uses that position to separate lake history from new log records.

Analyze Fluss data with SQL​

Analysts still need to compare events with history, join them to current state, and inspect individual records. How unusual is a new deviation, and which devices or users account for it? Doris queries Fluss tables directly to answer those questions with SQL.

Suppose human interventions spike at one intersection during an autonomous driving road test. The team queries the last 30 minutes of events, counts the test runs through that segment, and compares the intervention rate with the same time of day over previous weeks. More interventions may simply mean more vehicles passed through. If the rate rose too, the team needs to investigate.

The team then checks current vehicle states, narrows down the trips and time window, and reviews the driving records.

Doris 5.0 queries Fluss tables through the Fluss Catalog. If a table has been tiered to Paimon, the same query can read lake history and records still in the Fluss log.

2. How Doris reads new events, history, and current state from Fluss​

2.1 Query newly written events​

To investigate, the road-test team filters recent intervention records by road segment, trigger, and the software version running when each event occurred. Copying the records into Doris first would slow down that initial query.

Fluss divides tables into Buckets for parallel reads and writes. In a partitioned table, records go to a partition and then a Bucket. Unpartitioned tables go straight to Buckets. Each Bucket has its own Offset, a position in its log.

Doris reads Fluss log tables through the Fluss Catalog. It can filter and aggregate those records or join them with vehicle models, test plans, and other dimensions in Doris tables. Teams may still store recurring, stable query results in Doris internal tables.

Analysts can inspect Fluss databases, tables, and columns through the Catalog before they query. Fluss still owns the source tables; storing results in Doris is optional.

At planning time, Doris fixes the log end position for each Bucket. It stops there even if more records arrive during execution. Those records appear in a later query, so each execution has a fixed endpoint.

2.2 Combine lake history with new Fluss events​

To establish a baseline, the team compares the same road segment across several weeks of morning rush hour. Recent intervention and trip events may still be in the Fluss log, outside the latest readable Paimon snapshot. Separate lake and log queries would force the team to reconcile overlap and gaps itself.

The tiering service writes log records to Paimon, commits a readable snapshot, and records the last tiered Offset for each Bucket. Fluss still holds records written after those positions.

For Fluss tables with lake tiering enabled, Doris can use Union Read. At query planning, Doris fixes a Paimon snapshot and the end Offset for each Bucket. Paimon supplies the history covered by the snapshot; Fluss supplies records from the tiering position to the fixed end. Analysts query one logical table. Records that arrive later appear in the next query.

For append-only events such as interventions, the Offset boundary helps prevent duplicate or missing reads. Analysts can inspect the lake and log separately to check tiering progress. If no readable lake snapshot exists yet, the default query can read only data still retained by Fluss. A task that needs complete history should first confirm that a snapshot is ready and check the log retention range.

2.3 Reconstruct the current value of a primary-key table​

Vehicles and device configurations change state too. A vehicle might move from "road testing" to "awaiting maintenance." Counting both changes as current would put two versions of the same vehicle in the report.

A primary-key table has two possible read baselines. For a Fluss-only read, Doris starts with a KV snapshot and applies the later changelog. For a tiered table using Union Read, it starts with a Paimon lake snapshot and applies the Fluss changelog after that snapshot. Union Read does not combine the KV and lake snapshots.

Figure 5. The two paths use a KV snapshot and a lake snapshot as their respective baselines. In Union Read, an update returns the new value and a deletion removes the primary key.

Union Read finds keys changed since the lake snapshot, filters their older lake rows, and emits the latest values from the log tail. An update replaces the old state; a deletion removes the key. Each vehicle appears once in the current-state result.

For road-test analysis, the event table records the software version at the time of the event. The vehicle state table shows the version running now. Those versions answer different questions.

2.4 Choose the read range for the task​

Trend analysis needs both history and new events. A baseline report for an archived month may need only lake data. To check tiering progress, analysts need to see what remains in the log. Doris provides three entry points for Fluss tables tiered to Paimon:

Figure 6. The table name covers both lake and log. $lake reads only the lake; $log reads only the append-only log after the readable lake snapshot.

When querying the table name directly, fluss.union_read.mode controls how Doris chooses the read path. The $lake and $log entry points already identify a specific data segment.

$log and disabled start at different positions. $log starts at the tiering position recorded by the lake snapshot. For log tables, disabled reads records still retained by Fluss; for primary-key tables, it reconstructs current state. If Fluss has removed older log segments, a disabled query on a log table cannot see history that exists only in the lake.

Several factors affect scan cost and the choice of read path:

• The lake side uses Doris's Paimon reader and file cache. Partition filters and selecting only the necessary columns can reduce scanning and decoding. A filter on an ordinary data column gives the right result but may not reduce the data read from the source.
• Tiering delays lengthen the primary-key changelog tail that must be merged, increasing memory use. Doris limits the number of tail records for primary-key Union Read. Above that limit, auto changes the read path and required returns an error.
• EXPLAIN shows whether Union Read is used and the ranges planned for the lake and log. Query Profile shows row counts and time spent on each side. Together, they reveal whether the cost comes from scanning history, reading the log tail, or tiering progress.

Teams can save the results of stable, frequently run queries in Doris internal tables when needed.

3. Performance measurements​

We tested the read paths on an 18-core development machine with 48 GB of memory. Doris 5.0 ran as one FE and one BE, with the BE's default 2 GB JVM heap. Fluss 1.0.0 ran one CoordinatorServer and one TabletServer; Paimon used MinIO on the same machine. The primary-key test table had 30 million keys, 13 columns (about 210 bytes per row), and 16 Buckets. After one warm-up, we ran each query five times and report the median. Docker port forwarding limited bandwidth to both Fluss and the lake, so comparisons on this machine are more useful than the absolute times.

3.1 Primary-key tables: pure JNI reads versus Union Read​

We measured three read paths:

• With a pure JNI read (fluss.union_read.mode = 'disabled'), the BE does not read the lake. For each Bucket, it copies the full KV snapshot to local disk, scans it with RocksDB, and merges the later changelog to reconstruct current values. The BE calls the Fluss Java SDK through JNI.
• With Union Read and lake auto-compaction, Doris uses the Paimon lake snapshot as the baseline and reads lake data with its native Parquet reader. For each Bucket with a log tail, the BE reads that tail once from Fluss through JNI and filters older lake rows by primary key. The lake table has table.datalake.auto-compaction enabled.
• With Union Read but no lake auto-compaction, the logical read is the same. Overlapping lake files, however, require the Paimon Java reader through JNI to merge them. This is the default Fluss configuration.

The first column counts keys updated after the lake snapshot, which determines how much Union Read must fetch from Fluss: 0, 10,000, or 100,000 keys per Bucket. Each cell gives the time in seconds for COUNT(*), a grouped aggregation (count, amount sum, and average discount by category), and SELECT * (excluding transfer of results to the client), in that order.

• For SELECT *, pure JNI finishes 1 to 3 seconds sooner than Union Read, but it uses an average of 9 to 10 cores (68 to 75 core-seconds). Union Read uses 1 to 2 cores (9 to 18 core-seconds). Pure JNI consumes 4 to 8 times as much CPU time. The KV snapshot is on local disk, while lake data crosses a forwarded port, which favors pure JNI in this setup.
• Pure JNI copies each Bucket's KV snapshot to a BE temporary directory so RocksDB can open it. A query uses a sampled peak of 2.9 to 4.3 GB and releases the space afterward. Union Read does not copy the KV snapshot and does not use that BE disk space.
• Without lake auto-compaction, a merge read can load data from several files into the BE's JVM heap at once. A full-table read uses 16 scanners in parallel by default and fails with OutOfMemoryError on the default 2 GB heap. enable_jni_heap_admission prevents that failure in this test, as shown below. Enabling table.datalake.auto-compaction for the primary-key lake table addresses the underlying cause.

3.2 BE JVM Memory Control​

We repeated the full-table read without lake auto-compaction on a BE with a 2 GB JVM heap, then enabled enable_jni_heap_admission and ran it again:

enable_jni_heap_admission is new in Doris 5.0 and defaults to off. When enabled, readers that expect heavy JVM heap use, including the merge readers in this test, declare an estimate before they start. The BE queues a new reader if admitting it would push declared usage beyond half the maximum JVM heap. Readers proceed as earlier ones finish. This prevented OOM in the test, though queueing increased elapsed time.

4. Planned work​

Apache Doris 5.0.0 provides the Fluss Catalog as an experimental feature for reads and analysis. Union Read currently works with Paimon lake tables; Iceberg, Lance, and other lake formats are planned. We plan to benchmark historical scans, log-tail merging, and concurrent queries on reproducible workloads, then use those results to improve performance and resource use.

Today, Doris reads Fluss log and KV data through the Java SDK. Fluss 1.0 has a Rust SDK, and we plan to move these reads to it gradually. Before switching, we need to preserve the semantics of log tables, primary-key tables, and Union Read. Real workloads will show whether Rust improves data conversion, memory use, and resource control.

Preview: Apache Doris 5.0 multimodal lakehouse (4) | Scenario and solution

Apache Doris introduced Variant as a column type in version 2.1 for storing semi-structured data such as JSON. Users do not need to define fields before creating a table. During ingestion, Doris automatically splits the data into typed subcolumns. This retains the flexibility of JSON ingestion while providing analytical performance close to columnar storage. Apache Doris 5.0 will extend the Variant type from internal tables to open lake formats such as Iceberg and Paimon. The same VARIANT type, SQL syntax, and execution engine will process Variant data across these formats. Using an Agent trace data pipeline as an example, this article explains the need for this capability, the business problems it addresses, and when to use Variant in internal tables or data lakes.

1. Agent traces: The divide between Variant producers and consumers​

JSON is a natural representation for an AI Agent execution trace. Protocols such as MCP and A2A use JSON-RPC, while tool-call parameters, model output, and span attributes for each step all contain nested structures. A complete trace can range from tens of KB to several MB and may be seven or eight levels deep. Tool lists and parameters also change as models evolve. Eight fields yesterday may become twelve today, so defining a stable schema in advance is often impossible.

Analytical systems have traditionally handled this type of data in three ways:

The Variant type addresses the trade-off between flexibility and query performance. Apache Doris has provided the Variant column type since version 2.1 for storing semi-structured data such as JSON:

• When creating a table, users declare a single VARIANT column without defining its fields in advance.
• During ingestion, Doris automatically splits JSON into typed subcolumns, performs type inference and promotion, stores infrequently used fields in sparse columns, and allows indexes on subcolumns.
• Queries access values by path and read only the relevant subcolumns instead of parsing the entire JSON document.

Variant combines the ingestion flexibility of JSON with query performance close to columnar storage.

For Agent observability platforms, the recommended data model is:

• Store stable fields such as trace_id, tool_name, and latency as regular columns.
• Store inputs, outputs, and tool parameters in a VARIANT column, allowing upstream systems to add fields without altering the table.
• Use regular SQL with path-based value access for failure attribution, cost aggregation by tool, and effectiveness comparisons across model versions.
• Use inverted indexes for full-text searches over text in model outputs.

Later Apache Doris releases added Variant features such as Schema Template, secondary indexes at the subcolumn level, DOC mode for JSON documents hundreds of MB in size, and NestedGroup for searching nested arrays.

Changes in enterprise data architectures created another problem. Agent traces are produced by ingestion pipelines running on Flink or Spark. The data usually lands in Paimon first, then feeds training, evaluation, observability, and other workloads. Before version 5.0, Doris could process Variant data only in its own format. Consumers that wanted to analyze the data with Variant had to write and synchronize it to both internal tables and the lake. This duplicated storage and introduced synchronization delays and data inconsistencies.

Other Variant use cases have the same problem:

• Event tracking: Event properties can be stored in a VARIANT column, allowing product teams to add properties without altering the table. The same event data is also a shared enterprise asset that algorithm teams need for training features. This requires another copy in Iceberg.
• IoT telemetry: Device properties vary by model and firmware. A wide table would be excessively sparse, while a VARIANT column allows one SQL query to aggregate data across device models. Complete datasets must also be retained for years, and object storage with an open format is the only way to reduce storage costs significantly. Real-time alerts and historical analysis must therefore access different systems, which fragments the user experience.

Variant needs to follow the data. When data resides in the lake, Variant must be available there too, and the compute engine must support unified analysis of that Variant data.

2. Variant standardization: Bringing semi-structured data into the "one copy, multiple engines" model​

Semi-structured data in a lake traditionally had two options: storage as strings or conversion to STRUCT through ETL. As this data became more widely used, the performance and operational costs of those approaches became harder to ignore. Variant began moving toward open standardization in 2025:

• In March 2025, Apache Parquet 2.11 defined a binary encoding for Variant and introduced the Shredding specification. Writers can split frequently accessed fields into typed columns while retaining the remaining data in a binary value, giving the file itself a columnar shape. The specification was finalized in August of the same year.
• In June 2025, the Apache Iceberg V3 specification was approved, making Variant a first-class table type. Apache Paimon has supported Variant and Shredding since version 1.4.
• Spark 4.0/4.1, Flink 2.1, and DuckDB subsequently added support, while Trino began offering experimental support. Snowflake and Databricks also announced general availability of their respective support for Variant in open formats in 2026.

The open lakehouse model of "one copy of data shared by multiple engines" is extending to semi-structured data. With a standard format, any compliant engine can read and write Variant data in a lake. We therefore decided to extend the Variant processing capabilities available for internal tables to standard Variant formats.

An engine needs to solve three problems to use Variant effectively in a lake:

1. High-performance reads: The engine must be able to use the Shredded layout directly.
2. Variant production: An engine that can read but not write Variant cannot write its analytical results back to the open lakehouse for reuse.
3. Interoperability between internal and external formats: Variant should behave consistently across formats.

The following sections describe the unified Variant solution planned for Apache Doris 5.0. A later article about Variant internals will explain how Doris reads the Shredded layout.

3. Doris 5.0: One Variant type, one SQL syntax, one execution engine​

Doris 5.0 will bring the Variant analytical capabilities of internal tables to open lake formats such as Iceberg and Paimon while keeping the user experience consistent across formats.

3.1 Three layers of unification​

• Upper layer: Unified type and SQL syntax. Doris has one VARIANT type, with consistent SQL syntax and semantics for internal and external tables.

  • Unified path access syntax: payload['event']['type']
  • Unified conversion logic: use CAST for type conversion and VARIANT_TYPE to inspect the actual type
  • Unified constructor: use PARSE_TO_VARIANT to construct values from JSON text
  • Unified nesting: ARRAY, MAP, and STRUCT support nested VARIANT values
  • Unified NULL semantics: SQL NULL is distinguished from JSON null

• Middle layer: Unified execution operators. Data read from internal table files or Parquet files in the lake is converted into the execution engine's internal Variant column representation. The same operators then process the data, providing consistent behavior.

  • Variant values from internal and lake tables can appear in the same SQL statement and participate in UNION, JOIN, and other operations.
  • Internal and external tables can write to each other, for example, INSERT INTO LakeTable SELECT ... FROM DorisTable.
  • Variant can participate directly in GROUP BY and DISTINCT, with deduplication based on logical values rather than byte representations.

• Lower layer: Unified format access interfaces. Logical abstractions for Scan and Sink operators allow each format to implement only its own reader and writer. The Doris internal table format, for example, supports subcolumn splitting, inverted indexes, BloomFilters, real-time ingestion, and primary-key updates. Open lake formats support both Plain and Shredded layouts, with each format implementing its own write layout as described in Section 3.2. Format-specific differences remain at the lowest layer wherever possible, allowing most of the execution pipeline to be reused.

3.2 Capability matrix​

The following Variant capabilities for data lakes are planned for Doris 5.0:

Iceberg currently writes only the Plain layout. Reading the Shredded layout is fully supported. Shredded writes are planned for a later iteration, as described in the conclusion.

Availability: The lake Variant capabilities and SQL workflow below are planned for Doris 5.0. They describe the intended behavior rather than a currently released feature.

3.3 Example: Unified lakehouse analysis of Agent traces​

The following example uses an Agent observability platform.

First, Flink writes trace data to Paimon. Stable fields are stored as regular columns, while variable data is stored in a VARIANT column.

-- Flink writes the paimon.obs.agent_traces trace table:
-- (trace_id, tool_name, latency_ms, dt, payload VARIANT)

-- 1. Analyze failures in lake tables using path extraction, type conversion, and predicates
SELECT tool_name, COUNT(*) AS failures
FROM paimon.obs.agent_traces
WHERE CAST(payload['result']['status'] AS STRING) = 'error'
  AND CAST(payload['model']['version'] AS STRING) = 'v2'
GROUP BY tool_name;

-- 2. Load hot data into internal tables for real-time alerts and full-text search
INSERT INTO internal.obs.agent_traces_hot
SELECT * FROM paimon.obs.agent_traces WHERE dt = CURRENT_DATE();

-- 3. Write derived results and the VARIANT column back to the lake for other engines
INSERT INTO paimon.obs.failed_traces
SELECT trace_id, tool_name, payload
FROM internal.obs.agent_traces_hot
WHERE CAST(payload['result']['status'] AS STRING) = 'error';

In this pipeline:

• The lake table and internal table use the same VARIANT type, so they do not require separate schemas.
• Internal and lake tables use the same SQL query syntax.
• Standard SQL INSERT ... SELECT statements handle data synchronization and write-back.

The same pipeline also works for event tracking and IoT:

• Event tracking: Events are shared as Iceberg Variant data. Spark creates features, while Doris produces reports. Both use one copy of the data and one schema-evolution process.
• IoT: The complete dataset remains in the lake for long-term retention. Hot data enters internal tables through INSERT ... SELECT for real-time alerting, while hot and cold data use the same data model.

4. Recommendations for choosing a Variant format​

The following comparison of Variant in internal and external tables can help users select a format.

Variant in internal tables provides more features and better read and write performance. Variant in the lake allows multiple engines to access the same data and offers lower storage costs.

The recommendations are:

5. Conclusion: Variant follows the data​

Doris introduced Variant for internal tables in version 2.1 and plans to support open lake formats in version 5.0. Future work will strengthen Shredded writes for Variant and improve the performance of reading Variant from data lakes. These changes are intended to meet enterprise requirements for semi-structured data analysis and cover more use cases.

The next article, Apache Doris Variant in the Lake: How Shredding Works and Performance Benchmarks, will examine the implementation details of Variant and compare the performance characteristics of different formats. It will explain in more detail how Variant works and how to use it.

Introduction:

Kwai uses Apache Doris in production to query vector indexes directly on Apache Paimon tables. Vectors and business data remain in Paimon, while Doris SQL initiates retrieval and pushes execution down to the data lake. Existing ingestion pipelines and OLAP access patterns do not change. This article explains the indexing foundation in Paimon 2.0, the read path developed jointly by the Doris and Paimon communities, and benchmark results for tens of millions and hundreds of millions of vectors.

As LLM and RAG applications have become more common, semantic search has become a standard data platform requirement. One common design synchronizes vectors, filter fields, and result fields into a standalone vector database. The main cost is not the initial synchronization but the ongoing maintenance. Synchronization pipelines must run continuously, which creates a data freshness window. Vectors and business fields are split across two systems, as are access control and governance. Teams also need additional safeguards to maintain consistency between the systems.

Kwai took a different approach. Vectors and business columns remain in Paimon tables, and the Vector Index in Paimon 2.0 handles retrieval. Doris reads indexes and data directly through Paimon Rust. It retrieves candidates and looks up rows in parallel by bucket. Users continue to write Doris SQL with the same syntax they use for Doris internal tables.

Availability: Kwai has validated this integration in production, while the Doris 4.2 capabilities described here are scheduled for public release at the end of September.

1. Data foundation: indexing in Paimon 2.0​

1.1 Physical separation of indexes and data​

The Vector Index in Paimon 2.0 works with ordinary Paimon tables and has two forms: the Global Vector Index for append tables and the PK Vector Index for primary-key tables. Both use the same index implementation and retrieval model, but they build indexes and retrieve matching rows differently. Kwai's business tables are primary-key tables, so the rest of this article focuses on the PK Vector Index.

Both forms store index files separately from data files. A Snapshot references a Data Manifest and an Index Manifest. The Index Manifest records each index file's type, field, location, and coverage. For append tables, it records the covered Row ID range. For primary-key tables, it records the covered source data files. Adding an index therefore does not require a different data layout, and the same table can still support batch processing, stream processing, and regular OLAP reads.

Figure 1 shows how index files relate to data files.

Figure 1: Index files and data files are stored separately. The same Snapshot links them through two types of Manifest. Index segments for primary-key tables are generated during compaction and record their source files.

Vector columns are declared as ARRAY<FLOAT> or fixed-length VECTOR<FLOAT, N>. For primary-key tables, table properties configure vector indexes, which are built automatically during compaction without a separate trigger. The following Spark SQL specifies the indexed column, index type, and distance metric when creating a primary-key table:

CREATE TABLE vector_items (
item_id BIGINT,
embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;2048',
payload STRING
) TBLPROPERTIES (
'primary-key' = 'item_id',
'bucket' = '32',
'deletion-vectors.enabled' = 'true',
'pk-vector.index.columns' = 'embedding',
'fields.embedding.pk-vector.index.type' = 'ivf-rq',
'fields.embedding.pk-vector.distance.metric' = 'l2'
);

• Spark SQL does not have a VECTOR type. The column comment __VECTOR_FIELD;2048 declares ARRAY<FLOAT> as a fixed-length, 2,048-dimensional vector.
• Build parameters such as nlist are passed as JSON through fields.<col>.pk-vector.index.options.
• The reranking multiplier refine_factor can be set in either table properties or query options.

After data is written, compaction merges files within a bucket into a level above Level 0 and generates an index segment for the vector column. A new Snapshot references the segment after it is registered in the Index Manifest. The segment records the source data files it covers, which allows matches to map back to physical row positions within those files. Newly written data does not belong to an index segment until the next compaction. Section 2.2 explains how retrieval handles this intermediate state.

1.2 Five types of vector indexes​

The Paimon 2.0 Vector Index currently supports five index implementations, each with different characteristics and trade-offs:

The index types balance query performance, recall, index size, and build time differently. Storing hundreds of millions of 2,048-dimensional vectors at their original precision produces a correspondingly large index and increases the amount of data read from remote storage. RQ uses rotated residual quantization to achieve greater compression, though it must be tuned to meet quality targets. Kwai chose IVF-RQ as a balance between compression and recall. The measured Recall in Section 3.3 ranges from 96.0% to 99.6%.

Figure 2 compares the five index types and shows Kwai's choice.

Figure 2: Each index type makes a different trade-off between index size and recall. Kwai chose IVF-RQ and used its higher compression to reduce remote reads.

2. Bringing indexes back to the data lake: how Doris directly queries Paimon indexes​

A compute engine needs the following capabilities to use indexes on lake data directly for retrieval:

• A high-performance data read path.
• Awareness of index distribution and index coverage.
• Optimized index access for common vector-retrieval workloads such as Top-K.

Together, these capabilities allow the engine to retrieve and analyze lake data efficiently. Section 2.1 covers the read path. Section 2.2 explains index awareness and retrieval pushdown. Sections 2.3 and 2.4 describe two index-access optimizations for Top-K workloads.

2.1 High-performance reads: from JNI to Paimon Rust​

In earlier Doris versions, reading Paimon data in MOR (Merge-On-Read) scenarios required the Paimon Java SDK. Doris called the SDK through JNI to read and merge the data. This approach had two problems:

• Doris could not track and control JVM memory at query granularity. If a large query exhausted memory, Doris lacked fine-grained visibility and control over its usage.
• MOR used substantial memory, and high concurrency could cause JVM OOM errors. Increasing the JVM heap reduced the native memory available to the BE.

After testing both options, Kwai chose Paimon Rust. The official Paimon team maintains Paimon Rust and keeps its MOR and index reads in sync with the Java implementation. Tests found that Paimon Rust provided two to three times the read performance of Paimon Java and was approximately 10% slower than the native Doris reader. Most of the difference came from converting Arrow's in-memory format into Doris Blocks. Paimon Rust uses native memory, so Doris can include its consumption in the existing memory-control system and manage it together with other native memory.

At the code level, the Doris BE, which is the C++ execution process, calls the Paimon Rust Reader through a C ABI. Rust and C++ exchange results through the Arrow C Data Interface. Doris then converts those results into Doris Blocks and passes them to the execution layer. Because Arrow is a columnar format, C++ can consume the data produced by Rust directly by column without an intermediate row-by-row conversion.

Figure 3 shows this read path.

Figure 3: Doris BE calls Paimon Rust through a C ABI. Paimon Rust returns results in Arrow's columnar format, which Doris converts into Blocks. Regular reads and vector retrieval use the same path.

2.2 Pushing retrieval down to the data lake​

Unlike a regular read, a vector Top-K query does not know which rows it needs before reading starts. It must identify those rows through the index first. Without the index, a vector Top-K query written as ORDER BY ... LIMIT becomes a full computation. Doris loads the entire vector column into BE memory, calculates distances row by row, and sorts the results. For 128 million rows with 2,048 dimensions, one query would need to transfer approximately 1 TB of vector data.

The query engine must push retrieval down to the index on the data lake. To do this correctly, it must account for four properties of vector indexes on Paimon primary-key tables:

• Indexes are organized by (Partition, Bucket). A bucket is both the data-sharding unit and the indexing unit.
• Paimon's background compaction process generates indexes, so indexes cover only compacted data. Newly written L0 files are not included in any index. Correct retrieval must merge three operations: index retrieval, exact fallback over uncovered files, and Deletion Vector mask computation.
• An index segment is valid only when its set of covered source files still matches the bucket's current set of active files. Checking this condition requires the complete file set for the entire bucket.
• Kwai's business tables are Paimon primary-key tables. Rows in primary-key tables can be updated and merged, so they do not have stable global Row IDs. Instead, the index stores physical coordinates: the data file and the row number within that file. Retrieval uses these physical row numbers to fetch rows from the data files.

Doris uses the following index-query plan to account for these properties:

• The Doris FE, the Frontend responsible for query planning, enumerates scan units by bucket for the Paimon table and obtains the complete file list for each unit.
• Doris dispatches scan units to different BE nodes. Multiple BEs process the file lists for their assigned buckets in parallel. Within each bucket, they perform retrieval, exact fallback, reranking, row lookup, and Deletion Vector application.
• Doris's sort-and-merge operator computes the global Top-K across buckets.

In this design, Paimon Rust performs vector retrieval, while Doris schedules the distributed work and merges the results.

Figure 4 shows whole-bucket dispatch and parallel processing by bucket.

Figure 4: Each scan unit contains one complete bucket. The BE performs index retrieval, exact fallback, and deletion masking within that bucket. Doris then merges the Top-K across buckets.

2.3 Top-K pushdown and index-only scan​

A standard vector-retrieval SQL query sorts rows by distance and returns the first K. The Doris FE uses an optimizer rule to recognize this pattern and rewrite the execution plan:

• Add a retrieval-score column to the scan node.
• Rewrite the distance expression in the original SQL as a conversion of this score, which avoids calculating the distance again.
• Change the sort order from ascending distance to descending score.
• When the vector column does not appear in the SELECT list, do not read the original vector column. This Index-Only Scan removes vector-column reads and transfers.

The optimizer applies the rule when all of the following conditions hold:

• The distance function matches the metric configured for the index.
• A vector index exists on the column.
• The SQL contains no aggregation or JOIN.

If any condition is not met, Doris uses a regular scan to preserve correct results.

The same rule handles queries with filters and pagination. The FE uses partition predicates for partition pruning and retains only buckets from relevant partitions. Predicates that can be converted into Paimon predicates are pushed down to Paimon Rust, which uses them to narrow the candidate set before retrieval. This provides pre-filtering. If the filter column is not indexed, the reader must scan that column. Doris functions such as length(a) > 1 cannot be pushed down, so Doris applies those predicates as post-filters to the retrieval results. For pagination, Doris combines OFFSET and LIMIT into a larger K value for pushdown, then applies the actual offset in the upper-level sort operator.

Users can write the same SQL they use for vector retrieval on Doris internal tables. They do not need to modify their queries to use vector indexes on Paimon tables.

Figure 5 shows how the optimizer identifies and rewrites the query.

Figure 5: A vector Top-K query that meets all three conditions is rewritten to sort by score. If the query does not select the vector column, Doris does not read it. If any condition is unmet, Doris uses a regular scan.

2.4 Late materialization for large Top-K queries: separating retrieval from materialization​

A typical Top-K SQL query looks like this:

select id, embedding from vec_table
order by l2_distance_approximate(embedding, [...])
limit <K>;

This SQL retrieves the id and embedding columns in a single scan and then performs retrieval. When K is small, this execution method adds little overhead.

When K is large, the cost of a single scan grows with it. Each bucket must read all user columns for K candidate rows, although the global merge will discard most of those rows. With K = 1 million and a 2,048-dimensional vector column, each candidate row carries 8 KB of vector data (2,048 × 4 bytes). As more candidates are discarded, more of those reads are wasted.

Kwai currently rewrites these queries as self-joins. A subquery performs retrieval and returns only IDs, then joins the original table to fetch vectors for the matching rows, as shown in Section 3.2. During the join, Doris pushes the candidate ID set down to the Paimon read layer. The reader uses this set to exclude unrelated files and row groups. It then uses Parquet page-level indexes to skip unrelated pages and materializes the vector column only for matching rows. In the Section 3.3 results, returning only IDs with Top-K = 1 million takes 7.9 to 8.0 seconds, while returning vectors as well takes 65 to 76 seconds. Most of that difference comes from reading and materializing vectors for the 1 million result rows. This approach works, but it requires users to rewrite their SQL.

The next execution plan is still under development. It separates retrieval and materialization into three stages:

• Candidate stage: Each BE performs retrieval on its assigned buckets without reading user columns and produces only a candidate set.
• Merge stage: All candidates are collected for global Top-K selection and reranking. The result is a list of rows to materialize, represented as data files and row ranges within those files. The list is grouped by data file, and matching row ranges are merged.
• Materialization stage: Read user columns only for rows in the row list.

The merge stage must run on a single node for algorithmic reasons. Reranking needs the global candidate set to calculate the correct Top-K, and grouping and merging row ranges also require the complete set. The aggregated candidate set is only on the order of megabytes, and this stage performs no hash or range repartitioning.

Figure 6 compares the one-pass scan with the three-stage plan.

Figure 6: A one-pass scan makes every bucket read wide columns for K candidates. The three-stage plan first produces candidates alone, then materializes matching rows after the global Top-K step.

The expected benefits are:

• As Top-K grows, more candidate rows are discarded, so the system can avoid more wide-column reads.
• The materialization stage replaces the user-written self-join and removes that join from the execution plan.
• The FE uses a cost model to choose between the two forms automatically. Users do not need to know which form it selects or change their SQL.

Section 3 reports measured performance for the current one-pass form. Section 4 gives the current status of the three-stage implementation.

3. Performance: scale and results in Kwai's production environment​

3.1 Deployment and benchmark methodology​

Kwai already uses Doris external tables in production to query vector indexes on Paimon tables. The deployment is configured as follows:

The tests use the following dataset sizes:

3.2 Two query types​

The first query type returns only business identifiers. The vector index produces the Top-K directly:

select id
from vec_table
order by l2_distance_approximate(embedding, [...]) /* 2,048-dimensional query vector */
limit <K>; -- K is 100, 10,000, or 1,000,000

This is a typical Index-Only Scan test case. It primarily measures the cost of retrieval itself.

The second query type returns both business identifiers and vectors. With the current implementation, users rewrite these queries as self-joins. A subquery retrieves the Top-K IDs, then joins the original table to fetch vectors for the matching rows:

select t1.id, embedding
from (select id, embedding from vec_table) t1
join (
select id
from vec_table
order by l2_distance_approximate(embedding, [...]) /* 2,048-dimensional query vector */
limit <K>
) t2 on t1.id = t2.id;

The "Business identifier + vector" column in the Section 3.3 table uses this self-join SQL. It primarily measures the cost of reading and materializing the vector column. After the three-stage plan described in Section 2.4 is implemented, users will be able to write the same query directly as:

select id, embedding
from vec_table
order by l2_distance_approximate(embedding, [...]) /* 2,048-dimensional query vector */
limit <K>; -- K is 100, 10,000, or 1,000,000

The materialization stage will then replace the self-join, and the SQL will be identical to a query against a Doris internal table.

3.3 Measured results from tens of millions to hundreds of millions of vectors​

The following table reports query performance and recall for different parameter combinations. Both latency columns measure end-to-end latency. Queries that return only business identifiers use the first SQL statement in Section 3.2. Queries that also return vectors use the self-join SQL. The values are representative results from individual queries:

The table supports two conclusions:

• Index-retrieval latency changes little with data scale. When the row count grows from 16 million to 128 million, latency for returning only business identifiers increases by 0.4 seconds, 0.2 seconds, and 0.1 seconds across the three Top-K settings, respectively.
• When Top-K = 1 million, vector-column materialization is the bottleneck, not retrieval.

These are representative results from individual queries. The table does not cover QPS.

3.4 Comparing vector retrieval on Paimon tables and Doris internal tables​

Kwai also compared vector retrieval on Paimon tables with retrieval on Doris internal tables. The team loaded the same Paimon data into Doris internal tables and queried it through Doris's own vector indexes. The following table gives the results.

• External-table latency is 1.07 to 4.18 times internal-table latency. When the query returns only business identifiers, the absolute difference is 0.7 to 1.3 seconds.
• At Top-K = 1 million, the difference narrows to 1.07 to 1.36 times. Most of the latency at this setting comes from row lookup and materialization. Both sides read the same amount of data, which reduces the effect of cache location.

Figure 7 summarizes the results measured by Kwai.

Figure 7: Recall is 96.0% to 99.6% at both scales. Queries that return only business identifiers finish within 2.0 seconds for Top-K values up to 10,000. At Top-K = 1 million, materialization accounts for most of the latency.

Internal tables perform better, but returning only business identifiers adds just 0.7 to 1.3 seconds of query latency in absolute terms. In return, Kwai gets the following operational benefits:

• Better data freshness between writing data to Paimon and making it searchable.
• Unified storage without redundant data copies.
• No compute-resource overhead for transferring and converting data between different systems.
• No separate access-control and governance systems to maintain.

4. Results and roadmap of joint development by the Doris and Paimon communities​

Kwai, the Apache Doris community, and the Apache Paimon community developed this read path together. Kwai has contributed more than 120 PRs to the Apache Paimon and Apache Paimon Rust communities. The Doris-side changes are planned for merger into the Doris community within the next two months.

Kwai is also working on the following changes to improve vector retrieval performance when Doris queries Paimon:

• Three-stage late materialization: The plan described in Section 2.4 separates candidate retrieval from vector-column materialization. At operating points such as Top-K = 1 million, the system will stop reading wide columns for candidate rows that are later discarded. Users will also no longer need to write self-joins manually.
• Local index caching: Add local disk and memory caches within the calling process, with Doris providing consistent-hash distribution.
• Full-text retrieval and Hybrid Search: The full-text retrieval path is already complete on the Paimon side. After the Doris-side integration is complete, users will be able to combine structured filtering, keyword matching, and semantic retrieval on the same lake table.
• Completing retrieval semantics: Add similarity-range retrieval. Kwai has begun an internal grayscale rollout and plans to contribute the feature to the community after large-scale production validation. Retrieval parameters will also become overridable per query instead of remaining limited to table-level configuration.

Learn more​

The multimodal lakehouse capabilities for Paimon, Iceberg, Lance, and other formats have been merged into the Doris 4.2 release branch. They are scheduled for official release with version 4.2 at the end of September.

Abstract

Apache Doris 5.0 Multimodal Lakehouse Preview (2) | Scenario solutions:

Agentic AI is moving enterprise AI beyond one-off question answering and into a continuous "perceive-retrieve-analyze-act-feedback" cycle. Paimon 2.0 manages changing business facts, multimodal data, and indexes in one system, while Doris uses unified SQL to connect vector retrieval, real-time analytics, and result write-back. Together, they give agents current, consistent, and verifiable business context.

Large models determine what an agent can understand. The data platform determines what the agent knows and can trust at a given moment, along with what it retains after taking action. Agentic AI is moving enterprise AI beyond one-off question answering and into a continuous "perceive-retrieve-analyze-act-feedback" cycle.

Agents need access to more than documents and knowledge bases. They also need current orders, inventory, device status, images, videos, vectors, model outputs, and historical results. A real request requires more than semantic retrieval. It may combine vector and full-text search with structured filtering, cross-table joins, metric computation, and permission checks. This creates two requirements for the data platform. The underlying data must support continuous updates and schema evolution in an open format. The query engine must connect multimodal retrieval, real-time analytics, and result write-back in one execution pipeline.

Apache Paimon 2.0 and Apache Doris address these requirements at different layers. Paimon 2.0 manages continuously changing business facts, multimodal content, model features, and indexes. Doris provides unified SQL for reads, writes, vector retrieval, relational computation, and application serving. Together, they supply agents with current, consistent, and verifiable business context.

1. Core challenges of Agentic AI​

Traditional BI primarily answers, "What happened in the past?" Traditional RAG primarily answers, "Which documents are relevant to this question?" Agentic AI must answer additional questions: Does the current state permit an action? Do separate pieces of evidence support the same conclusion? What happened after similar actions in the past? How will this action become part of the context for the next request? Consider an industrial fault-diagnosis agent that receives a new defect image. Returning a few visually similar historical images is not enough.

The agent must also determine whether those defects came from the same equipment model and production stage. It needs to check whether sensor metrics were abnormal before or after the event, whether the equipment is currently under maintenance, whether similar problems are occurring in a cluster, and which remediation method improved the yield of subsequent batches. Requests of this kind expose four data challenges.

1.1 Context: continuously growing multimodal data​

A business record is rarely completed in one operation. Orders, equipment records, and user states may arrive first. Images, videos, and logs appear later. Different jobs then add OCR results, captions, embeddings, model scores, and human-generated labels asynchronously. Model upgrades add new versions of features to the same data. Agents therefore need more than a static wide table. They need a data asset that can keep acquiring fields, content, features, and indexes.

1.2 Retrieval produces candidates; business analysis produces answers​

Vector retrieval can find semantically or visually similar objects, but it cannot determine inventory, permissions, equipment status, time ranges, or business outcomes on its own. To produce context on which an agent can act, the system must combine vector or full-text retrieval with structured filtering, joins, aggregations, window computations, and business-specific ranking.

1.3 Agents need current and consistent context​

When business facts and model features are spread across lake tables, object storage, vector databases, and search systems, each system may have different update times, versions, and deletion states. A stale index may still return a delisted product, a device under maintenance, or a withdrawn document and pass it to an agent.

1.4 Agent actions must feed back into the data loop​

Agents produce classifications, summaries, risk scores, tool-call results, and processing states. Human review adds final labels. If those results remain in the application layer, the next analysis cannot use the latest feedback. The data platform must support reads and write-back while preserving transactional consistency. Agentic AI therefore needs more than a vector database. It needs a complete data pipeline in which multimodal data can evolve, indexes can expand their coverage, business facts remain current, retrieval results can be analyzed, and action results can be written back.

2. Paimon 2.0: continuously evolving the data required by agents​

Paimon has traditionally used Flink to ingest database CDC and message streams continuously. It maintains real-time lake tables on object storage and supports updates, deletions, snapshots, and incremental reads. Paimon 2.0 expands that scope from "continuously updated structured rows" to "continuously growing multimodal data assets."

• BLOB stores large objects such as images, videos, and audio in separate files. Queries do not need to read the object contents when the column is not selected.
• VECTOR<t, n> provides fixed-dimension semantics and a specialized storage layout for embeddings.
• VARIANT stores events, model outputs, and tool-call parameters with flexible structures, while allowing frequently used fields to be extracted into typed subcolumns.
• Data Evolution allows models to add or update only changed columns without repeatedly rewriting unchanged large objects and business fields.
• Global Row ID aligns data files, BLOBs, vectors, indexes, and deletion states with the same logical record.
• Global Index incorporates scalar, full-text, and vector indexes into the lake table's metadata and snapshot system.

These capabilities do not force every type of data into the same file format. Each modality can use a suitable physical layout while remaining consistent with the same table, snapshot, and logical record. The system can identify newly written data and the current index coverage, then address freshness by backfilling indexes or selecting an appropriate query mode.

Paimon 2.0 handles how agent data is stored, updated, evolved, and indexed. It does not, by itself, perform cross-table joins, global Top-K computation, real-time metric calculation, permission checks, or high-concurrency serving. Turning this open data into context that agents can use also requires Doris's query and execution capabilities.

3. Doris and Paimon integration: turning open data into agent context​

Doris support for Paimon 2.0 goes beyond scanning Paimon files as ordinary Parquet files. The integration connects three paths: snapshot-consistent lake-table reads, multimodal retrieval through Vector Index, and standard SQL writes to Paimon.

3.1 Reading the latest business facts while preserving Paimon table semantics​

Doris discovers databases and tables through the Paimon Catalog and caches their metadata. Within one SQL statement, it binds a consistent snapshot and schema. The query planner understands manifests, Primary Key merging, Deletion Vector, incremental ranges, Time Travel, Branch, and Tag, so it does not treat Paimon Primary Key tables as ordinary file scans. Agents can retrieve a consistent business state from a specified snapshot and use incremental queries to detect changes in orders, inventory, equipment, and user states.

3.2 Vector retrieval: bringing Vector Index into the Doris query plan​

Doris FE plans and distributes splits, while BE calls the Paimon Rust Reader through the C ABI. A vector query first reads the Vector Index, searches index shards in parallel, and merges the candidates. It then pushes the candidate range down to files, Row Groups, and Parquet Pages. The reader loads vectors, BLOB metadata, and other wide columns only after confirming matches. Results from Rust enter C++ through the Arrow C Data Interface and are converted into Doris Blocks.

The candidate set then enters the Doris MPP engine for scalar filtering, cross-table joins, aggregation, permission checks, reranking, and global Top-K computation. Vector retrieval is a candidate-generation stage within a distributed SQL plan, not a separate system beside Doris. Paimon Vector Index finds relevant data, while Doris determines whether that data is valid, important, and actionable in the current business context.

Community work and Kwai's production deployments have validated this path. Paimon external tables connect to Doris through the Rust Reader, with data scales ranging from millions to billions of records and Top-K values ranging from 100 to 1 million. Candidate pushdown and on-demand materialization limit the cost of reading wide vectors.

3.3 Real-time writes: writing to Paimon with standard SQL​

An Agentic AI data pipeline cannot stop after reading data. If aggregation results, model labels, risk scores, and human-review states produced by Doris still require separate Spark or Flink jobs for write-back, the architecture again has fragmented transactional boundaries and additional operational pipelines.

For Doris 4.2, the Paimon Catalog is planned to expand from read-only access to integrated read and write support, including:

• INSERT INTO ... SELECT and INSERT ... VALUES for appending query results or application data to Paimon.
• INSERT OVERWRITE for overwriting non-partitioned tables or data in static or dynamic partitions.
• UPDATE, DELETE, and MERGE for updating business states, model labels, and human feedback in Primary Key tables.
• CREATE TABLE and schema evolution for adding, deleting, renaming, and changing the types of fields.
• Variant reads and writes, allowing model results and tool-call parameters with continuously changing structures to enter open lake tables directly.

Doris distributes data across BEs according to Paimon's partitioning and bucket rules, and the BEs generate files in parallel. The FE then collects Commit Messages from multiple writers and creates an atomically visible Paimon snapshot. When a failure or retry occurs, the commit coordinator handles termination, cleanup, deduplication, and idempotent recovery. This prevents partially visible data and duplicate commits. Results produced by agents can return to the same open dataset and become context for later queries and decisions.

4. Execution framework: how an agent request completes retrieval, analysis, and write-back​

For vector retrieval in an agent request, Paimon Vector Index first narrows the candidates. Doris then performs the business analysis. After a vector query enters Doris, execution proceeds in three stages:

1. Plan the context. Doris FE parses the Vector SQL, binds the Paimon snapshot and schema used by the query, and generates parallel splits.

2. Retrieve and read on demand. BE invokes Paimon Rust through the C ABI. Vector Index produces Top-K candidates, after which the reader prunes files, Row Groups, and Pages according to the candidate range and materializes only matching columns.

3. Generate the business answer.

   1. Arrow data is converted into Doris Blocks and enters the MPP engine for joins, filters, aggregation, reranking, and global Top-K computation. The result is then returned to the agent through SQL, an API, or a tool call.
   2. When an agent or human confirmation produces new labels, states, or processing results, Doris starts another write path. Nereids generates a DML plan, data is routed to parallel writers on the BEs, and Commit Messages are returned to the FE for aggregation and atomic commit as a new Paimon snapshot. The new results then become available for the next round of incremental reads, index backfilling, and agent analysis.
   3. With further native integration, Doris will directly consume Paimon Global Index Results, use Global Row IDs to read matching rows, and understand index coverage, the fast / full / detail freshness modes, and the Deletion Vector of the current snapshot. Manifests, index shards, vector pages, and matching data pages will also be managed separately through a tiered local cache.

5. Production use: Kwai brings vector retrieval back to Paimon lake tables​

In Kwai's production environment, vectors and business data remain in Paimon, while Doris handles retrieval, row lookup, and analysis. Kwai needed to provide large-scale vector retrieval on its existing Paimon real-time lakehouse while keeping business data, vectors, permission fields, and update states consistent.

Copying vectors, filter fields, and return fields into a separate vector database would create a second copy of the business data and another synchronization pipeline that would need long-term maintenance. As model versions, business states, and deletion states change, the retrieval system could return stale results that no longer match the current business state.

Kwai chose to keep vectors and business data in Paimon Primary Key tables and connect vector retrieval to Doris external-table queries:

1. Paimon tables store business fields and 2,048-dimensional vectors inline. An IVF-RQ Vector Index retrieves candidates, while remote Alluxio provides caching for data and index reads.
2. Users continue to use Doris SQL. Doris FE plans and distributes splits, and BE invokes the Paimon Rust Reader through the C ABI.
3. Paimon Rust executes vector retrieval in parallel and merges the candidates. Candidate ranges are pushed down to files, Row Groups, and Parquet Pages, avoiding the need to scan and decode the complete vector column in advance.
4. When only business identifiers are required, queries can skip materializing wide vectors. When vectors need to be returned, candidate identifiers are used to look up the corresponding rows, reading only matching pages and records.
5. Read results are converted into Doris Blocks through the Arrow C Data Interface and continue through MPP computations such as filtering, joins, aggregation, and Top-K.

Production validation covered datasets with tens of millions and hundreds of millions of records. Public tests used 16 million and 128 million real 2,048-dimensional vectors, with Top-K values of 100, 10,000, and 1 million. Recall through the Paimon external-table path ranged from 96.0% to 99.6%. When comparing "Paimon external tables with remote Alluxio caching" against "Doris internal tables with local caching," query latency was 1.07 to 4.18 times that of the latter. At a Top-K of 1 million, the gap narrowed to 1.07 to 1.36 times.

These results show that users can keep open data in Paimon and use Doris SQL for vector retrieval and business analysis, even with hundreds of millions of high-dimensional vectors and Top-K values as large as 1 million. They do not have to copy the entire dataset into Doris internal tables or another vector database first.

For Agentic AI, retrieval results can be joined directly with the latest business state in Paimon. Doris can then apply permission filters, compute metrics, and perform relational analysis. Agents receive similar objects together with context that matches current business facts and can be verified before the system acts on it.

Availability: The Paimon capabilities used in these examples are in the Doris 4.2 release branch and are scheduled for public release at the end of September. Use the examples as preview syntax until that release is available.

6. Getting started​

Teams already using Flink and Paimon do not need to change their existing data pipeline as a first step. They can create a Paimon Catalog in Doris:

CREATE CATALOG paimon_catalog PROPERTIES (
    'type' = 'paimon',
    'warehouse' = 's3://example-bucket/warehouse',
    's3.endpoint' = 'https://object-storage.example.com',
    's3.region' = 'region-id',
    's3.access_key' = 'YOUR_ACCESS_KEY',
    's3.secret_key' = 'YOUR_SECRET_KEY'
);

Perform structured analysis through Doris SQL:

SELECT device_model, defect_type, COUNT(*) AS defect_count
FROM paimon_catalog.quality.inspection_records
WHERE event_time >= NOW() - INTERVAL 30 DAY
GROUP BY device_model, defect_type
ORDER BY defect_count DESC;

In the community-developed vector retrieval path, express Top-K queries through Doris SQL:

SELECT id
FROM paimon_catalog.quality.vector_items
ORDER BY l2_distance_approximate(embedding, [...])
LIMIT 100;

After the analysis is complete, write the aggregated results directly back to Paimon:

INSERT INTO paimon_catalog.quality.daily_defect_summary
SELECT DATE(event_time), device_model, defect_type, COUNT(*)
FROM paimon_catalog.quality.inspection_records
GROUP BY DATE(event_time), device_model, defect_type;

For Primary Key tables that require continuous state corrections, use MERGE to incorporate human reviews or new model results:

MERGE INTO paimon_catalog.quality.inspection_records AS target
USING review_results AS source
ON target.record_id = source.record_id
WHEN MATCHED THEN
    UPDATE SET review_status = source.review_status,
               defect_type = source.defect_type
WHEN NOT MATCHED THEN
    INSERT (record_id, review_status, defect_type)
    VALUES (source.record_id, source.review_status, source.defect_type);

7. Conclusion: using business facts as they emerge​

The data problem in Agentic AI is broader than adding a vector field or deploying a vector database. The system must keep changing business facts, multimodal content, model features, retrieval indexes, and action results consistent. It must then turn that data into verifiable context when an agent needs it.

Paimon 2.0 provides an open foundation for storing, updating, evolving, and indexing this data. Apache Doris connects Paimon snapshot reads, Vector Index, MPP OLAP, and standard SQL writes. This allows agents to maintain a continuous "perceive changes-retrieve evidence-analyze and decide-take action-write back feedback" loop.

Doris users do not need to maintain a separate closed copy of their data for AI workloads. The data can remain in Paimon while Doris handles retrieval and analysis. Doris can then write query results and agent feedback back to the open lake tables. Paimon allows the data used by agents to evolve continuously, while Doris turns that data into current, verifiable business context on which agents can act.

We will publish more articles about the Apache Doris 5.0 multimodal lakehouse and open formats such as Iceberg and Paimon. The series will cover the underlying technical principles, performance evaluations, and typical deployment scenarios.

Multimodal lakehouse capabilities for Iceberg, Paimon, Lance, and other formats have now been merged into the Doris 4.2 release branch and are scheduled for official release with version 4.2 at the end of September.

Abstract:

Apache Doris 5.0 Multimodal Lakehouse Preview (1) | Scenario solutions: Autonomous vehicles and embodied intelligence systems generate massive volumes of multimodal data every day. The practical problem is finding high-value episodes quickly and identifying the scenarios, versions, and tasks associated with failures. This article explains how Apache Doris and Lance combine semantic search, structured filtering, and OLAP analytics for hard-case mining, failure attribution, and closed-loop training and evaluation on an open data architecture.

Autonomous vehicles generate video, point clouds, vehicle states, and model outputs every day. Each robot task also produces multiple camera streams, joint states, action trajectories, language instructions, and execution results. Storing this data is only part of the problem. Teams must also answer the questions that determine how efficiently they can iterate on their models:

• Which segments contain scenarios similar to the current accident?
• Has the intervention rate of the new model increased at night, in construction zones, or around irregularly shaped obstacles?
• Which types of robot tasks are most likely to fail, and what similar visual or action patterns appear before failure?
• How can selected high-value episodes be passed to annotation, training, and regression evaluation?

Traditional data warehouses handle structured aggregation well. Vector databases specialize in image similarity search, while object storage holds large numbers of files at low cost. Physical AI needs these capabilities to work together, so semantic search, structured filtering, business metrics, and raw evidence can be used in the same analytical workflow.

Apache Doris support for Lance Catalog provides that connection. Lance stores open multimodal datasets and vector indexes designed for AI. Doris queries the data directly and performs predicate pushdown, parallel scans, vector retrieval, global Top-K selection, and OLAP aggregation within a single SQL execution framework. Teams do not need to copy the data into a dedicated analytical replica first, and their AI data assets remain available to more than one compute system.

1. Why physical AI needs semantic search and analytics, not vector search alone​

Data for autonomous driving and embodied intelligence is naturally organized around episodes. An episode is a reconstructable physical process rather than a single image: sensors observe the environment, a model makes a decision, a control system executes an action, and the physical world produces an outcome.

In autonomous driving, finding images similar to the five seconds before an accident is only the first step. Engineering teams must further constrain the search by vehicle model, sensor version, road type, weather, model version, and time range. They then need to compare intervention rates, false-positive rates, or planning deviations among the candidate segments. The same applies to embodied intelligence. A failed grasp involves visual semantics, but it must also be examined in the context of the task type, robot arm model, policy version, joint states, action trajectory, and final outcome.

Finding high-value data therefore requires at least three layers of computation:

1. Use structured conditions to narrow the search space to the correct business context.
2. Use vector similarity to find related scenarios that the labeling system does not yet cover.
3. Use aggregation, joins, sorting, and window analysis to turn candidate segments into model evaluation findings and actionable datasets.

Lance and Doris handle different parts of this process. Lance manages AI datasets, random access, and vector indexes. Doris brings the retrieval results into an explainable, composable SQL analytics system.

2. Overall architecture: Lance manages AI data assets, and Doris provides unified queries and analytics​

Figure 1. Apache Doris and Lance in the Physical AI data architecture

In this architecture, upstream systems continue to use tools such as the Lance SDK, Ray, and Spark to generate and maintain Lance datasets. These datasets include features, embeddings, labels, model outputs, and existing Lance vector indexes. The data can reside in a local file system, S3-compatible object storage, or OSS.

Apache Doris uses Lance Catalog to discover datasets and schemas. During a query, Doris fixes the snapshot, divides fragments or index segments, runs scans and searches in parallel, and performs global Top-K selection and OLAP computation.

Apache Doris covers a layer that can otherwise be fragmented across separate tools before and after training. Engineers, algorithm teams, business analysts, and agents can use the same system to find data, validate models, locate problems, and construct the next training dataset.

3. Autonomous driving: Find similar video and build a closed hard-case mining loop​

3.1 Finding high-value scenarios is the bottleneck​

In autonomous driving development, hard-case mining means finding high-value scenarios within massive volumes of driving data where models are prone to false positives, false negatives, abnormal braking, or human intervention. These segments make up only a small proportion of all fleet data, but they directly affect model safety and coverage of long-tail scenarios.

Collecting the data is usually easier than finding the right records within the full set of driving data. The main bottlenecks and corresponding solutions include:

• Anomalous segments are rare, and their context is scattered. Vehicles continuously generate data from multiple cameras, point clouds, localization systems, vehicle states, and model outputs. An anomaly may exist only in a segment lasting a few seconds and may occur across different vehicle models, roads, weather conditions, and sensor versions. Doris can first narrow the search by business conditions such as time, vehicle model, road, weather, and model version.
• Existing labels cannot cover all semantics. Queries based only on time, vehicle, or labels can easily miss semantically similar segments that have not yet been labeled. Lance stores episode metadata, embeddings, and vector indexes, and Doris reuses those indexes to retrieve similar scenarios.
• Similarity results are not yet engineering conclusions. Vector similarity search alone may return results from irrelevant business contexts and cannot directly determine issue frequency, model-version regressions, or sample value. Once the candidate results enter Doris, distributed statistics and version comparisons can identify the episodes that should be included in annotation, training, and evaluation.

3.2 A practical hard-case mining workflow​

Suppose a team wants to find segments that are visually and semantically similar to an emergency-braking event involving an irregularly shaped obstacle. The event occurred on urban roads, and the team wants to determine whether the problem is concentrated in a particular perception model version:

1. The Lance dataset stores episode_id, time, road and weather labels, vehicle model, model version, intervention or emergency-braking results, image or segment embeddings, and raw media location information.
2. Doris pushes down conditions such as time range, urban roads, vehicle model, and model version to narrow the search space before generating vector candidates.
3. Doris reuses existing Lance vector indexes and searches them in parallel. Fragments not covered by an index are automatically supplemented with Flat Search.
4. Each parallel task returns local candidates, which Doris merges into a global Top-K result.
5. SQL aggregates intervention rates, emergency-braking rates, and sample counts by model version, weather, and road type to produce a hard-case distribution.
6. Candidate episode_id values are sent for human review, annotation, inclusion in a training dataset, or regression evaluation. The Lance SDK reads the raw media precisely.

Figure 2. Autonomous-driving hard-case mining: from fleet data to model iteration

This workflow does not require every capability to live in one storage format. Instead, it keeps similar scenarios connected to their business context. Algorithm engineers receive more than a page of similar images. They can determine which versions exhibit similar problems, whether a regression is occurring, how many vehicles and road conditions are affected, and which episodes are most useful for the next training cycle.

3.3 Find similar scenarios with a single SQL query​

Suppose an upstream tool has already created a scene_embedding index in the Lance dataset. Doris can start a search through vector_search():

SELECT
    episode_id,
    event_time,
    road_type,
    weather,
    model_version,
    intervention,
    _distance
FROM vector_search(
    "table" = "lance_ai.autodrive.scene_episodes",
    "column" = "scene_embedding",
    "query_vector" = "[0.12, -0.08, 0.31, ...]",
    "top_k" = "200",
    "metric" = "cosine",
    "nprobes" = "20",
    "refine_factor" = "10",
    "filter" = "road_type = 'urban' AND event_time >= '2026-08-01'",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

Here, filter performs pre-filtering. It first restricts the search to the correct scenario range, then generates vector candidates. This behavior usually fits the semantics of hard-case mining better than post-filtering with WHERE outside the TVF. A smaller _distance value indicates greater similarity. Adding a unique key explicitly as the second sort field produces a stable return order when distances are equal.

Vector candidates can also feed standard SQL aggregation. For example, the following query calculates the distribution of interventions in similar scenarios by model version:

WITH similar_scenes AS (
    SELECT model_version, weather, intervention, _distance
    FROM vector_search(
        "table" = "lance_ai.autodrive.scene_episodes",
        "column" = "scene_embedding",
        "query_vector" = "[0.12, -0.08, 0.31, ...]",
        "top_k" = "1000",
        "metric" = "cosine",
        "filter" = "road_type = 'urban'",
        "use_index" = "true"
    )
)
SELECT
    model_version,
    weather,
    COUNT(*) AS scene_count,
    SUM(CASE WHEN intervention THEN 1 ELSE 0 END) AS intervention_count
FROM similar_scenes
GROUP BY model_version, weather
ORDER BY intervention_count DESC;

4. Embodied intelligence: Make every episode searchable, comparable, and reviewable​

4.1 Robot data requires cross-frequency alignment, random access, and cross-episode analytics​

An embodied intelligence task typically produces data in several forms and at different frequencies. Robot arm states and control signals may be collected at hundreds of hertz, while multiple cameras record environmental changes at tens of frames per second. Other data includes language instructions, frame-level labels, intermediate model results, and information about task success, failure, and retries.

If this data is stored separately in video files, columnar files, and JSON, teams must maintain timestamp alignment and version mappings across those files. They must also repeatedly implement data-assembly logic for training, playback, and problem analysis. A random read may require locating the video segment first, then finding the corresponding state and action records. When schemas or features change, separate copies of the data can become inconsistent.

Training frameworks focus on reading batches of samples at high throughput. Research and development analytics has a different job: locating similar failures among millions of episodes, comparing policy and hardware versions, and determining whether problems originate in perception, planning, control, or execution. Random access alone cannot provide this type of cross-episode aggregation and attribution. Traditional data warehouse full-table scans are also unsuitable for all multimodal sample access.

Lance organizes states, actions, features, embeddings, and media location information by episode and timeline while preserving random-access paths for training and review. Doris performs conditional filtering, vector retrieval, and cross-episode aggregation over the supported fields. With this division of responsibilities, the same robot data can support model training, unified analysis, and review.

4.2 From failed tasks to a closed training-data loop​

For failures involving robot arms attempting to grasp transparent or reflective objects, teams can use the following workflow:

1. Data from multiple cameras, joint states, action trajectories, instructions, policy versions, and task results enters Lance by episode.
2. An upstream model generates visual or trajectory embeddings and creates vector indexes in Lance.
3. Doris first performs pre-filtering by robot model, task type, policy version, and success = false.
4. Vector search finds images or action segments similar to the target failure pattern.
5. Doris aggregates failure reasons, retry counts, task duration, hardware versions, and policy versions to determine whether the issue is related to perception, planning, control, or hardware.
6. High-value episodes are sent for review, relabeling, and training. The corrected model is then subjected to regression validation against the same dataset.

Figure 3. Embodied-intelligence episode analysis: from multi-frequency sensors to a closed training loop

This workflow also works well for agentic data processing. An agent can use SQL to define the task scope, invoke vector search to locate similar failures, and generate distribution statistics and a list of candidate episodes. Each step retains its filter conditions, SQL, and data versions. Reviewers can inspect how the results were produced instead of relying on a model to search file directories or a vector database without a traceable procedure.

4.3 Example: Search for robot failure patterns​

The upstream system generates query_vector with the same embedding model used for frame_embedding. For example, it can pass a reference image, a representative video frame, or an action trajectory to the model and then provide the resulting floating-point array to vector_search(). The query vector's dimensions, numeric type, and normalization method must match the vector column in the dataset and the settings used when the index was created. The [-0.04, 0.27, 0.18, ...] value in this example only demonstrates the parameter format.

SELECT
    episode_id,
    robot_model,
    policy_version,
    task_type,
    failure_reason,
    retry_count,
    _distance
FROM vector_search(
    "table" = "lance_ai.robotics.task_episodes",
    "column" = "frame_embedding",
    "query_vector" = "[-0.04, 0.27, 0.18, ...]",
    "top_k" = "300",
    "metric" = "cosine",
    "filter" = "task_type = 'pick' AND success = false",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

The results can then be aggregated by policy_version, robot_model, or failure_reason. The query can instead return episode_id and media location fields, allowing an application to use the Lance SDK to open the corresponding video frames, point clouds, or other Blob content for review.

Availability: Lance Catalog support is in the Doris 4.2 release branch and is scheduled for public release at the end of September. The SQL below shows the planned public syntax.

5. How to use Lance Catalog in Apache Doris​

5.1 Connect to a Lance dataset in object storage​

The following example uses S3-compatible object storage. In production environments, supply access keys through secure credential management rather than writing them directly into shared scripts.

CREATE CATALOG lance_ai PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "s3://<bucket>/physical-ai",
    "s3.endpoint" = "https://<object-storage-endpoint>",
    "s3.region" = "<region>",
    "s3.access_key" = "<access-key>",
    "s3.secret_key" = "<secret-key>"
);

Doris also supports local file systems, OSS, and Lance REST Catalogs authenticated with Bearer Tokens, API Keys, or custom headers. Filesystem Catalog maps namespaces by directory: a first-level directory under the warehouse root corresponds to a Doris database, and a .lance directory corresponds to a table.

5.2 Discover datasets, schemas, and existing indexes​

SHOW DATABASES FROM lance_ai;
SHOW TABLES FROM lance_ai.autodrive;
DESC lance_ai.autodrive.scene_episodes;
SHOW INDEX FROM lance_ai.autodrive.scene_episodes;

SHOW INDEX displays indexes that tools in the Lance ecosystem have already created.

5.3 Run vector queries through the Catalog​

SELECT episode_id, event_time, model_version, _distance
FROM vector_search(
    "table" = "lance_ai.autodrive.scene_episodes",
    "column" = "scene_embedding",
    "query_vector" = "[0.12, -0.08, 0.31, ...]",
    "top_k" = "200",
    "metric" = "cosine",
    "filter" = "road_type = 'urban'",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

Catalog creation, dataset discovery, and vector search all remain within the same Catalog namespace. filter restricts candidates to urban-road scenarios before candidate generation, while _distance returns the similarity distance. The query vector's dimensions and distance metric must match those used by the upstream system to generate embeddings and create the Lance index.

Pre-filtering and post-filtering differ in when their conditions take part in candidate generation.

• A condition in the filter parameter of vector_search() is a pre-filter. Lance first filters the data and then performs vector retrieval within that subset. Therefore, top_k represents the Top-K results among the filtered data. This is suitable for conditions such as road type, time range, vehicle model, tenant, or data permissions that must constrain the search space.
• A condition in WHERE outside the TVF is a post-filter. Vector search produces candidates first, and Doris then filters those candidates. This is suitable for expressions that Lance cannot currently push down or for scenarios that require semantic retrieval before business filtering. However, the final result may contain fewer than top_k entries, and the system will not automatically expand the retrieval range to fill the requested count.

A query can combine both approaches. Put strong constraints that define the search scope in filter, place complex calculations or remaining conditions in the outer WHERE, and use EXPLAIN to confirm which conditions have been pushed down.

5.4 Use EXPLAIN to confirm predicate pushdown and index use​

Before running a query, use EXPLAIN with the same vector_search() query. In the execution plan, lancePushdownPredicate indicates that pre-search conditions have been pushed down, while lanceSearchIndexSegments indicates that Lance index segments have been planned. Doris supplements new fragments not covered by the index with Flat Search so that the query includes all data in the current snapshot.

EXPLAIN
SELECT episode_id, _distance
FROM vector_search(
    "table" = "lance_ai.autodrive.scene_episodes",
    "column" = "scene_embedding",
    "query_vector" = "[0.12, -0.08, 0.31, ...]",
    "top_k" = "200",
    "metric" = "cosine",
    "filter" = "road_type = 'urban'",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

5.5 Use a TVF for temporary reads without creating a Catalog​

If you only need to analyze a specific Lance dataset temporarily and do not want to manage it through a Catalog, you can use the S3 TVF directly. The TVF is suitable for path-based reads and scalar analysis. If you need to reuse datasets, schemas, and vector indexes from a Catalog, use the Lance Catalog and vector_search() described above.

SELECT episode_id, event_time, model_version
FROM s3(
    "uri" = "s3://<bucket>/physical-ai/autodrive/scene_episodes.lance",
    "s3.endpoint" = "https://<object-storage-endpoint>",
    "s3.region" = "<region>",
    "s3.access_key" = "<access-key>",
    "s3.secret_key" = "<secret-key>",
    "format" = "lance"
)
WHERE event_time >= '2026-08-01';

6. Conclusion: Connect AI data to decisions​

Autonomous driving and embodied intelligence systems will continue to produce large amounts of data. The scarce resource is the set of high-value episodes that can explain model problems, cover long-tail scenarios, and inform the next iteration.

Apache Doris and Lance provide a direct path from multimodal assets to business decisions while retaining open formats. Lance organizes and indexes AI data and supports random access. Doris makes the data filterable, searchable, aggregatable, and understandable by both people and agents through unified SQL.

With this workflow, finding similar scenarios leads directly to questions about which version is affected, how often the issue occurred, whether it is a regression, and which training dataset should include it. Multimodal data can then participate in the closed loop for model iteration and business decision-making.

6.1 Doris and Lance development roadmap​

The multimodal lakehouse capabilities for Iceberg, Paimon, Lance, and other formats have currently been merged into the Doris 4.2 release branch and are planned for official release with version 4.2 at the end of September.

Future work will focus on the performance, stability, and usability of Doris and Lance in the following areas:

• Establishing reproducible benchmarks across different data volumes, vector dimensions, Top-K values, filter selectivity, index coverage, concurrency levels, and cold or warm caches.
• Measuring P50, P95, and P99 latency, throughput, recall, bytes scanned, object-storage requests, network traffic, peak CPU usage, and peak memory usage.
• Continuing to optimize split balancing, large Top-K merges, high-dimensional vectors, lookups of wide result columns, object-storage reads, and cache paths.
• Providing scenario-specific recommendations for parameters such as nprobes, refine_factor, ef, and use_index, and developing capacity-planning and troubleshooting guides.
• Improving Lance table-management capabilities.
• Supporting data export to Lance files.

Overview

This article covers the core changes planned for Apache Doris 5.0 for the multimodal lakehouse:

1. Core concept: Extend unified search to the open lakehouse and use Doris to build a unified platform for multimodal data analytics.

2. Integration with open formats: Provide read and write integration with mainstream open formats such as Iceberg, Paimon, and Lance.

3. Multimodal data analytics: Adapt Doris to multimodal data types such as Variant, Vector, GEO, and Blob.

4. Use cases: Build a unified real-time multimodal data analytics platform for more AI scenarios, including real-time context for agents and embodied intelligence.

For more than a decade, most data platforms have been organized around structured tables and SQL. Data is consolidated in a data warehouse, where stable batch processing and OLAP queries support reporting and business analysis.

AI changes the assumptions behind this architecture.

Enterprises now need to process business tables, JSON, logs, documents, images, audio, video, embeddings, and model outputs together. The set of data consumers has also grown beyond analysts to include applications and AI agents. A real-world request often combines structured filtering with full-text and vector retrieval, then joins the results with the latest business status, permissions, and historical metrics.

The traditional approach spreads these capabilities across separate systems. Object storage holds raw content, lakehouses hold table data and metadata, search engines handle full-text retrieval, vector databases provide semantic retrieval, OLAP systems perform metric analysis, and the application layer assembles the results. Each system solves a specific problem, but users are responsible for data consistency, version alignment, access governance, and result fusion across them.

Open formats are also expanding beyond structured data. AI workloads now include dynamic attributes, blobs, embeddings, and model results. Iceberg, Paimon, and Lance serve long-term analytical facts, real-time updated data, and AI datasets, respectively, so enterprises can select the appropriate format for each workload. In this article, a multimodal lakehouse refers to structured, semi-structured, and multimodal data stored collectively in these open formats.

For the multimodal lakehouse, Doris 5.0 will focus unification on query and execution instead of requiring users to move all data into one storage system. Data can remain in the open format best suited to it, while Doris provides unified search, analytics, and real-time serving.

Figure 1: Overall architecture of an open-format multimodal lakehouse. Iceberg, Paimon, Lance, and Fluss hold different types of open data assets. At the unified query and compute layer, Doris understands format semantics and connects hybrid search, cross-source joins, MPP OLAP, real-time serving, and access control in one execution pipeline.

Availability: The Iceberg, Paimon, and Lance capabilities discussed here are in the Doris 4.2 release branch and are scheduled for public release at the end of September. The broader Doris 5.0 work remains a preview.

From 4.x to 5.0: Extending unified search to the open lakehouse​

Apache Doris 4.x already provides high-performance unified search and analytics for structured, semi-structured, and vector data in internal tables. Users can combine SQL analytics, full-text search, vector search, and real-time updates in one system. For open lake formats such as Iceberg and Paimon, Doris has primarily supported queries and analysis of structured data.

Doris 5.0 will continue this work by extending capabilities already proven on internal tables to the open lakehouse. Variant data in Iceberg and Paimon will become part of a unified read and write pipeline. Paimon and Lance will gain native full-text and vector search. A new Fluss Catalog will support joint queries across real-time data in Fluss and historical data tiered to Paimon, which will further improve data freshness for analytics.

Figure 2: Capability evolution from 4.x to 5.0. Doris 4.x provides unified search and OLAP on internal tables, along with structured SQL for Iceberg and Paimon. Doris 5.0 extends unified querying to the open lakehouse with Variant reads and writes, native vector search, and joint queries across real-time Fluss data and historical Paimon data.

Doris 5.0 will provide the following capabilities across open formats and the unified execution layer:

• Iceberg: Building on mature support for reads, writes, time travel, row-level updates, and table lifecycle management, Doris will add reads and writes for Iceberg V3 Variant data in both plain and shredded layouts.
• Paimon: The Paimon Catalog will move from read-only access to full read and write support, including INSERT, INSERT OVERWRITE, UPDATE, DELETE, MERGE, DDL, and reads and writes for Variant, full-text indexes, and vector indexes. It will provide a unified entry point for real-time lakehouses and evolving multimodal data.
• Lance: Doris will introduce native read and retrieval capabilities for multimodal AI, including Catalog support, parallel scans, column pruning, predicate pushdown, vector and full-text search, and vector index management.
• Fluss: A new Fluss Catalog will support reads from log tables and primary-key tables. Through Union Read, it will jointly query real-time incremental data in Fluss and historical data tiered to Paimon.
• Unified Doris execution: Doris will connect internal tables, open lake tables, full-text and vector retrieval, cross-source joins, aggregations, window calculations, and permission filtering in one SQL pipeline.

From multiple systems to a unified platform​

The difficulty of assembling multiple systems goes beyond the number of systems involved. The same business entity may have different update times, versions, and permission boundaries in each system. A product that has been delisted, a device currently under maintenance, or a document that has been withdrawn may still appear in a stale vector index and be passed to an agent.

Placing unification at the query and execution layer makes the division of responsibilities clearer. Open formats continue to store long-term data and multimodal assets. Doris internal tables hold real-time business facts that require high concurrency and low latency. The Doris MPP engine performs retrieval, filtering, joins, and analytics in one query, so applications no longer need to move and assemble results across multiple systems. The same layer evaluates business state and applies permission filters for each entity, preventing stale candidates from bypassing these checks and reaching an agent.

Figure 3: Evolution of the multimodal lakehouse data architecture. The architecture moves from traditional multi-system assembly to unified execution with Doris. Data remains in the appropriate open formats, while Doris unifies querying, computing, and serving.

OLAP: From retrieving candidates to determining business outcomes​

Vector or full-text search discovers candidates. Answering the business question usually requires joins, aggregations, window calculations, and multidimensional ranking as well.

For example, image-based product search cannot return products based only on visual similarity. It must also account for inventory, regional availability, price, conversion rate, and return rate. Equipment fault diagnosis cannot stop at finding similar logs. It must compare models, firmware versions, sensor metrics, and repair outcomes.

Whether candidates come from Doris internal tables or open lake tables, Doris can pass search results directly into its MPP OLAP execution pipeline. This allows the query to turn a "similar" result into a business conclusion that is currently valid, complies with the relevant rules, and is worth acting on.

Variant, full text, and vector: Combining different forms of evidence in one SQL query​

Model-generated data rarely fits a stable wide-table schema. OCR results, captions, labels, tool-call parameters, and evaluation results change as model versions evolve. Variant provides a flexible way to store semi-structured data. Full-text search supports precise keyword and rule matching, while vector search broadens semantic recall. Structured conditions constrain time, status, permissions, and business scope.

Doris 5.0 will make Variant data, full-text indexes, and vector indexes in open lake tables available within the same SQL query. Users will not need to copy multimodal data into additional indexing systems solely to make it searchable.

Physical AI: From a single record to a complete episode​

Data generated by autonomous driving systems, robots, and industrial equipment records a continuous process of observation, decision, action, and feedback rather than a collection of static documents. The smallest unit of analysis is therefore no longer an image. It is an Episode that contains time, space, device state, model version, and business outcome.

An Episode involving a robotic arm's failed grasp might contain RGB-D images, 3D poses, joint and torque data, planned trajectories, control commands, model outputs, device logs, and the final sorting result. Images describe the environment, trajectories describe the action, sensor data records the execution process, and the business outcome shows whether the task succeeded.

Open formats allow teams to retain this data over the long term, evolve it, and share it among different tools. Doris organizes these forms of evidence by Episode into an interactive query. The query first filters by time, device, and status, then retrieves similar situations through full-text and vector search, and finally joins model versions with business outcomes to compare failure rates, retry rates, and task duration.

Figure 4: Closed-loop Physical AI Episode analysis. Doris organizes multisensor evidence, model context, and business outcomes into a reusable data feedback loop.

Agentic AI: Using real-time analytics to connect models with business context​

Large language models can understand general knowledge, but they do not inherently know the enterprise's current state. They cannot know whether an order has been canceled, whether inventory is sufficient, whether a device is online, whether a customer still has permission, or whether a particular anomaly is currently surging without access to current enterprise data.

The challenge for an agent is therefore broader than generating SQL. It needs a continuous supply of fresh, trustworthy, explainable, and permission-compliant data context.

Real-time writes: Bringing business changes directly into the open lakehouse​

Doris integration with Iceberg has already progressed from querying a data source to writing efficiently to an open data foundation. INSERT, INSERT OVERWRITE, CTAS, and row-level updates are already available. Doris 5.0 will bring the same write capabilities to Paimon and add Variant writes for both Iceberg and Paimon.

Users will be able to use unified SQL to write structured and Variant data directly to open tables. They will not need to write the data to Doris internal tables first or build another synchronization pipeline. Real-time business events, tool-call parameters, and continuously changing model results can be stored directly as open data assets.

Within the execution pipeline, Doris BEs generate data files through distributed parallel execution. The FE coordinates snapshots and atomic commits, and it handles idempotent retries and failure cleanup. Doris manages efficient writes, commit consistency, and schema evolution through a common implementation, so applications do not need to assemble open-table transactions themselves.

Real-time analytics: Building agent context from the latest open data​

Orders, device states, behavioral events, and model-derived results written to Iceberg or Paimon can be shared among different compute engines. Doris queries the latest data directly, combines it with real-time business facts in internal tables, and applies cross-table joins and permission filtering to construct fresh, verifiable business context for agents.

For event streams with stricter freshness requirements, the new Fluss Catalog uses Union Read to combine real-time increments in Fluss with historical data tiered to Paimon and queries them as a single table. The resulting agent context includes both the complete history and changes that have just occurred.

Figure 5: A real-time Agentic AI feedback loop using open formats selected according to workload requirements. Doris provides read, write, and Variant capabilities for Iceberg or Paimon selected by the user, along with read and multimodal search capabilities for Lance. It then constructs agent context through unified SQL and MPP OLAP.

Multimodal lakehouse use cases with Doris​

Scenario 1: Real-time decision-making for customer service agents​

Knowledge documents, product images, and samples of similar faults are stored in Lance. Orders, support tickets, and historical service data are stored in Iceberg and Paimon. The latest inventory, warranty, and customer status data is loaded into Doris internal tables.

After an agent receives a customer question, Doris performs semantic retrieval, keyword matching, permission and status filtering, then joins the results with repair success rates and inventory data. The result is more than "the most similar document." It is a resolution that the current customer can act on.

Scenario 2: Selecting high-value Episodes for Physical AI​

Autonomous driving or robotics teams store video, point clouds, and embeddings in Lance. They write continuously changing model outputs and task results to Paimon as Variant data, while retaining historical evaluation results in Iceberg. These teams need to answer a question beyond "Which Episodes are similar?" They need to know, "Which Episodes are worth investing in for annotation, training, and review?"

After the query pipeline described above filters the Episodes, high-confidence anomalies enter training and regression datasets. Episodes with inconsistent evidence go to manual review. Failure and intervention rates aggregated by model version then show whether a new strategy has improved a specific scenario.

Scenario 3: From similar defects to root-cause analysis in industrial quality inspection​

A quality engineer uploads an image of a defect. Doris retrieves visually similar samples from Lance, matches defect descriptions in Variant and full-text fields, and joins the results with continuously updated equipment events in Paimon and with batch, process, and repair history in Iceberg.

The system then determines whether defects are occurring in clusters, which equipment and batches are affected, how accurately different model versions identify the defect, and whether yields recover in subsequent batches. This expands a similar-image search into a complete closed-loop quality analysis.

Conclusion: Keep data in open formats and use Doris for search and analytics​

Doris has one primary direction for the multimodal lakehouse: extend the unified search and real-time analytics capabilities already proven on internal tables to data in Iceberg, Paimon, Lance, and Fluss. Data can remain in open formats for search and analytics. Structured data, Variant data, full-text data, and vectors do not need to reside in separate systems. Doris will provide the unified execution and serving layer above this data:

• Use unified SQL to access Doris internal tables and multiple open formats.
• Use Variant, full-text search, vector search, and structured filtering to query different forms of data.
• Use MPP OLAP to turn retrieved results into business metrics and decisions.
• Use real-time writes to supply the latest business state that agents otherwise lack.
• Use unified permission filtering so applications and agents see only the data they are authorized to access.

A series of articles about Doris 5.0 will examine these capabilities for open formats such as Iceberg, Paimon, Lance, and Fluss. The articles will cover their technical principles, performance benchmarks, and representative implementation scenarios.

The multimodal lakehouse capabilities for Iceberg, Paimon, and Lance have been merged into the Doris 4.2 release branch and are scheduled for official release with version 4.2 at the end of September. Additional capabilities, including Fluss support, have also entered the Doris main branch and are planned for release with Doris 5.0 at the end of the year.

dbt-for-apache-doris 1.1.0 was released on August 12, 2026. It targets dbt Core 1.12, manages Apache Doris asynchronous materialized views through the standard dbt materialized_view materialization, and covers the everyday workflows: Table, View, Incremental, Snapshot, Seed, and Data Test.

python -m pip install "dbt-for-apache-doris==1.1.0"

The rest of this post covers what each of the two systems is responsible for, what the adapter supports today, and five demos that run a full pipeline in Doris.

1. dbt, Doris, and what 1.1.0 adds​

1.1 What is dbt​

dbt (data build tool) is an open-source data transformation framework that handles the T of ELT inside a data warehouse. Every model is a SQL SELECT statement, and models reference each other with ref(). dbt compiles them into the target database's table creation and write statements, runs them in dependency order, and materializes each one as a table, view, incremental table, or snapshot according to its configuration.

It exists because the transformation layer spent years without much engineering discipline:

• SQL is scattered across schedulers, repos, and personal machines. Dependencies are maintained by hand, and nobody can say for certain what a change will break.
• Nothing is tested automatically, so a wrong metric usually surfaces when someone in the business questions a number on a report.
• There is no version control, no code review, and no separation between development and production, which makes changes hard to trace afterwards.
• The same boilerplate for creating tables, writing incrementally, and keeping history gets rewritten in every project.

dbt's answer is models as code, a dependency graph derived from ref(), declarative Data Tests, built-in materializations (Table, View, Incremental, Snapshot), generated documentation and lineage, and a package ecosystem of Jinja macros such as dbt_utils.

1.2 What dbt and Doris solve together​

dbt does not compute anything itself. Every transformation is pushed down to the target database, so that database sets the ceiling for the project. Apache Doris is an MPP real-time analytical database that speaks the MySQL protocol and runs aggregations, multi-table JOINs, and window functions in place. dbt-for-apache-doris connects the two: dbt organizes, orchestrates, and validates the transformation logic; Doris stores the data and executes the SQL.

Putting them together settles a few things that data teams otherwise handle by convention. Table conventions become part of the code, since Doris Key models, partitions, buckets, and table properties are declared in the model's config() and go through version control and code review with everything else, rather than living in someone's memory. Late-arriving data gets a defined path too: a write to a Doris Unique Key table is an Upsert, and dbt Incremental merge sits directly on that semantic, so replaying the same batch produces the same table and a failed job can simply be run again.

Pre-aggregation stops being a separate system. Doris asynchronous materialized views join the dbt dependency graph through the standard materialized_view materialization, with build and refresh strategy in the model configuration, so they change and get tested alongside the models they read from. The computation never leaves Doris. dbt only ships SQL, so no second engine is needed to move data around. Quality checks run in the same place: Data Tests, Unit Tests, and Model Contracts all execute against Doris at build time, which is a better moment to catch a broken metric than the morning after a report goes out.

1.3 Adapter feature support​

1. Materializations

   Capability	Status	Current scope and boundaries
   Table	✅	Duplicate Key CTAS; configurable HASH bucketing and bucket count, RANGE / LIST partitioning, table properties, Contracts, docs, Grants, and Hooks. Unique Key tables are created by Incremental merge
   View	✅	Standard lifecycle, Contracts, docs, Grants, and Hooks; switching relation types is not zero-downtime
   Incremental	✅	Four strategies, append, merge, insert_overwrite, and microbatch, plus every on_schema_change mode (strategy details in the next table)
   Snapshot	✅	check / timestamp strategies, hard-delete handling, schema evolution, atomic replacement and recovery; jobs targeting the same relation must be serialized by the scheduler
   Materialized view	✅	Standard dbt materialized_view, implemented as a Doris Async MV; covers the build / refresh lifecycle, task waiting, configuration changes, atomic replacement and recovery; dbt jobs targeting the same relation must be serialized
   Seed	✅	CSV loading, type inference, column_types, and ref()
   Ephemeral	✅	Compiled and inlined by dbt Core

2. Incremental strategies and Doris target tables. When no strategy is specified explicitly, models configured with a unique_key use merge and all others use append.

   Strategy	Doris target table	Behavior
   append	Duplicate Key table	Appends rows with INSERT INTO
   merge	Unique Key table (MoW or MoR)	Full-row INSERT INTO Upsert based on Unique Key semantics; requires unique_key, does not generate MERGE INTO
   insert_overwrite	Any writable Doris table	INSERT OVERWRITE of the whole table, specified partitions, or dynamic partitions; does not accept unique_key
   microbatch	Duplicate Key table with exact RANGE partitions	Each dbt time window overwrites one named partition; supports hour / day / month / year windows, and batches run serially

3. dbt engineering capabilities

   Capability	Status	Current scope and boundaries
   Sources and freshness	✅	loaded_at_field, filter, and loaded_at_query; cross-database access is implemented as database-as-schema, not through an External Catalog
   Data Test	✅	Singular, Generic, and Ephemeral tests, plus store_failures
   Unit Test	✅	Inline and CSV fixtures, case-insensitive column names, invalid-input validation, reserved-word quoting, and data type fixtures adapted to Doris
   Model Contract	✅	Column name and type constraints for Table, View, and Incremental; not mapped to database primary keys or NOT NULL constraints
   Persisted docs	✅	Table / column comments for Table, View, Incremental, Snapshot, Seed, and Async MV; updating a View comment, or a comment that contains both kinds of quotes, may require a rebuild or full refresh
   Grants	✅	Reconciles Doris table-level privileges for Table, View, Incremental, Seed, Snapshot, and Async MV, supporting user and user@host grantees; role grantees are not handled
   Hooks	✅	pre-hook and post-hook for every materialization; Doris does not provide transactional rollback for hook side effects
   Metadata and dbt docs catalog	✅	Relation discovery and documentation catalog for Doris databases, tables, views, columns, comments, and Async MVs
   Cross-database Sources	✅	References Sources in other Doris databases, including definitions that only declare a database; three-part External Catalog names are not supported

1.4 What's new in 1.1.0​

1.1.0 is the first release of dbt-for-apache-doris from its own repository, and the first published to PyPI. It targets dbt Core 1.12 on Python 3.10 and above, manages Doris Async Materialized Views through the standard dbt materialized_view materialization, and ships with explicit version constraints and a stated support boundary.

A materialized_view model maps to a Doris Async MV, and the build mode, refresh strategy, and task waiting are all written in the model configuration:

{{ config(
    materialized='materialized_view',
    refresh_trigger='manual',
    wait_for_refresh=true
) }}

select order_date, sum(amount) as sales
from {{ ref('orders') }}
group by order_date

2. Five demos​

Each demo starts from a business question:

The demos are selected and adapted from the Snowflake Labs data-eng-bench scenarios (the original project is licensed under the Apache License 2.0). We rewrote the profiles, SQL, fixtures, verifiers, and notebooks for Apache Doris and dbt-for-apache-doris.

Demo repository: https://github.com/velodb/dbt-for-apache-doris/tree/main/examples

Two of them are walked through below. For the other three, customer geography analysis, ad data integration, and customer history tracking, the Doris dbt demos README has the business description, the model list, and the exact commands.

2.1 Demo 01: Order summary and asynchronous materialized views​

The demo starts from 6 orders in Doris, drops the CANCELLED, RETURNED, and FAILED ones, and builds daily_order_summary. Data Tests check that dates are unique and key columns are not null, and monthly_order_summary_mv then aggregates the daily table through ref().

Figure 1: The chain from raw orders to the monthly asynchronous materialized view. The top row is what dbt handles: model dependencies, materialization, and tests. The bottom row is what Doris provides for the table model and the materialized view.

Three orders survive the filter, for a monthly revenue of 220.20. dbt owns the model dependencies (source() through ref()), the materializations, and the Data Tests. Doris decides how the two result tables are stored and how fast they answer:

2.2 Demo 04: Incremental updates for late-arriving orders​

The demo writes orders 101, 102, and 103, then receives two late events: the amount of order 101 is corrected from 100.00 to 125.00, and a new order 104 arrives.

Two steps absorb them:

1. order_version_history uses window functions to identify the current version of each order.
2. incremental_daily_sales takes order_id as its business key and runs Incremental merge on top of the Upsert semantics of the Doris Unique Key model.

Figure 2: Late events reach the source table, then version identification and Incremental merge update the current orders; a re-run with no new data leaves the result alone. The top row is what dbt handles, the bottom row is what Doris provides.

Run it again with no new data and nothing moves. incremental_daily_sales holds 4 orders, order 101 now at 125.00, and revenue for August 1 in the downstream daily_sales_summary becomes 245.00. The same project also builds data quality, arrival latency, and revenue reconciliation models. dbt's part is the incremental model definition, the is_incremental() filter, and the uniqueness test on order_id. The "no duplicates, safe to re-run" property comes from Doris:

All five demos run as JupyterLab notebooks, one step per cell, so you can stop at any cell and look at what Doris wrote. They live in the repository's examples/ directory, with examples/doris-demos as the entry point; its README has the full run instructions.

Links​

• GitHub repository: https://github.com/velodb/dbt-for-apache-doris
• 1.1.0 release: https://github.com/velodb/dbt-for-apache-doris/releases/tag/v1.1.0
• PyPI: https://pypi.org/project/dbt-for-apache-doris/1.1.0/
• Demo entry point: https://github.com/velodb/dbt-for-apache-doris/tree/main/examples/doris-demos
• Apache Doris all-in-one image: https://doris.apache.org/community/developer-guide/all-in-one-image

AI agent logs create wide, fast-changing JSON payloads that are difficult to analyze with predictable latency. We compare how Apache Doris, ClickHouse, Elasticsearch, and OpenSearch perform in such scenario. see more

Apache Doris offers native CDC for PostgreSQL, replacing separate CDC, Kafka, Flink, or Spark components with one built-in streaming job. see more

Kwai migrated its company-wide A/B testing metrics pipeline from Spark to Apache Doris. The rebuild delivered 145x faster metrics computation and cut resource consumption by 72%. see more

Apache Doris adds Python UDF in 4.1.3. This enables teams to create and call Python functions directly in SQL, bringing Pandas, PyArrow, and the rest of the Python ecosystem into the Doris query path. see more

Meituan replaced a multi-engine analytics stack of Hadoop, Kylin, and Druid with a single Apache Doris platform spanning 300+ clusters and tens of PB of data. see more

Apache Doris is the fastest open source database project built for real-time analytics. People often first encounter Doris through architecture diagrams, SQL, performance benchmarks, and technical documentation. But beyond its technical capabilities, we also want Doris to have a more vivid and approachable identity, one that helps more people understand it, remember it, and connect with the community.

Today, we are excited to introduce three new friends of Apache Doris: the D-Crew.

D-Crew is the Apache Doris mascot family, made up of three members: Pip, Dori, and Flux. Each grew out of one of the three geometric elements in the Apache Doris logo. They are three distinct characters, but together they represent the capabilities that make Apache Doris powerful.

Meet the D-Crew​

D-Crew offers a lighter, more visual way to get to know Doris, understand how it works, and follow the story of this evolving open source project.

Flux​

Flux is the streamlined green character and D-Crew's data explorer.

Flux connects different data worlds, including lakehouses, search, vectors, open data formats, and more data modalities to come. Flux makes data discoverable, understandable, and easier to use in different ways. This reflects how Doris is bringing these data capabilities together to provide context and analytics for Agents.

Dori​

Dori is the teal capsule and the character closest to the Doris core engine.

Dori is the dependable foundation. Low-latency queries, high-concurrency access, complex analytics, and multi-tenant workloads all rely on Dori's steady support behind the scenes. Dori's reliability represents the continued evolution of Doris as a powerful foundation for data analytics.

Pip​

Pip is the purple triangle. It looks like a play button and the starting point of a query.

Starting queries, running SQL, and launching real-time analytics are all Pip's domain. Pip represents the interface between Doris and its users and applications. It starts with the standard SQL interface and extends to capabilities designed for AI Agents, including a semantic layer, CLI, MCP, and Skills.

Explore Apache Doris with D-Crew​

Pip, Dori, and Flux are more than three separate characters. Their different roles and the way they work together offer a more intuitive view of how Apache Doris supports a range of use cases.

Real-time Analytics​

In a Real-time Analytics use case, Flux connects to real-time data, Dori handles high-concurrency and high-throughput data processing and storage, and Pip delivers low-latency query responses. Together, they create an end-to-end, low-latency data analytics pipeline.

Lakehouse Analytics​

In a Lakehouse Analytics use case, Flux connects directly to open data ecosystems such as Iceberg, Paimon, and Delta Lake. Dori efficiently analyzes structured data, text, and vectors through one engine, while Pip presents the resulting hybrid analytics and retrieval results to users.

Agent Observability​

In an Agent Observability use case, Flux ingests observability data such as Log, Metric, and Trace data. Dori uses Variant and time-series data capabilities to organize and process JSON and time-series data efficiently. Pip presents detected system anomalies and behavioral patterns in real time, helping organizations evaluate Agent performance more efficiently and keep track of system health.

Agentic Data Engine​

Apache Doris is evolving into a data analytics system for Agents. In an Agentic Data Engine use case, Flux connects an Agent to real-time, trusted data and finds the context it needs. Pip receives the Agent's query and analytics requests, while Dori handles the computation and processing quickly behind the scenes.

From real-time analytics and lakehouses to hybrid search and Agent-oriented data analytics, the three characters will appear together in different ways. Through these visual stories, we hope to make it easier to understand what Apache Doris can do and how it works with real-world data.

Join the Community​

If you see Pip, Dori, and Flux in our documentation, blog posts, Meetups, or on stickers, remember to say hello.

Welcome to the Apache Doris community, D-Crew. And everyone else, you are welcome to join us too.

Apache Doris 4.1 uses search(), inverted indexes, BM25 scoring, and standard SQL to unify full-text log search with real-time analytics for observability, SIEM, and product search workloads. see more

Apache Doris now supports Rust-native streaming ingestion from Apache Iggy through a new sink connector built with LaserData. see more

Real-time database categories now span across OLTP, OLAP, real-time OLAP, HTAP, and Databricks' new LTAP, but each makes different trade-offs across freshness, query latency, concurrency, and updates. see more

StepFun built StepTrace, a PB-scale agent observability platform on Apache Doris, to analyze agent traces, costs, search, evaluation loops, and infrastructure behavior in real time. see more

Tuhu Car rebuilt its fragmented Hive, HBase, MySQL, and Trino OLAP stack with Apache Doris, improving audience segmentation, user profiling, BI analysis, and real-time analytics. see more

AI agents need real-time analytics, not just vector search. Apache Doris unifies both capabilities, offering native hybrid search in one real-time engine built for agent workloads. see more

Apache Doris 4.1 outperforms ClickHouse and DuckDB on ASOF JOIN across all 11 benchmark scenarios. see more