Lance Catalog

note

This is an experimental feature.

Lance Catalog is supported starting from Apache Doris 4.2.

Lance is a columnar data format designed for analytics and AI workloads. Doris can use a Lance Catalog to discover databases and tables in a Lance Namespace and directly query Lance datasets stored on a local file system or S3-compatible object storage.

Doris currently provides read-only access to Lance. Creating, writing, updating, or deleting Lance tables is not supported.

Feature Overview

Feature	Support
Filesystem Catalog	Supports warehouses on a local file system, file://, or s3://
REST Catalog	Supports Lance REST Namespace with no authentication, Bearer Token, API Key, or custom HTTP headers
Metadata access	Supports SHOW DATABASES, SHOW TABLES, and DESC
Data queries	Supports column pruning, parallel Lance Fragment scans, and snapshot-consistent reads of the current version
Predicate pushdown	Supports pushing compatible scalar predicates down to Lance
File TVFs	Supports querying Lance datasets directly through s3() and local()
Vector search	Uses physical Lance index segments as parallel splits, keeps uncovered Fragments as Flat Search splits, and performs a Doris global Top-K merge
Writing to Lance	Not supported
Time Travel	Not supported
Full-Text Search / Hybrid Search	Not supported

Lance Version and Compatibility

The Doris BE data reader is built with lance-c v0.1.6. In Doris, this version is bound to Lance 9.1.0-beta.3 at Lance commit e934cc2c. The lance-c and Lance Rust crate versions identify the reader implementation integrated with Doris. They are different from the Lance data_storage_version recorded in a dataset.

The following table describes the file-format compatibility of this reader:

data_storage_version	Read support	Notes
0.1 / legacy	Supported	Original Lance file format.
2.0 (writer-option alias 0.3)	Supported	An earlier version of the Lance v2 file format.
2.1 / stable	Supported; default stable format	In the embedded Lance version, the stable writer option and the default format for new datasets both resolve to 2.1.
2.2	Supported	The embedded Lance version treats this as a stable format, but it is not the default writer format.
2.3 / next	Experimental; not guaranteed	The embedded Lance version marks 2.3 as unstable, and the next writer option resolves to 2.3.
A later or unknown version	Not supported	Opening or scanning the dataset may fail with an unsupported storage-version error.

Lance SDK release numbers and file-format versions are independent. A dataset written by an older or newer Lance SDK is readable only when its storage format, required table feature flags, index format, and Arrow/Lance data types are all understood by the versions embedded in Doris. Consequently:

• Doris is expected to read datasets written with the 0.1, 2.0, 2.1, and 2.2 storage formats, subject to the type limitations documented below.
• Forward compatibility is not guaranteed. A dataset written or modified by a later Lance release may be unreadable if it uses a newer storage format, an unknown required manifest feature, a newer index format, or an unsupported extension type.
• For datasets that must remain readable by this Doris release, use the current default stable format, 2.1, and do not use next. If a newer writer or optional Lance feature is introduced, validate the resulting dataset with the target Doris release before using it in production.

Configure a Catalog

Syntax

CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "<filesystem|rest>",
    {CatalogProperties},
    {StorageProperties},
    {CommonProperties}
);

Common Properties

Property	Required	Default	Description
type	Yes	-	Must be lance.
lance.catalog.type	No	filesystem	Catalog type. Valid values are filesystem and rest.
lance.namespace.parent	No	Empty	Limits access to the specified Lance Namespace and its child Namespaces. With the default delimiter, for example, production$analytics represents a two-level Namespace.
lance.namespace.delimiter	No	$	Delimiter used to parse lance.namespace.parent. It is also passed to the REST Namespace client. This property does not change how multilevel Namespaces are displayed in Doris.
lance.namespace.root_database	No	default	Doris database name to which the root Lance Namespace is mapped.

Filesystem Catalog

A Filesystem Catalog discovers Lance Namespaces and tables directly from a warehouse directory.

Property	Required	Description
warehouse	Yes	Root path of the Lance warehouse. Local absolute paths, file:// URIs, and s3:// URIs are supported.

Use S3-Compatible Object Storage

The following example creates a Catalog for MinIO:

CREATE CATALOG lance_catalog PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "s3://my-bucket/lance",
    "s3.endpoint" = "http://127.0.0.1:9000",
    "s3.access_key" = "admin",
    "s3.secret_key" = "password",
    "s3.region" = "us-east-1",
    "use_path_style" = "true"
);

When accessing AWS S3, you can omit s3.endpoint and configure credentials, Region, and Path Style for your environment.

Use a Local File System

CREATE CATALOG lance_local PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "/data/lance"
);

For a local file system, warehouse must be an absolute path. The FE must be able to read Namespace and table metadata through this path, and each BE that executes a query must be able to access the data through the same path. In a multi-node deployment, mount the same shared directory on all relevant FE and BE nodes.

REST Catalog

A REST Catalog obtains Namespaces, table locations, and storage access parameters through Lance REST Namespace. A REST Catalog neither requires nor permits the warehouse property.

Property	Required	Default	Description
lance.rest.uri	Yes	-	REST service URI. It must use http:// or https://.
lance.rest.security.type	No	none	Authentication type. Valid values are none, bearer, and api_key.
lance.rest.bearer-token	Yes for Bearer authentication	-	Bearer Token.
lance.rest.api-key	Yes for API Key authentication	-	API Key sent in the x-api-key header.
lance.rest.header.<header-name>	No	-	Custom HTTP header sent to the REST service. Use the dedicated authentication properties above for authentication headers.

The following example creates a REST Catalog using a Bearer Token:

CREATE CATALOG lance_rest PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "rest",
    "lance.rest.uri" = "https://lance.example.com",
    "lance.rest.security.type" = "bearer",
    "lance.rest.bearer-token" = "your-token"
);

For API Key authentication, replace the authentication properties with:

"lance.rest.security.type" = "api_key",
"lance.rest.api-key" = "your-api-key"

If the REST service returns temporary storage credentials, Doris uses those credentials to access the corresponding Lance table. You can also configure s3.endpoint, s3.access_key, s3.secret_key, s3.region, and use_path_style in the Catalog as the default object storage access parameters.

caution

The current BE Reader does not support Lance tables whose versions are managed by REST Namespace (Managed Versioning).

Namespace Mapping

Lance supports multilevel Namespaces, while a Doris Catalog represents each Namespace as a database name:

Lance Namespace	Doris Database Name
Root Namespace	default; configurable through lance.namespace.root_database
doris	doris
doris.analytics	doris.analytics

Doris joins the levels of a multilevel Namespace with . to form a database name. Use backticks when referencing a database name that contains .:

SHOW TABLES FROM lance_catalog.`doris.analytics`;

SELECT *
FROM lance_catalog.`doris.analytics`.user_features;

Use lance.namespace.parent to limit a Catalog to a Namespace subtree. For example:

CREATE CATALOG lance_analytics PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "s3://my-bucket/lance",
    "lance.namespace.parent" = "production$analytics",
    "s3.region" = "us-east-1"
);

Doris then displays only the tables and child Namespaces below production.analytics.

Query Lance Tables

After creating a Catalog, you can browse and query Lance tables in the same way as other external tables:

SHOW DATABASES FROM lance_catalog;

SHOW TABLES FROM lance_catalog.default;

DESC lance_catalog.default.user_profiles;

SELECT user_id, name, age
FROM lance_catalog.default.user_profiles
WHERE age >= 18
ORDER BY user_id
LIMIT 100;

You can also load data from Lance into a Doris internal table:

INSERT INTO internal.demo.user_profiles
SELECT user_id, name, age
FROM lance_catalog.default.user_profiles;

For a regular Catalog query, Doris pins a Lance dataset version during planning and generates scan tasks by Fragment. A query therefore reads a consistent snapshot, while multiple Scanners can read different Fragments in parallel without every Scanner repeatedly scanning the entire dataset.

Type Mapping

Lance / Arrow Type	Doris Type	Description
bool	BOOLEAN
int8	TINYINT
uint8	SMALLINT	Losslessly widened unsigned integer
int16	SMALLINT
uint16	INT	Losslessly widened unsigned integer
int32	INT
uint32	BIGINT	Losslessly widened unsigned integer
int64	BIGINT
uint64	LARGEINT	Losslessly widened unsigned integer
float16	FLOAT	Widened to a 32-bit floating-point value
float32	FLOAT
float64	DOUBLE
decimal128(P,S)	DECIMAL(P,S)	Maximum precision is 38
decimal256(P,S)	DECIMAL(P,S)	Maximum precision is 76
utf8, large_utf8	TEXT
binary, large_binary	VARBINARY(2147483647)
fixed_size_binary(N)	VARBINARY(N)	Preserves the fixed byte width
date32(day), date64(ms)	DATE	A date64 value must represent a complete calendar day
time32(s)	TIME(0)
time32(ms)	TIME(3)
time64(us), time64(ns)	TIME(6)	Nanosecond precision is truncated to microseconds
Timezone-naive timestamp(s)	DATETIME	Not converted according to the Session Time Zone
Timezone-naive timestamp(ms)	DATETIME(3)	Not converted according to the Session Time Zone
Timezone-naive timestamp(us), timestamp(ns)	DATETIME(6)	Nanosecond precision is truncated to microseconds
Timezone-aware timestamp	TIMESTAMPTZ(0-6)	Preserves the instant and displays it in the Doris Session Time Zone
struct	STRUCT	Child fields are mapped recursively
list, large_list, fixed_size_list	ARRAY	Element types are mapped recursively
map	MAP	Key and value types are mapped recursively

The following types are not currently supported:

• Arrow null and duration.
• Arrow/Lance Extension types with ARROW:extension:name metadata, including Lance Blob v2, Arrow JSON Extension, and Lance BFloat16 Extension.
• Complex types whose child types cannot be mapped recursively.
• Arrow Dictionary types that preserve the Dictionary marker.

For an unsupported top-level column, DESC on a Catalog table and DESC FUNCTION on a Lance file TVF both preserve the column and display unknown type: UNSUPPORTED_TYPE. If any child of a complex type cannot be mapped, the whole top-level complex column is marked unsupported. Queries can still project only supported columns. Doris reports an error during analysis when SQL projects an unsupported column. For example:

SELECT * EXCEPT(blob_col, json_col)
FROM lance_catalog.default.all_types;

note

Some Lance Java SDK versions may lose the Dictionary marker while reading a Schema and expose a Dictionary column as its physical index type. This behavior does not mean that Doris supports the logical Dictionary values and must not be relied upon.

Predicate Pushdown

Doris converts semantically compatible predicates into Substrait expressions and passes them to Lance for evaluation during reads. The Doris BE does not evaluate a condition again after the entire condition has been pushed down. Conditions that cannot be pushed down safely remain in Doris.

Data Types Supported for Pushdown

Lance / Arrow Type	Pushdown Support
bool	Equality, null checks, and logical operations; ordering comparisons are not supported
int8/16/32/64	Supported
uint8/16/32/64	Supported
float32/64	Supported
decimal128	Precision 1 through 38, with Scale from 0 through Precision
utf8, large_utf8	Supported
date32(day)	Supported
Timezone-naive timestamp(s/ms/us)	Supported

Predicates on other readable types, including float16, decimal256, Binary, date64, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex types, currently remain in Doris.

Operators Supported for Pushdown

SQL Predicate	Pushdown Condition
=, !=, <>, <, <=, >, >=	Direct comparison between a column and a constant. The constant may be on the left side.
<=>	Direct null-safe equality comparison between a column and a constant; preserves a non-NULL, two-valued result inside NOT, AND, or OR
IN, NOT IN	Non-empty constant list that does not contain NULL
IS NULL, IS NOT NULL	Direct column reference
AND	Top-level conjuncts can be pushed down independently, with unsupported conjuncts retained in Doris
OR	Both branches must be fully convertible
NOT	The operand must be fully convertible

The following forms are generally not pushed down:

• Functions or arithmetic expressions applied to a column.
• An empty IN list or an IN list containing NULL.
• An OR or NOT expression in which only part of the expression can be converted.
• A data type or constant value that cannot be converted to Lance without loss.

Use lancePushdownPredicate in EXPLAIN to inspect the conditions that are actually pushed down:

EXPLAIN
SELECT user_id
FROM lance_catalog.default.user_profiles
WHERE age >= 18 AND country IN ('CN', 'US');

Query Lance with File TVFs

If you only need to read a Lance dataset at a known path, you can use the s3() or local() TVF without creating a Catalog. uri or file_path must point to the root directory of a Lance dataset, rather than an internal data file.

S3 TVF

SELECT user_id, name
FROM s3(
    "uri" = "s3://my-bucket/lance/user_profiles.lance",
    "s3.endpoint" = "http://127.0.0.1:9000",
    "s3.access_key" = "admin",
    "s3.secret_key" = "password",
    "s3.region" = "us-east-1",
    "use_path_style" = "true",
    "format" = "lance"
)
WHERE user_id > 100;

For an S3 TVF, the FE obtains the Schema, current version, and Fragment list. Doris pins that version and scans its Fragments in parallel.

Local TVF

SELECT user_id, name
FROM local(
    "file_path" = "/data/lance/user_profiles.lance",
    "backend_id" = "10001",
    "format" = "lance"
);

file_path is passed as written to the Lance Reader on the target BE. Doris does not prepend user_files_secure_path or expand this path as a Glob, so it must point directly to the root directory of one Lance dataset that the target BE can access. An absolute path is recommended.

Local TVF Schema discovery and execution each open the latest dataset version independently. The version resolved during Schema discovery is not currently pinned for the subsequent scan. If the dataset changes between query analysis and execution, the discovered Schema and scanned snapshot can differ. Avoid modifying the dataset while a Local TVF query is being analyzed and executed. The current Local Lance TVF uses one Scanner.

Lance file TVFs have the following additional limitations:

• Only s3() and local() are supported. Other file TVFs, such as HDFS and HTTP, are not currently supported.
• path_partition_keys is not supported.
• One TVF path can represent only one Lance dataset.
• DESC FUNCTION can display a Schema containing unsupported types, but SQL cannot project unsupported columns.

Vector Search

vector_search() is a relational TVF that performs Top-K search on a vector column in a Lance table. It can use an existing Lance vector index or perform Flat Search.

Syntax and Example

SELECT user_id, label, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10",
    "offset" = "3",
    "metric" = "l2",
    "nprobes" = "20",
    "refine_factor" = "10",
    "filter" = "category = 'book'",
    "use_index" = "true"
)
ORDER BY _distance ASC, user_id;

The relation schema of vector_search() contains all columns from the Lance source table plus the _distance column generated by the Lance Scanner for the nearest-neighbor query. The final SQL result contains only columns projected by SELECT. Doris exposes _distance as FLOAT. It is a distance, not a generic similarity score: a lower value means that two vectors are closer. The source table must not already contain a column named _distance. A SQL relation does not guarantee final display order, so explicitly specify ORDER BY _distance ASC when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance.

table must parse as exactly three catalog.database.table name parts. A multilevel Lance Namespace maps to one Doris database name containing ., so quote the database part with backticks. For example, use the following value for table items in Namespace doris.analytics:

"table" = "lance_catalog.`doris.analytics`.items"

Do not use the unquoted form lance_catalog.doris.analytics.items; it parses as four name parts and is rejected. A table-only name or database.table name is also rejected.

Parameters

Parameter	Required	Default	Description
table	Yes	-	Fully qualified, three-part catalog.database.table name. If a multilevel Namespace maps to a database name containing ., quote the database part with backticks. It must identify a table in a Lance Catalog, and the user must have the SELECT privilege on the table.
column	Yes	-	Vector column name. fixed_size_list<float16|float32|float64|uint8|int8> is currently supported.
query_vector	Yes	-	JSON number array. Its dimension must match the vector column, and each value must be representable by the vector element type.
top_k	No	10	Number of results returned after skipping offset. It must be a positive integer.
offset	No	0	Number of nearest neighbors skipped inside the vector search. It must be a non-negative integer. top_k + offset must not exceed the maximum unsigned 32-bit integer.
metric	No	Metric of the matching index; without an index, hamming for uint8 and l2 for other supported types	Distance metric: l2, cosine, dot, or hamming. dot_product is an alias for dot. uint8 vectors support only hamming; the other currently supported vector element types support l2, cosine, and dot.
filter	No	-	Lance SQL condition evaluated before vector candidates are generated; that is, a Prefilter.
nprobes	No	Minimum 1, with no maximum	Number of IVF index partitions to probe. It must be a positive integer. When unset, Lance starts with one partition and can probe additional partitions when a Prefilter leaves too few candidates. Setting it explicitly to N fixes both the minimum and maximum number of probes to N.
refine_factor	No	Refinement disabled	Candidate refinement multiplier. It must be a positive integer. When unset, Lance does not recompute distances from the original vectors, so _distance from a quantized index may be approximate. When set to N, Lance first retrieves (top_k + offset) × N candidates, recomputes their exact distances from the original vectors, and reorders them. Setting it to 1 still enables refinement and therefore differs from leaving it unset.
ef	No	floor(1.5 × (top_k + offset))	Candidate width retained during HNSW graph search. It must be a positive integer. If refine_factor is also set, the default is floor(1.5 × (top_k + offset) × refine_factor). It has no effect on non-HNSW indexes.
use_index	No	true	When true, Doris plans compatible physical Lance index segments as indexed splits and keeps uncovered Fragments as Flat Search splits. If no usable compatible index metadata is available, Doris falls back to Fragment splits. When false, Doris creates one split per visible Fragment and forces Flat Search.

These defaults correspond to the Lance Scanner behavior currently integrated with Doris. When metric is omitted, Doris uses the metric configured when a compatible vector index was created. If there is no compatible index, or if "use_index" = "false", uint8 vectors use hamming, while the other currently supported vector element types use l2.

Supported Vector Index Types

The embedded lance-c v0.1.6 explicitly supports the following Lance vector index combinations:

Index type	Description	Main query parameters
IVF_FLAT	IVF partitions with original-vector distance computation inside each partition	nprobes
IVF_SQ	IVF with Scalar Quantization	nprobes, refine_factor
IVF_PQ	IVF with Product Quantization	nprobes, refine_factor
IVF_HNSW_FLAT	IVF with HNSW whose graph nodes retain original vectors	nprobes, ef
IVF_HNSW_SQ	IVF and HNSW with Scalar Quantization	nprobes, ef, refine_factor
IVF_HNSW_PQ	IVF and HNSW with Product Quantization	nprobes, ef, refine_factor

vector_search() only queries indexes. It does not create an index in Doris and does not expose an index-type or index-name parameter. With use_index=true, the FE reads vector-index metadata from the pinned dataset snapshot and selects a logical index compatible with the vector column and metric. It then assigns each physical segment of that logical index that still covers visible data to an indexed Scan Split. Each indexed Split carries the segment UUID and the currently visible Fragments covered by that segment, so the BE searches that specific segment instead of asking Lance to choose an index again.

A logical Lance index can contain multiple physical index segments, and one physical segment can cover multiple Fragments. Fragments not covered by the selected index are not omitted: Doris adds one fallback Split for each such Fragment, which uses Flat Search. If the FE cannot construct a usable index-segment plan, it falls back to Fragment-level splits. With use_index=false, Doris skips index metadata planning and forces Flat Search for every visible Fragment. Flat Search is not an ANN index type; Lance must directly read and compare vectors.

Prefilter and Post-Filter

The TVF filter parameter is a Prefilter. Doris passes the string to the Lance Scanner for each search Split, and Lance evaluates it before ANN or Flat Search generates candidates:

SELECT user_id, category, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10",
    "filter" = "category = 'book'"
)
ORDER BY _distance ASC, user_id;

Lance reads and evaluates columns referenced only by filter internally. If such a column is not referenced by SELECT or another Doris expression, it does not have to be returned to Doris.

An outer WHERE is a Post-filter. The optimizer moves it into the Doris Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance generates candidates for each search Split and before Doris performs its local and global TopN operations.

SELECT user_id, category, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10"
)
WHERE category = 'book'
ORDER BY _distance ASC, user_id;

Consequently, an outer WHERE only filters candidates that have already been generated and does not cause Lance to replenish them. The final result may contain fewer than top_k rows. If the filter must reduce the vector search space and nearest neighbors must be selected from the filtered rows, use the TVF filter parameter.

Current Execution Model

vector_search() uses distributed candidate search instead of one Scanner for the entire dataset. Its Split boundary depends on index coverage:

1. During planning, the FE pins a positive Lance dataset snapshot version and reads the visible Fragments in that snapshot. When use_index=true, it also reads vector-index metadata.
2. If a compatible logical vector index has usable segment coverage, each physical index segment that still covers visible Fragments becomes one indexed Scan Split. The Split contains that segment's UUID and the intersection of its Fragment bitmap with the visible Fragments in the pinned snapshot. A Split can therefore contain multiple Fragment IDs.
3. Every visible Fragment not covered by those indexed Splits becomes an independent fallback Fragment Split. This keeps data appended after index creation searchable without requiring the index to be optimized first. If no usable index-segment plan exists, all visible Fragments use Fragment splits. With use_index=false, all visible Fragments use Flat Search splits directly.
4. For query parameters top_k=K and offset=n, every indexed or fallback Split requests at most K+n candidates and does not apply the offset locally. An indexed Split searches only its assigned physical index segment; a fallback Fragment Split performs Flat Search for its Fragment. Lance evaluates the TVF filter before generating candidates, while Doris Scan evaluates an outer WHERE afterward.
5. Doris performs local TopN, Exchange, and global TopN over candidates from all Splits, merging by _distance ASC. Only the global TopN applies offset=n: it skips the first n rows and returns K rows.

A Split-level candidate set therefore only supplies candidates for global merging and is not the final result. Index-segment splits, fallback Fragment splits, and any later Row-ID fetches all use the same pinned snapshot throughout the query. Refreshing index coverage changes how newly appended Fragments are searched, but uncovered Fragments remain part of the result space through Flat Search.

The execution order is:

Pinned dataset snapshot
  -> FE Split planning
       -> Indexed coverage: one Split per physical Index Segment -> ANN Search
       -> Uncovered or unindexed data: one Split per Fragment -> Flat Search
  -> Per Split: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates
  -> Doris Scan Post-filter
  -> Doris local TopN
  -> Exchange
  -> Doris global TopN (applies offset=n and limit=K)
  -> Optional lazy-materialization Fetch

Two-Phase TopN Read and Lazy Materialization

vector_search() can use a two-phase read when experimental_topn_lazy_materialization_threshold is greater than 0, top_k does not exceed the threshold, and at least one top-level column is eligible for deferred reading. The default threshold is 1024. Phase 1 carries only the columns required for candidate filtering and TopN, plus an internal Row Location. After global TopN, Phase 2 reads the other output columns only for the retained rows.

For example, assume the source table has these columns:

Column	Purpose
user_id	Final output column
category	Post-filter column used by the outer WHERE
title, payload	Final output columns
embedding	Lance vector-search column

Run this query with K=10 and n=3:

SET experimental_topn_lazy_materialization_threshold = 1024;

SELECT user_id, title, payload, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10",
    "offset" = "3"
)
WHERE category = 'book';

A typical two-phase column flow is:

Stage or operator	Columns read or output	Description
Lance Split Search	Uses embedding internally; returns _distance, category, and the internal Lance Row ID to Doris	embedding participates in ANN/Flat Search but is not returned as a result column unless SQL projects it. Each Index Segment or fallback Fragment Split produces at most K+n candidates.
Doris Scan Post-filter	_distance, category, and the internal Row Location	Evaluates category = 'book'. A column used by an outer WHERE must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into an internal Row Location, which Fetch resolves to the same pinned snapshot.
Local and global TopN	Phase-1 required columns and the internal Row Location	Global TopN merges by _distance and applies offset=n and limit=K.
Row ID Fetch	Uses the internal Row Location to read user_id, title, and payload	Reads deferred columns for rows retained by global TopN from the same Lance dataset snapshot without rescanning Fragments.
Final Materialize	user_id, title, payload, and _distance	Combines deferred columns with columns retained from Phase 1 to produce the final SQL output.

Phase-1 required columns are not limited to _distance and Post-filter columns. Any column referenced by a Doris expression or operator before global TopN is operative and must be read in Phase 1. For example, adding ORDER BY _distance, user_id makes user_id a Phase-1 column, so it cannot be deferred to the Row-ID Fetch. Nested subcolumn projections are also not currently deferred. A top-level column used only by the final projection can be fetched in Phase 2.

A Prefilter column referenced in the TVF filter differs from an outer WHERE column. Lance uses the former internally during search, so appearing in the filter string alone does not require the column to be returned to Doris. Doris Scan evaluates the latter, so it must be present in Phase 1.

Setting experimental_topn_lazy_materialization_threshold to -1 disables the two-phase read. A single-phase read is also used when top_k exceeds the threshold or no column can be deferred. Single-phase mode returns all query-required output columns from Scan, but vector search still generates candidates in parallel per Index Segment or fallback Fragment Split and Doris still merges a global TopN. It does not become an ordinary Doris full-column table scan. Indexed splits use their assigned physical index segments, while forced or fallback Flat Search directly compares vectors.

Current Limitations and Recommendations

• Lance Catalogs and Lance TVFs are read-only. CREATE TABLE, INSERT, UPDATE, DELETE, TRUNCATE TABLE, and writing data back to Lance are not supported.
• Queries always read the current version selected during planning. SQL cannot select a Version or perform Time Travel.
• For tables containing unsupported column types, explicitly list the columns to read instead of projecting unsupported columns through SELECT *.
• For regular scans, inspect lancePushdownPredicate in EXPLAIN to verify which conditions have been pushed down.
• Create a vector index in Lance that matches the intended query before running indexed vector search. For small datasets or validation, set "use_index" = "false" to perform Flat Search.
• For deterministic vector result ordering, explicitly use ORDER BY _distance ASC and add a unique tie-breaker.
• Use the vector_search() filter parameter when filtering must occur before vector candidates are generated. An outer WHERE filters only the candidates already generated by each search Split and runs before Doris global TopN, so allow for a final result with fewer than top_k rows.
• Use EXPLAIN to inspect lanceSearchFragments and lanceSearchIndexSegments. The former is the number of visible Fragments in the pinned snapshot; the latter is the number of physical Index Segment splits selected by the FE. Additional fallback Fragment splits may also be present.