[Feature](lance) Add vector and scalar index lifecycle management

Parent

Tracked by #66340. #65730 provides read-only index consumption and automatic vector-index selection.

Scope

Create vector indexes.
Create scalar indexes.
Show the number of indexes. Exact logical-index counts remain deferred by the accepted inspection contract; physical entry counts must not be presented as logical-index counts.
Show logical index names, columns, and types for Directory catalogs through SHOW INDEX ([feature](lance) Support SHOW INDEX for Lance Directory tables #66637; [doc](lance) add doc about SHOW INDEX for lance catalog doris-website#4090).
Show index names, UUIDs, columns, types, and dataset versions. Implementation/tests are merged in [feature](lance) Support SHOW INDEX for Lance Directory tables #66637 / [feature](lance) Add lance_index_entries inspection TVF #66671; user documentation for the physical lance_index_entries TVF remains outstanding. REST inspection is outside this delivery scope.
Replace or rebuild an index with the same name.
Drop indexes by name.
Define authorization, concurrency, versioning, failure recovery, and metadata refresh semantics.

Completion criteria

FE statements and metadata APIs are implemented.
BE or service-side index operations are integrated with Lance.
Unit and regression tests cover create, inspect, replace, rebuild, drop, and failures.
User documentation describes syntax, lifecycle semantics, and limitations.
The behavior is verified for vector and scalar index types.

Delivery status — 2026-09-30

This summarizes the existing staged roadmap; it does not broaden the accepted mutation or inspection contract. Completion on branch-4.1 satisfies the parent feature matrix's branch criterion.

Merged read-only inspection: [feature](lance) Support SHOW INDEX for Lance Directory tables #66637 (SHOW INDEX, with [doc](lance) add doc about SHOW INDEX for lance catalog doris-website#4090) and [feature](lance) Add lance_index_entries inspection TVF #66671 (lance_index_entries, with physical UUID/dataset version and FE/BE/regression verification). The physical-TVF website documentation was deferred in [feature](lance) Add lance_index_entries inspection TVF #66671 and is not present in the current catalog guide, so the combined inspection delivery item is not yet fully checked.
Merged mutation foundations: [feature](lance) Add thrift definitions for Lance index entries inspection #67275, branch-4.1: [feature](lance) Add statically validated Lance index DDL surface #67201, branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235, and branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630 provide protocol/static validation, durable job state, admission, job inspection, and guards. They do not complete enabled end-to-end index mutation.
Still open: branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978 (dispatcher), branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC #67754 (resolution/retention), branch-4.1: [feature](lance) Add the cgroup-isolated one-shot worker for Lance index mutations #68668 (isolated worker), and branch-4.1: [test](lance) Prove the IVF_PQ tracer and worker-fault convergence through the isolated worker #68669 (tracer/fault tests). The mutation gate remains off. Scalar lifecycle coverage, acceptance evidence, and user documentation still need completion before marking create/replace/rebuild/drop done.

The issue remains open. The original combined completion criteria still cover the unfinished lifecycle, not only the delivered inspection slice.

@u70b3

@u70b3, would you be available to take ownership of this issue and implement it?

@Gabriel39

Thanks @Gabriel39 — I'd be glad to take ownership of this issue. Please assign it to me.

@Gabriel39

Superseded: This proposal has been replaced by Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle. This version is retained for review history.

Here is a revised design proposal for this issue. It builds on #65730, the Lance Java SDK / lance-c versions pinned by Doris, and the existing Doris pattern for mutable external lake-table catalogs (Iceberg/Hive/Paimon): Doris parses a common SQL surface, dispatches through a catalog-specific operation seam, lets the external format remain the source of truth, and refreshes Doris metadata after a successful external commit.

/cc @Gabriel39 @zhangstar333

1. Goals and Phase-1 boundary

Phase 1 provides index lifecycle management for writable filesystem-backed Lance catalogs:

create vector and scalar indexes;
inspect logical indexes, physical segment metadata, coverage, and the logical index count;
replace/rebuild an index by name;
drop an index by name;
define privilege, concurrency, version, failure, and metadata-refresh behavior.

Lance dataset manifests remain the authoritative index metadata. Doris does not persist a second copy of Lance index definitions in its internal table/index metadata.

Phase 1 uses the Lance Java SDK in FE and is intentionally synchronous. A BE-distributed or asynchronous build path is a follow-up and should not change the SQL contract.

2. SQL surface

Reuse the existing Doris index statements and command chain, extended for Lance external tables:

-- Vector index
CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (vec_col)
    USING IVF_PQ
    PROPERTIES (
        "metric" = "l2",
        "num_partitions" = "256",
        "num_sub_vectors" = "16"
    );

-- Scalar indexes
CREATE INDEX idx ON lance_ctl.db.tbl (col) USING BTREE;
-- BITMAP / INVERTED are also candidates for Phase 1.

-- Full rebuild and atomic replacement by logical name
CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (vec_col)
    USING IVF_PQ
    PROPERTIES (...);

SHOW INDEX FROM lance_ctl.db.tbl;

DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;

Proposed semantics:

USING is required for Lance tables; Doris's internal default-to-INVERTED behavior should not be applied implicitly to an external format.
IF NOT EXISTS and OR REPLACE are mutually exclusive.
CREATE INDEX fails when the logical name already exists; IF NOT EXISTS returns success without rebuilding; CREATE OR REPLACE INDEX performs a full rebuild and maps to Lance replace=true.
Incremental indexing of newly appended fragments (for example via Lance index optimization APIs) is not part of Phase 1.
Phase 1 supports one indexed column per statement. Multi-column/composite indexes can be added later without changing the lifecycle model.
Index names are preserved and matched exactly, following the pinned Lance implementation. Doris should not silently lowercase external index names or merge case-only names.
Only index types verified end-to-end by both index creation and the Doris query path are accepted. Unsupported types and unknown properties fail during analysis instead of being accepted syntactically and rejected later.
The exact per-type property/type/metric matrix and SDK mapping will be documented and tested in the PR. The implementation should use a Lance-specific type/definition model rather than adding Lance physical types to Doris's persisted internal catalog.IndexType.

3. SHOW INDEX and the Lance index model

Lance has two levels of index metadata:

a logical index, identified by name, columns, type, and aggregate row/fragment coverage;
one or more physical index segments, each with its own UUID, dataset version, and fragment set.

A distributed or incrementally maintained logical index may therefore have multiple UUIDs and dataset versions. The logical count must not be derived from the number of physical manifest entries.

The proposal is to keep Doris's existing MySQL-compatible 13-column SHOW INDEX schema instead of returning a different column layout only for Lance tables:

Key_name: logical index name;
Column_name: indexed column;
Index_type: logical Lance index type;
Properties: aggregate coverage, index-specific details, and a JSON list of physical segments (uuid, dataset_version, and fragment coverage).

For example:

{
  "indexed_fragments": 8,
  "unindexed_fragments": 2,
  "indexed_rows": 100000,
  "unindexed_rows": 24000,
  "segments": [
    {
      "uuid": "...",
      "dataset_version": 12,
      "fragments": [0, 1, 2, 3]
    }
  ]
}

The index count is the number of distinct user-visible logical indexes, excluding Lance system indexes. It is obtained from logical index descriptions, not from getIndexes() / lance-c physical entry counts. No additional count statement is proposed.

If reviewers prefer UUIDs and dataset versions as first-class result columns, a dedicated Lance system table or statement is safer than changing the established SHOW INDEX result schema.

4. Execution model

The Phase-1 flow follows Doris external-catalog DDL precedent:

Doris SQL / privilege validation
    -> Lance-specific index operator
    -> Lance Java SDK (JNI)
    -> Lance manifest commit
    -> Doris external-table refresh/edit-log path

The pinned Lance Java SDK provides create, drop, logical description, physical metadata, and distributed segment/commit primitives. Therefore Phase 1 does not require a new FE-to-BE thrift/RPC protocol.

There is an important difference from lightweight Iceberg/Paimon metadata DDL: Lance index creation scans column data and performs native CPU, memory, and object-store I/O work. In Phase 1 this work runs in Lance native threads inside FE. The synchronous implementation should therefore be documented with the following limitations:

build latency scales with the dataset and index parameters;
the pinned Java API does not expose a reliable cancellation token or Java progress callback for createIndex;
client disconnect or KILL QUERY does not guarantee that the native build has stopped;
FE failover can make the client-visible completion result ambiguous.

The operator boundary keeps a future implementation free to move segment building to BE workers or an external service while FE retains orchestration and the final commit. Lance exposes the required distributed model, but Lance itself does not provide the scheduler; the current lance-c v0.1.2 surface should not be assumed to contain the same distributed APIs as the Java SDK.

5. Authorization

CREATE INDEX, CREATE OR REPLACE INDEX, and DROP INDEX require the table-level ALTER privilege.
SHOW INDEX requires the table-level SHOW privilege.

This matches Doris internal index DDL and existing external-table ALTER behavior. The remote filesystem/object-store access still uses the catalog credential; Doris table privilege is the user-facing authorization boundary.

6. Lifecycle semantics

Versioning and visibility

A successful create/replace/drop is committed by Lance as a new dataset version with an atomic manifest switch.
Queries already pinned to an older dataset version continue to see that version.
A failed build before manifest commit does not make the new index visible. It may leave unreferenced physical files under the Lance index directory; later Lance cleanup/VACUUM reclaims them according to retention policy.
A catalog/table pinned to a historical dataset version is read-only for index mutation. SHOW INDEX reads the pinned snapshot; mutation statements fail instead of silently switching to latest.
For an ordinary writable table, mutation and live inspection target the latest dataset version.

Concurrency

Concurrent append is compatible with an index build; appended fragments may remain uncovered and are reported as unindexed.
Overwrite, restore, data replacement, or fragment rewrite may conflict according to Lance transaction rules.
Although current Lance format documentation describes compatible same-name commits, the pinned v9.1.0-beta.3 implementation tests same-name concurrent create/replace as a retryable commit conflict. Phase 1 should surface that failure and ask the user to retry instead of promising last-committer-wins.
The pinned JNI mapping does not expose a distinct Java exception class for commit conflict, so Phase 1 should not implement automatic retry by parsing exception text.
Doris metadata locks should not be held for the duration of native index training; Lance optimistic concurrency controls the external commit.

Failure recovery and retries

FE crash or client timeout before the response is ambiguous: the Lance commit may already have succeeded. SHOW INDEX is the reconciliation mechanism.
IF NOT EXISTS provides no-op retry behavior once the logical index is visible.
CREATE OR REPLACE INDEX is convergent by name but is not strictly idempotent: retrying may rebuild work and produce a new physical UUID/dataset version.
DROP INDEX on a missing name fails unless IF EXISTS is specified.

Drop and metadata refresh

Drop removes the logical index from a new manifest version. Physical files are reclaimed later according to Lance cleanup retention, preserving access through retained historical versions.
After a successful mutation, Doris follows the existing external DDL bookkeeping path: emit the refresh edit log and invalidate/refresh the table metadata on FE nodes.
Failed mutations do not run the post-DDL refresh path.
SHOW INDEX reads live index metadata from the target Lance snapshot rather than persisting index information in the Doris catalog cache.

7. Initial supported types and catalogs

Proposed initial set:

Vector: IVF_PQ. IVF_FLAT is included only if the PR verifies that the [feature](lance) support lance catalog in doris #65730 Doris query path actually consumes it end-to-end.
Scalar: BTREE, BITMAP, and INVERTED, each gated by schema/property validation and an end-to-end query test.
Other Lance vector/scalar types are rejected until their create and query paths are verified.

Catalog scope:

Filesystem-backed Lance datasets: local/file and supported object stores such as S3.
REST Namespace create/replace/drop: rejected with a clear unsupported error in Phase 1. REST index mutation is a separate asynchronous transaction/capability model and does not have the same request surface as direct Dataset.createIndex.
Read-only REST SHOW INDEX can be supported through namespace list/stats capabilities if reviewers want it in Phase 1; otherwise all REST lifecycle statements are rejected consistently and documented.

8. Implementation and test orientation

Implementation details will be reviewed in the PR. The intended direction is:

reuse AlterTableCommand + create/drop index ops and the existing ShowIndexCommand;
add explicit Lance external-table gates and catalog-specific dispatch;
place SDK calls and Lance-specific validation/mapping behind a LanceIndexOperator-style seam;
keep Lance logical/physical index DTOs separate from Doris internal index serialization;
run post-DDL refresh only after a successful Lance commit.

Tests should cover parser/to-SQL behavior, privileges, supported/unsupported types and properties, duplicate/replace/drop behavior, logical versus physical metadata, append coverage, concurrency conflicts, historical-version mutation rejection, REST behavior, and vector/scalar query use. Regression tests should use an isolated writable Lance dataset rather than mutating a shared read-only fixture.

Questions for reviewers

Q1: Is a filesystem-only, FE-side synchronous Java SDK implementation acceptable for Phase 1 with the resource, cancellation, and failover limitations above, while keeping distributed BE execution as a follow-up?
Q2: Is the explicit CREATE OR REPLACE INDEX syntax acceptable? The alternative PROPERTIES("replace"="true") avoids grammar work but hides lifecycle semantics.
Q3: Should Lance reuse the existing 13-column SHOW INDEX schema with physical segment metadata in Properties, or should this issue introduce a dedicated Lance index result surface?
Q4: Is IVF_PQ + BTREE/BITMAP/INVERTED the right initial end-to-end set, with IVF_FLAT added only after query-side verification?
Q5: For REST Namespace in Phase 1, should read-only SHOW INDEX be supported through list/stats while mutation remains unsupported, or should all index lifecycle statements be rejected consistently?

References

Doris external-catalog mutation precedent: Iceberg/Paimon catalog-specific metadata operations and the external DDL refresh/edit-log path
Doris internal index DDL chain: AlterTableCommand, create/drop index ops, and ShowIndexCommand
[feature](lance) support lance catalog in doris #65730 and tracking issue [Tracking] Lance feature matrix for Doris 4.2 #66340
Lance transaction semantics
Lance index logical/physical model
Lance distributed indexing
Lance Spark SHOW INDEXES logical result model
Lance Namespace index operations
Pinned Java SDK sources: Dataset, IndexOptions, and IndexDescription

Thanks for the detailed proposal. The source-of-truth model, logical/physical index distinction, version semantics, and post-commit metadata refresh direction are sound. After comparing the proposal with the current Doris command path, the pinned Lance SDK/Namespace API, and the Lance Spark, StarRocks, ClickHouse, DuckDB, and Trino designs, I think the following points should be resolved before implementation.

Blocking concerns

1. A synchronous native build in FE is not a safe production execution model

The proposal already lists the essential problems: index creation performs data scans and native CPU/memory/object-store I/O in FE, while the pinned API has no reliable cancellation/progress mechanism and FE failover can leave the result ambiguous. This also bypasses Doris workload management and allows a user with ALTER to trigger unbounded native work in the control plane.

The closest Lance integration distributes fragment builds to Spark executors and leaves the driver to coordinate/finalize the commit. ClickHouse similarly separates index metadata operations from the heavy MATERIALIZE INDEX mutation.

My recommendation is an asynchronous job with FE orchestration and BE or external-service execution, including status, cancellation, concurrency limits, and retry/reconciliation. If FE-side execution must exist temporarily, it should be explicitly experimental, disabled by default, and protected by dataset-size/fragment/concurrency limits. It should not be presented as the final production lifecycle implementation.

2. Reusing the current Doris index command chain requires a larger refactor than the proposal describes

The current parser only accepts NGRAM_BF, INVERTED, and ANN; it does not accept IVF_PQ, BTREE, BITMAP, or OR REPLACE:

doris/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4

Lines 237 to 240 in e3289c1

    
           | CREATE INDEX (IF NOT EXISTS)? name=identifier 
        
               ON tableName=multipartIdentifier identifierList 
        
               (USING (NGRAM_BF | INVERTED | ANN))? 
        
               properties=propertyClause? (COMMENT STRING_LITERAL)?                    #createIndex

More importantly, CreateIndexOp.validate() always translates the definition into the persisted internal catalog.Index, while AlterTableCommand currently rejects CreateIndexOp for external tables:

doris/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateIndexOp.java

Lines 73 to 89 in e3289c1

    
           @Override 
        
           public void validate(ConnectContext ctx) throws UserException { 
        
               if (indexDef == null) { 
        
                   throw new AnalysisException("index definition expected."); 
        
               } 
        
               if (tableName != null) { 
        
                   tableName.analyze(ctx); 
        
               } 
        
               indexDef.validate(); 
        
               index = indexDef.translateToCatalogStyle(); 
        
           } 
        
           @Override 
        
           public AlterTableClause translateToLegacyAlterClause() { 
        
               return new CreateIndexClause(tableName != null ? tableName.transferToTableName() : null, 
        
                       indexDef.translateToLegacyIndexDef(), index, alter);

That conflicts with the stated goal of keeping Lance index DTOs separate from Doris internal index serialization. The design should explicitly introduce a neutral parsed index definition, resolve the target table/catalog first, and then choose internal-index validation/serialization or Lance-specific validation/dispatch. Adding only a LanceIndexOperator after the existing validation path will not be sufficient.

For vector syntax, I would prefer reusing the Doris category-level surface:

CREATE INDEX idx ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    ...
);

This also follows the StarRocks pattern of USING VECTOR plus an index_type property and avoids adding a grammar keyword for every Lance algorithm.

Other required revisions

3. Incremental maintenance is necessary for a usable external index lifecycle

External writers can append fragments immediately after an index is created. Coverage will then decay. CREATE OR REPLACE INDEX performs an expensive full rebuild and is not an adequate maintenance operation.

The pinned Java SDK exposes Dataset.optimizeIndices(), and Lance documents it as the incremental path for indexing newly appended fragments. I suggest adding a separate operation such as:

BUILD INDEX idx ON lance_ctl.db.tbl;
-- or
ALTER TABLE lance_ctl.db.tbl OPTIMIZE INDEX idx;

The tests must also prove that queries include unindexed fragments correctly. Otherwise partial coverage becomes a correctness issue, not merely a performance issue.

4. Do not put unbounded physical segment metadata in SHOW INDEX.Properties

A JSON segments array can become extremely large for a fragmented or distributed index. It can create FE memory, MySQL result packet, and client usability problems.

Lance Spark intentionally returns only logical summaries from SHOW INDEXES and excludes per-segment metadata:

https://lance.org/integrations/spark/operations/ddl/show-indexes/

I recommend keeping the existing 13-column SHOW INDEX surface for logical name, columns, type, and aggregate coverage only. UUIDs, dataset versions, and fragment sets should be exposed through a structured and filterable system table/TVF such as lance_index_segments, not embedded in one string field.

5. REST behavior should be capability-based, not rejected by catalog type

The pinned lance-namespace-core 0.7.7 interface already defines createTableIndex, createTableScalarIndex, listTableIndices, describeTableIndexStats, and dropTableIndex:

https://github.com/lance-format/lance-namespace/blob/v0.7.7/java/lance-namespace-core/src/main/java/org/lance/namespace/LanceNamespace.java#L394-L447

For a Directory Catalog, direct Dataset operations are appropriate. For a REST Catalog, index operations should go through the Namespace service when the server supports them; Doris should return a clear unsupported error when it does not. Directly mutating the dataset with vended credentials would bypass service-side authorization, managed versioning, and transaction policy. At minimum, REST SHOW INDEX should use list/stats operations in Phase 1.

6. Narrow and rename the initial type set

I suggest Phase 1 support:

vector: IVF_PQ;
scalar: BTREE and BITMAP;
IVF_FLAT only after the Doris query path is verified end to end;
no INVERTED until the Lance FTS/BM25 query path is implemented.

Lance INVERTED is an FTS index with different tokenizer/query semantics from a Doris internal inverted index. Exposing the same keyword before Doris can query it is misleading. A future SQL surface should call it FTS.

The complete type/property matrix should be part of the design, not deferred entirely to the PR. It should include column types, vector element type/dimension/nullability, metric compatibility, parameter ranges, and build/query ABI compatibility.

7. Define SQL identifier semantics for index names

Allowing both idx and IDX while matching names exactly makes DROP INDEX idx ambiguous under Doris/MySQL identifier behavior. We should preserve display case but define quoted/unquoted resolution and reject new case-only duplicates, or provide an unambiguous quoted-name rule for pre-existing external collisions.

8. Refresh/recovery must handle post-commit failures

“Failed mutations do not refresh metadata” is insufficient when the Lance commit succeeds but FE fails before the edit log or client response. A later IF NOT EXISTS retry may be a no-op and still leave Doris caches stale.

The implementation should distinguish pre-commit failure from unknown/post-commit outcome, reopen the latest snapshot to verify the postcondition, and invalidate/refresh Doris metadata even for a reconciled no-op. SHOW INDEX helps the user observe state but does not repair the query path cached metadata.

Also, local/file catalogs need an explicit shared-filesystem or single-node restriction; otherwise a master failover can move execution to an FE that cannot see the same path.

Answers to the review questions

Q1: No for the production implementation. FE-side synchronous execution is acceptable only as a tightly gated experimental fallback.
Q2: CREATE OR REPLACE INDEX is reasonable if replacement is atomic, but incremental BUILD/OPTIMIZE INDEX is more important. For vector indexes, prefer USING ANN plus index_type.
Q3: Keep SHOW INDEX as a logical summary; expose physical segments through a system table/TVF.
Q4: Start with IVF_PQ + BTREE + BITMAP; add IVF_FLAT only after end-to-end verification, and exclude INVERTED until FTS querying is available.
Q5: Support REST inspection through Namespace list/stats. Support REST mutation only through Namespace index operations and server capability detection, never by bypassing the metadata service.

The proposal demonstrates a good understanding of the Lance index model, but the execution placement and the split between Doris internal index objects and external Lance operations need to be redesigned before implementation starts.

@Gabriel39

Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle

This revision supersedes the previous proposal and incorporates the review feedback from @Gabriel39.

Major changes from v1

Heavy index creation, replacement, and incremental maintenance no longer run synchronously in FE. Production execution is a durable asynchronous job: FE orchestrates and finalizes commits, Directory catalogs use distributed BE segment construction, and REST catalogs use Namespace-managed execution.
The SQL path introduces a neutral parsed index specification. Doris resolves the target table and catalog before choosing internal-index serialization or Lance-specific validation and dispatch.
Vector syntax reuses the Doris category-level surface: USING ANN with index_type=IVF_PQ.
BUILD INDEX is added to the Phase-1 lifecycle as the incremental operation for newly appended or otherwise uncovered fragments.
SHOW INDEX returns only bounded logical summaries. A filterable physical-segment TVF exposes UUIDs, dataset versions, and aggregate segment coverage; explicit segment-to-fragment membership is deferred.
REST behavior is capability-based. REST mutations go through Lance Namespace APIs and never bypass the metadata service with vended credentials.
The initial type set is narrowed to IVF_PQ, BTREE, and BITMAP. Lance INVERTED/FTS and IVF_FLAT are deferred until the Doris query path is verified end to end.
Index-name resolution, post-commit reconciliation, metadata refresh retries, and local-filesystem restrictions are defined explicitly.

1. Goals and Phase-1 boundary

Phase 1 provides a production-oriented lifecycle for user-visible Lance indexes:

create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes;
inspect logical index definitions, aggregate coverage, and logical index count;
inspect physical index segments through a separate metadata surface;
incrementally index newly appended or uncovered fragments;
atomically replace/rebuild an index by logical name;
drop an index by logical name;
provide durable status, cancellation, concurrency limits, retry/reconciliation, and metadata refresh;
support Directory Catalog operations and capability-based REST Namespace operations.

Lance dataset manifests or the Namespace service remain the authoritative index metadata. Doris does not persist a second copy of Lance index definitions in its internal table/index metadata. Doris persists only its own asynchronous job state and the information needed to reconcile an external operation.

Historical-version tables remain read-only for index mutation. Inspection uses the selected snapshot; mutation targets only the latest writable table state.

An FE-side synchronous native build is not part of the production implementation. If retained temporarily for development, it must be experimental, disabled by default, and protected by strict dataset-size, fragment-count, and concurrency limits. It does not satisfy Phase-1 completion.

2. SQL surface

Vector creation

CREATE INDEX [IF NOT EXISTS] idx
ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    "metric" = "l2",
    "num_partitions" = "256",
    "num_sub_vectors" = "16"
);

Scalar creation

CREATE INDEX idx
ON lance_ctl.db.tbl (scalar_col)
USING BTREE
PROPERTIES (
    "zone_size" = "4096"
);

CREATE INDEX idx_bitmap
ON lance_ctl.db.tbl (category_col)
USING BITMAP;

Full replacement and incremental maintenance

CREATE OR REPLACE INDEX idx
ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    "metric" = "l2",
    "num_partitions" = "256",
    "num_sub_vectors" = "16"
);

BUILD INDEX idx ON lance_ctl.db.tbl;

CREATE OR REPLACE INDEX performs a full rebuild and atomically replaces the logical index. BUILD INDEX indexes only fragments not covered by the current logical index and maps to the Lance incremental optimization path.

Inspection, status, cancellation, and drop

SHOW INDEX FROM lance_ctl.db.tbl;

SHOW BUILD INDEX FROM db
WHERE TableName = "tbl";

CANCEL BUILD INDEX ON lance_ctl.db.tbl (job_id);

DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;

The existing Doris SHOW BUILD INDEX and CANCEL BUILD INDEX surfaces are extended to Lance external-index jobs instead of introducing a second job-control vocabulary.

Statement semantics

USING is required for Lance tables.
IF NOT EXISTS and OR REPLACE are mutually exclusive.
Plain CREATE INDEX fails if the logical name exists.
IF NOT EXISTS verifies the visible logical definition and returns a reconciled no-op when it already matches. A conflicting definition with the same name is an error rather than a silent no-op.
CREATE OR REPLACE INDEX performs a full rebuild. It is convergent by name but not strictly idempotent: a retry can produce new segment UUIDs and a new dataset version.
BUILD INDEX is incremental and does not change the logical definition.
Phase 1 supports one indexed column per statement. Composite scalar indexes are deferred.
Unknown index types, unknown properties, unsupported column types, and invalid property values fail during analysis.
Create, replace, and build return after a durable job is accepted, not after native index construction completes.
Drop is a metadata operation. It may complete synchronously when the selected provider exposes an atomic drop, but it still uses the same post-commit reconciliation and refresh rules.

3. Neutral command model and catalog dispatch

The current internal path must not translate every parsed definition into persisted catalog.Index before the target catalog is known.

The revised analysis flow is:

parsed SQL
    -> ParsedIndexSpec
    -> resolve catalog / database / table
    -> select InternalIndexProvider or LanceIndexProvider
    -> provider-specific validation
    -> internal schema-change operation or Lance external-index job

ParsedIndexSpec contains only SQL-level information:

logical name;
target columns;
category from USING;
properties;
IF NOT EXISTS / OR REPLACE;
operation kind: create, replace, incremental build, or drop.

For an internal OLAP table, the existing provider validates and translates the specification into Doris catalog.Index and the legacy schema-change path.

For a Lance external table, LanceIndexProvider produces a Lance-specific immutable definition and operation request. Lance types and physical metadata are never added to Doris internal IndexType serialization.

The provider boundary is also used by SHOW, BUILD, DROP, reconciliation, and capability checks so lifecycle behavior does not drift across separate command implementations.

4. Asynchronous execution model

Durable job

Create, replace, and incremental build create a persistent LanceIndexJob before heavy work starts. The minimum states are:

PENDING
  -> PREPARING
  -> BUILDING
  -> COMMITTING
  -> REFRESHING
  -> FINISHED

Terminal or recovery states:
FAILED
CANCELLING
CANCELLED
COMMITTED_REFRESH_PENDING
OUTCOME_UNKNOWN

The persisted job records:

catalog, table identity, dataset URI or Namespace table ID;
logical index name and normalized definition;
operation kind and starting dataset version;
old segment UUIDs for replace/reconciliation;
provider and executor identity;
worker task IDs or remote transaction ID;
current state, progress, timestamps, and sanitized failure information.

FE failover reconstructs unfinished jobs from the edit log and resumes polling, reconciliation, commit, or refresh. It does not blindly restart native training.

Responsibility split

FE performs:

SQL analysis and table-level privilege checks;
durable job creation and state transitions;
executor selection and concurrency admission;
optimistic commit/finalization when the selected execution model separates build from commit;
outcome reconciliation;
metadata invalidation, refresh edit log, and retry of post-commit refresh.

For Directory Catalog, BE workers perform:

fragment-scoped column scans;
vector training;
scalar/vector segment construction;
native CPU, memory, and object-store I/O work;
periodic progress reporting and cooperative cancellation.

For REST Namespace Catalog, the Namespace service owns execution and commit according to its advertised capabilities. Doris tracks the returned operation or transaction identifier and reconciles the result through the service; it does not substitute a direct Dataset mutation.

FE metadata locks are not held during native training. Doris serializes conflicting jobs for the same dataset and logical index name, while Lance transaction rules remain authoritative for conflicts with external writers.

Resource and cancellation behavior

Jobs are admitted through explicit per-catalog and cluster-wide concurrency limits.
BE execution is associated with workload-management accounting where the executor supports it.
Cancellation is persisted before workers are signaled and immediately fences the job from entering COMMITTING.
Every BE task carries the durable job ID and generation. FE accepts physical segment metadata only from the active generation.
The lance-c extension used by the production path must expose cooperative cancellation and progress. The production Directory executor remains disabled until those APIs are available.
A native task that finishes after cancellation may leave unreferenced physical files, but it cannot publish them because final commit is fenced in FE. The job becomes CANCELLED only after active tasks stop or their results are safely discarded.
Native exception strings are not parsed to implement automatic conflict retry. Typed conflict results are used where the provider exposes them.
Same-name concurrent create/replace conflicts are surfaced as retryable failures unless a later reconciliation proves that this job's requested postcondition was committed.

5. Provider and catalog behavior

Directory Catalog

For Directory Catalog, this proposal chooses Doris BE execution rather than introducing an external Java/Rust build service. Because BE is C++, this choice requires lance-c to expose the distributed Rust Lance APIs that already exist in core.
Directory-backed datasets use a distributed BE build pipeline. Phase 1 extends lance-c with the native capabilities needed by that pipeline:

fragment-scoped uncommitted vector and scalar segment construction;
serialization of physical segment metadata back to FE;
progress and cooperative cancellation;
merge support where the selected index type requires compatible segments.

FE selects disjoint fragment groups, dispatches BE tasks, validates the returned coverage, and performs the single atomic logical-index commit through the pinned Lance Java SDK. BE workers never publish the logical index independently.

CREATE and CREATE OR REPLACE build all fragments visible in the starting snapshot. BUILD INDEX computes the uncovered fragments from the latest logical description and sends only those fragments through the same segment pipeline. Concurrent appends after planning remain unindexed and can be covered by a later BUILD INDEX.

For object-store-backed datasets, workers use the catalog credentials and the dataset URI resolved from the latest table metadata.

A local/file dataset is mutable only when the deployment guarantees that the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation. Otherwise mutation is rejected. Local filesystem support is intended primarily for single-node development and tests; it must not silently depend on the current master FE's local disk.

REST Namespace Catalog

REST behavior is capability-based:

SHOW INDEX uses Namespace list/stats operations.
Create, replace, and drop use the Namespace index operations when supported by the server.
Managed-versioning and service-side transaction policy are preserved.
Doris never performs a direct Dataset mutation with vended credentials as a fallback for an unsupported Namespace mutation.
An unsupported Namespace operation returns a clear capability error.
If the Namespace service returns an asynchronous transaction or operation ID, the Doris job tracks and polls it.
Physical-segment inspection is exposed only when the Namespace capability supplies the required metadata; Doris does not bypass the service to obtain it.

6. Initial type and property matrix

Only types verified by both creation and the Doris query path are enabled.

IVF_PQ

SQL category: ANN
Required property: index_type=IVF_PQ
Column count: exactly one
Column shape: fixed-size list
Initial element types: FLOAT16, FLOAT32, UINT8, and INT8; FLOAT64 is deferred because the pinned lance-c creation surface does not advertise it
Nullability: the vector field and its elements must be non-null in Phase 1
Metrics: L2, COSINE, and DOT
num_partitions: optional positive integer; SDK default is used when absent
num_sub_vectors: optional positive integer; must divide the vector dimension
num_bits: fixed to 8 in Phase 1
Advanced centroids, codebooks, sample rate, training iterations, and HNSW/RQ/SQ properties are not user-visible in Phase 1
Build/query compatibility is verified against the exact Lance Java SDK and lance-c versions pinned by Doris

BTREE

Column count: exactly one
Initial column types: integral numeric, floating numeric, DECIMAL, STRING, DATE, DATETIME, and TIMESTAMPTZ types already supported by the Lance predicate pushdown path
Nullable columns are supported only with end-to-end coverage for equality, range, IS NULL, and IS NOT NULL over indexed and unindexed fragments
Optional zone_size must be a positive integer
Distributed-only/internal fields such as range identifiers are not exposed as user properties

BITMAP

Column count: exactly one
Initial column types: BOOLEAN, integral numeric, STRING, and DATE
Nullable columns are supported with explicit NULL predicate tests
No user-visible properties in Phase 1
Distributed shard identifiers remain executor-internal

Deferred types

IVF_FLAT remains disabled until an index built by the lifecycle path is consumed by the Doris vector query path in an end-to-end test.
Lance INVERTED is not exposed as Doris INVERTED. It is an FTS/BM25 index with different query and tokenizer semantics. A later design should expose it as FTS together with Doris FTS query support.
Other Lance scalar and vector types are rejected rather than passed through.

7. Logical and physical metadata surfaces

Lance has a logical index identified by name, columns, type, definition, and aggregate coverage. It may contain multiple physical segments with independent UUIDs, dataset versions, and fragment sets.

SHOW INDEX

The existing 13-column Doris schema is retained. It returns one row per logical index/column:

Key_name: logical name;
Column_name: indexed column;
Index_type: ANN, BTREE, or BITMAP;
Properties: bounded logical details only, such as Lance physical type, metric, indexed/unindexed row and fragment counts, and supported index-specific parameters.

The Properties field never contains a segment array or fragment-ID list.

Logical index count is the number of user-visible logical descriptions, excluding system indexes. It is not derived from physical manifest entries. No separate count statement is added.

Physical metadata

A filterable lance_index_segments TVF exposes one row per physical segment:

catalog, database, table;
logical index name;
segment UUID;
physical index type and version;
dataset version;
indexed row and fragment counts;
creation timestamp when available.

Phase 1 does not expose the complete segment-to-fragment mapping. Aggregate logical coverage is available from SHOW INDEX, and per-segment fragment counts are available from the TVF. Explicit fragment-ID membership is deferred until there is a demonstrated diagnostic use case, avoiding another potentially unbounded result surface.

The TVF reads live metadata from the selected Lance snapshot. It does not persist a copy in Doris catalog metadata.

8. Authorization and identifier semantics

CREATE, CREATE OR REPLACE, BUILD, DROP, and CANCEL require table-level ALTER.
SHOW INDEX, SHOW BUILD INDEX, and physical metadata inspection require table-level SHOW.
Remote storage still uses catalog credentials; Doris table privilege remains the user-facing authorization boundary.

Doris preserves the display case returned by Lance but resolves user-supplied index names case-insensitively, matching existing Doris/MySQL identifier behavior.

New indexes whose names differ only by case are rejected.
Create, build, replace, and drop first resolve a unique case-insensitive logical match, then pass the exact physical name to Lance.
If a pre-existing external dataset contains case-only collisions, SHOW displays all of them, while mutation by that ambiguous name fails with an explicit ambiguity error.
Phase 1 does not claim quoted exact-name resolution because the current parsed identifier path does not reliably preserve a distinct quoted-name matching mode.

9. Versioning, concurrency, and visibility

A successful create, replace, build, or drop commits a new table version atomically.
Queries already pinned to an older version continue to use that version.
Mutation of a historical-version table is rejected.
Ordinary mutation and live inspection target the latest version.
Concurrent append is compatible with a build; newly appended fragments may remain uncovered and are reported as unindexed.
BUILD INDEX later adds coverage for uncovered fragments.
Overwrite, restore, fragment rewrite, or data replacement follows Lance transaction-conflict rules.
Doris does not hold table metadata locks for the lifetime of a build.
Doris prevents two active jobs for the same dataset and case-insensitive logical name, but it does not assume it controls external writers.
Queries over a partially indexed table must combine indexed results with scans of unindexed fragments. End-to-end tests must prove result equivalence with index use disabled.

10. Failure recovery and metadata refresh

The implementation distinguishes three outcomes.

Confirmed pre-commit failure

The external manifest or Namespace transaction was not committed. The job becomes FAILED. Doris does not run post-DDL refresh. Unreferenced artifacts are reclaimed later by Lance cleanup according to retention policy.

Confirmed commit

The requested postcondition is visible in the latest snapshot:

create: a matching logical definition is present;
replace: the definition matches and the old segment identity has been replaced;
incremental build: coverage has advanced or is already complete;
drop: the logical name is absent.

Doris invalidates and refreshes table metadata and emits the normal external-DDL refresh edit log. A refresh failure does not turn the job into an ordinary build failure; the job becomes COMMITTED_REFRESH_PENDING and retries refresh.

Unknown or post-commit failure

After FE failover, timeout, lost worker response, or an ambiguous commit result, Doris reopens the latest snapshot or queries the Namespace transaction and verifies the operation-specific postcondition.

If the postcondition holds, Doris records the commit, runs refresh, and finishes the job.
If it does not hold and the provider proves no commit occurred, the job fails or is safely retried according to the operation policy.
If the outcome remains unknowable, the job remains OUTCOME_UNKNOWN and exposes a diagnostic rather than silently rebuilding.

A reconciled IF NOT EXISTS or IF EXISTS no-op still invalidates/refreshes Doris metadata when it observes external state that may be newer than the local cache.

SHOW INDEX is the user-visible reconciliation tool, but it is not a substitute for repairing Doris metadata caches.

11. Test plan

Tests cover:

parser and to-SQL behavior for ANN/index_type, BTREE, BITMAP, OR REPLACE, BUILD, SHOW, CANCEL, and DROP;
internal-table behavior remaining unchanged after neutral command refactoring;
ALTER/SHOW privileges;
the complete type/property matrix and rejection of unknown values;
duplicate, IF NOT EXISTS, case-only duplicate, ambiguous pre-existing name, replace, incremental build, and drop semantics;
durable job state transitions, FE restart/failover recovery, concurrency limits, cancellation, and retryable conflicts;
successful commit followed by edit-log/refresh failure and subsequent reconciliation;
logical versus physical metadata and multi-segment logical counts;
append after create, coverage decay, incremental BUILD, and query correctness across indexed plus unindexed fragments;
historical-version mutation rejection;
local/file visibility restrictions;
REST list/stats, supported mutation, unsupported capability, managed-versioning, and asynchronous transaction polling;
end-to-end IVF_PQ, BTREE, and BITMAP query use through the Doris BE reader.

Regression tests use isolated writable datasets and generate expected result files through the normal regression-test scripts.

12. Confirmed design decisions and implementation decomposition

The review questions from v1 are resolved as follows:

Production execution: FE-side native construction is rejected. Directory catalogs use durable asynchronous jobs with distributed BE segment construction; REST catalogs use Namespace-managed operations.
Native prerequisite: Phase 1 extends lance-c with fragment-scoped uncommitted build, physical metadata transfer, progress, and cooperative cancellation. BUILD INDEX uses this distributed uncovered-fragment path rather than running Dataset.optimizeIndices inside FE.
Commit ownership: BE workers produce uncommitted physical segments. FE validates disjoint coverage and performs the single atomic logical-index commit. A persisted cancellation or stale job generation fences commit.
SQL and job control: Vector creation uses USING ANN with index_type=IVF_PQ. CREATE OR REPLACE performs a full rebuild, BUILD INDEX performs incremental coverage, and the existing SHOW BUILD INDEX / CANCEL BUILD INDEX surfaces are extended for Lance jobs.
Metadata: SHOW INDEX remains a bounded logical summary. Phase 1 adds one lance_index_segments TVF for physical UUIDs, versions, and aggregate segment coverage. Full fragment membership is deferred.
REST: Inspection uses Namespace list/stats. Mutation uses Namespace index operations only when supported and never bypasses the service with vended credentials.
Initial types: Phase 1 enables IVF_PQ, BTREE, and BITMAP only. IVF_FLAT requires a separate end-to-end query verification, and Lance INVERTED remains deferred until Doris exposes FTS semantics.
Recovery: A confirmed or reconciled external commit always enters the refresh path. Refresh failures remain durable as COMMITTED_REFRESH_PENDING and are retried.

Implementation is split into the following dependency-ordered work items:

lance-c APIs: add fragment-scoped uncommitted builds, segment metadata transfer, progress, cancellation, and the required merge support.
FE command model: introduce ParsedIndexSpec and catalog-specific providers without changing internal OLAP index serialization.
Durable job framework integration: persist Lance job state, generation fencing, admission limits, failover recovery, SHOW, and CANCEL behavior.
BE execution: schedule fragment groups, build segments under workload management, report progress, and discard cancelled/stale results.
Commit and reconciliation: validate segment coverage, commit once, reconcile unknown outcomes, and retry metadata refresh.
Catalog and metadata surfaces: implement Directory and REST providers, logical SHOW INDEX, and the lance_index_segments TVF.
Tests and documentation: complete the matrix in Section 11 and document lifecycle guarantees and limitations.

These items may be delivered as separate dependent PRs, but the production Directory create/build path remains disabled until the native build, cancellation, durable job, commit-fencing, reconciliation, and end-to-end query requirements are all present.

Remaining implementation-planning question

The technical direction above is fixed, but the delivery boundary still needs reviewer agreement:

Is direct BE execution through an extended lance-c the preferred integration model, or should Doris use an external Java/Rust index-build service?

References:

Doris existing CREATE/BUILD/SHOW/CANCEL index command paths
[feature](lance) support lance catalog in doris #65730 and [Tracking] Lance feature matrix for Doris 4.2 #66340
Lance distributed indexing: https://lance.org/guide/distributed_indexing/
Lance Spark CREATE INDEX: https://lance.org/integrations/spark/operations/ddl/create-index/
Lance Spark SHOW INDEXES: https://lance.org/integrations/spark/operations/ddl/show-indexes/
Lance Namespace operations: https://lance.org/format/namespace/operations/
Pinned Lance Java SDK v9.1.0-beta.3 and lance-c v0.1.2

Thanks for the detailed revision. V2 addresses most of the architectural concerns from the first review: synchronous FE construction is no longer the production path, the neutral command model is clearer, BUILD INDEX is separated from full replacement, logical and physical metadata are separated, and the recovery states are substantially better defined.

I think the Directory Catalog direction is now viable, but I do not think the design is implementation-ready yet. The following issues still need to be resolved.

1. The REST design currently promises more than the pinned Namespace API can express

Doris pins Lance Namespace 0.7.7. Its CreateTableIndexRequest supports the column, index type, name, distance type, and FTS-related fields, but it does not carry num_partitions, num_sub_vectors, zone_size, replace semantics, or an uncovered-fragment/incremental-build request:

https://github.com/lancedb/lance-catalog/blob/v0.7.7/java/lance-namespace-apache-client/docs/CreateTableIndexRequest.md

There is also no separate Namespace BUILD INDEX operation. In addition, ListTableIndices exposes only name, UUID, columns, and status, while index stats expose only type, distance, indexed/unindexed row counts, and physical index count. That is not sufficient to verify that an existing REST index has the complete normalized definition required by the proposed IF NOT EXISTS reconciliation.

Please either:

make the required Namespace specification/API changes an explicit prerequisite; or
define a Phase-1 REST capability matrix and reject unsupported replace, build, and property combinations instead of promising uniform behavior.

“Advertised capabilities” should also be made concrete. In the pinned Java interface, optional operations are primarily represented by typed UnsupportedOperationException; there is no general capability-advertisement API.

2. Reusing SHOW/CANCEL BUILD INDEX requires a real external-job integration design

The current CancelBuildIndexCommand explicitly rejects external catalogs and eventually looks up an internal OlapTable and SchemaChangeHandler.IndexChangeJob:

https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CancelBuildIndexCommand.java

Similarly, SHOW BUILD INDEX reads the internal /jobs/<dbId>/build_index proc data and has internal-index-specific columns such as PartitionName and AlterInvertedIndexes:

https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowBuildIndexCommand.java

Extending the SQL vocabulary is reasonable, but the design should specify the common job registry/proc source, catalog qualification, filtering, privilege checks, output schema, and state mapping for internal and Lance jobs. Otherwise the SQL examples in this proposal cannot work through the existing implementation.

3. REST reconciliation needs job-specific provenance, not only a matching postcondition

The remote transaction_id is optional. If the service commits the operation but FE fails before persisting the returned transaction ID, a later “same name and matching definition” observation does not prove that this Doris job performed the commit; an external writer may have produced the same postcondition.

This is particularly important because Lance allows concurrent CreateIndex transactions. For two operations on the same logical name, the later operation can replace the earlier one rather than returning the retryable conflict assumed by this proposal:

https://lance.org/format/table/transaction/

Please persist a client operation ID before dispatch and propagate it through Namespace transaction properties or another queryable server-side idempotency/provenance mechanism. For Directory builds, expected segment UUIDs can provide this proof. Without job-specific evidence, the REST job should remain OUTCOME_UNKNOWN instead of being marked successful merely because the logical definition matches.

4. DROP also needs a durable operation record

The proposal creates durable jobs for create, replace, and build, but allows DROP to complete synchronously while still promising failover-safe post-commit reconciliation and refresh.

If DROP commits externally and FE fails before refresh/edit-log completion, there is no durable state from which to resume. DROP should therefore use the same durable Lance job/operation record, even if it normally moves through the states very quickly.

5. Please make the distributed segment contract explicit

The pinned Java SDK already exposes mergeExistingIndexSegments and commitExistingIndexSegments, so BE construction followed by one FE commit is technically plausible:

https://github.com/lance-format/lance/blob/v9.1.0-beta.3/java/src/main/java/org/lance/Dataset.java

However, the design should require:

one pinned and verified Lance core/index-metadata compatibility contract across lance-c and the Java SDK;
validation of field IDs, physical index type and parameters, starting dataset version, exact assigned fragment coverage, UUID uniqueness, and artifact location before FE accepts a worker result;
for incremental BUILD, committing the complete final logical segment set (existing compatible segments plus new segments), not only the newly built segments;
an explicit IVF model strategy. Independent per-segment models are valid for fan-out query, but physical merge requires compatible/shared model artifacts such as centroids and PQ codebooks.

See: https://lance.org/guide/distributed_indexing/

Credentials should not be persisted in the job edit log. Workers should obtain or refresh catalog credentials when a task or retry is dispatched.

6. Metadata details

I recommend reporting IVF_PQ, rather than ANN, in SHOW INDEX.Index_type. ANN is the SQL category, while IVF_PQ is the actual index type; reporting only ANN becomes ambiguous when additional ANN implementations are added.

REST metadata should also be explicitly capability-dependent: the pinned stats model does not expose fragment counts or the complete creation properties, so those SHOW fields cannot always be populated uniformly.

Integration-model recommendation

For the remaining planning question, I prefer direct BE execution through an extended lance-c for Directory Catalog. An external Java/Rust build service would introduce a new deployment, authentication, versioning, availability, and operational dependency for a core Apache Doris feature. Keeping execution in BE is a better fit as long as the production path remains gated until the required lance-c build, cancellation, progress, segment-metadata, and compatibility APIs are available.

Section 5 already says that the proposal chooses BE execution, while the final section presents this as unresolved. Please make the conclusion consistent.

Once the REST capability boundary, existing job-command integration, job-specific reconciliation, and durable DROP semantics are clarified, I think the design will be in much better shape for approval and decomposition into implementation issues.

@Gabriel39

Thank you @Gabriel39 for the patient and precise guidance across both review rounds — each round has made this design substantially stronger. Below is the v3 full revision, addressing every round-2 point directly and incorporating several self-identified corrections.

Revised Design Proposal v3 — Asynchronous Lance Index Lifecycle

Draft for #66497, prepared 2026-08-07. Incorporates Gabriel39's round-2 review (2026-08-06 13:15 UTC) + self-identified errata. This revision has passed three independent adversarial reviews (research-consistency, review-coverage, source-level fact-check); every load-bearing claim carries pinned-source evidence, and the distributed build path is verified by an end-to-end PoC against lance-core 9.1.0-beta.3 (fragment-scoped uncommitted build → coordinator commit → k-NN consumption). Post as a full standalone revision.

This revision incorporates the second-round review from @Gabriel39: REST is now an explicit capability matrix, SHOW/CANCEL BUILD INDEX get a concrete external-job integration design, reconciliation requires job-specific provenance, DROP uses the same durable job record, and the distributed segment contract is specified. It also contains several self-identified corrections to v2.

/cc @Gabriel39 @zhangstar333

Major changes from v2

REST capability matrix replaces uniform-behavior language: a per-operation/per-property support table; unsupported combinations are rejected at analysis time where statically known, otherwise surfaced as a clear capability error.
Job-specific provenance: every mutation job persists a client-generated operation_id before dispatch. Reconciliation requires job-specific evidence — for Directory, the expected segment UUID set plus the operation_id recorded in the Lance transaction properties at commit time; for REST, the server-returned transaction_id (see the honest boundary in §10). A matching postcondition alone is never accepted as success — including for DROP.
Concurrency semantics stated in both layers: the Lance format documentation defines same-name concurrent CreateIndex as last-committer-wins; the pinned implementation enforces it via a retryable commit conflict + rebase on the first attempt. Either way, a name/definition match never proves attribution.
DROP is durable: it uses the same LanceIndexJob record (it usually transitions quickly), closing the post-commit failover gap — and its reconciliation follows the same provenance rules.
Distributed segment contract: a pinned lance-c/Java-SDK compatibility requirement, a worker-result validation checklist, full-final-segment-set commits for incremental BUILD, an explicit shared IVF model strategy, a defined BE→FE segment-metadata schema, and no credentials in the edit log.
SHOW/CANCEL BUILD INDEX integration design for external jobs: registry, proc source, catalog qualification, filtering, privileges, output schema, and state mapping (current implementations verified internal-only on both master and branch-4.1).
Metadata: SHOW INDEX.Index_type reports the physical type (IVF_PQ); REST SHOW fields are capability-dependent.
Type matrix narrowed: vector element types are FLOAT16/FLOAT32 only (see corrections).
Execution model locked: Directory = BE via extended lance-c; no external build service. The FE-side build remains an experimental, disabled-by-default development fallback using the Java SDK's existing uncommitted-build primitives.
Provenance-carrying commits use the general Transaction path: the one-shot Dataset.createIndex / commitExistingIndexSegments / dropIndex APIs do not carry transaction properties in the pinned SDK; commits that must record operation_id go through Transaction.Builder.transactionProperties + operation.CreateIndex + CommitBuilder (verified end-to-end through JNI in the pinned sources).

Corrections to v2 (self-identified)

Concurrent same-name CreateIndex — both layers, precisely. The Lance format transaction documentation states: two concurrent CreateIndex operations are allowed; with the same name, "the second operation will win and replace the first" (docs/src/format/table/transaction.md:172-173). The pinned implementation reaches that semantics via conflict + rebase: check_create_index_txn returns a retryable conflict for a same-name regular index on the first attempt (rust/lance/src/io/commit/conflict_resolver.rs:577-617), and the Java CommitBuilder.maxRetries defaults to 0, so without retries the caller observes the conflict error. v2 saw only the implementation layer; the design must assume the doc layer for external writers (retried/ rebased commits can replace Doris-created indexes) — hence provenance in §10.
UINT8 element type: the Rust core treats UInt8 vectors as binary vectors and allows only Hamming distance (rust/lance/src/index/vector/utils.rs:193,204-211). v2's matrix (UINT8 with L2/COSINE/DOT) was inconsistent. UINT8 is deferred until a binary-vector/HAMMING decision is made. INT8 is deferred: it is advertised by the pinned lance-c creation surface and accepted by the Rust core (L2/COSINE/DOT allowed), but it is an undocumented branch there and lacks end-to-end verification. FLOAT64 remains deferred — not advertised by the pinned lance-c creation surface (lance.h:926 restricts to float32/float16/uint8/int8), although the Rust core accepts it.
REST monitoring channel: index progress is tracked through the Namespace list/stats operations — the spec states "Index creation is handled asynchronously. Use the ListTableIndices and DescribeTableIndexStats operations to monitor index creation progress" (spec.yaml:1633-1634). The optional transaction_id is a generic long-running-operation handle, not the index progress channel.
FTS deferral rationale: the BE reader already has a full-text search path (be/src/format_v2/table/lance_reader.cpp:521 on branch-4.1). Lance INVERTED is deferred because Doris has no SQL-level FTS semantics for it yet — not because the query path is absent.
TVF per-segment coverage dropped from Phase 1 (declared downgrade): v2 promised per-segment row/fragment counts in the physical-segment TVF. No pinned API exposes them: the lance-c list JSON is hard-coded to {name, uuid, columns, type, dataset_version} per entry (lance-c src/index.rs), and the Namespace stats model exposes only row counts (spec.yaml:3206-3234). This becomes lance-c extension item ⑥; the TVF ships without those columns until then. Reviewer approval for this deferral is requested explicitly.
Index-name case behavior (research correction): an earlier internal research note claimed Lance lowercases index names. The pinned sources show Lance index names are case-sensitive, exactly matched, and stored as-is (rust/lance/src/index/create.rs:202, rust/lance/src/index/api.rs:211; no normalization in builder/Python/Java layers). The §8 design — Doris-layer case-insensitive resolution over Lance's exact-match storage — stands precisely because idx/IDX coexistence is a real external state.

1. Goals and Phase-1 boundary

Phase 1 provides a production-oriented lifecycle for user-visible Lance indexes:

create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes (Directory full; REST restricted to the request model per §5);
incrementally index uncovered fragments and atomically replace an index by name (Directory);
drop an index by name (Directory and REST);
inspect logical definitions, aggregate coverage, and the logical index count (Directory full; REST capability-dependent);
durable status, cancellation, concurrency limits, provenance-based reconciliation, and metadata refresh;
REST behavior follows the capability matrix in §5; everything outside it is rejected explicitly.

Lance dataset manifests or the Namespace service remain the authoritative index metadata. Doris persists only its own job state and reconciliation evidence. Historical-version tables remain read-only for index mutation; inspection uses the selected snapshot, mutation targets only the latest writable state.

An FE-side synchronous build is not part of the production implementation. A development fallback (experimental, disabled by default, limited by dataset-size/fragment/concurrency settings) MAY use the pinned Java SDK's existing uncommitted primitives (IndexOptions.withFragmentIds / withIndexUUID, javadoc-verified uncommitted + fragment-scoped). The production objection is compute placement in the control plane, not missing APIs.

2. SQL surface

Unchanged from v2 except the property-key question (Q2):

vector: CREATE INDEX ... USING ANN PROPERTIES ("index_type"="IVF_PQ", "metric_type"="l2", "num_partitions"="256", "num_sub_vectors"="16") (key naming is Q2 below; example uses the proposed key);
scalar: USING BTREE / USING BITMAP;
CREATE OR REPLACE INDEX = full rebuild + atomic name replacement;
BUILD INDEX = incremental coverage of uncovered fragments;
SHOW INDEX FROM t; SHOW BUILD INDEX FROM db WHERE ...; CANCEL BUILD INDEX ON ctl.db.tbl (job_id); DROP INDEX [IF EXISTS].

Statement semantics (delta from v2):

Create, replace, build, and drop each return after a durable job is accepted. Drop typically completes within one commit cycle but is equally durable.
REST IF NOT EXISTS reconciliation compares only the fields observable through the Namespace stats model (name, columns, index type, distance type); properties the REST path cannot express are rejected at analysis for REST tables, so no hidden definition skew is possible (§5).
All other semantics (OR REPLACE convergent-but-not-idempotent, single column, analysis-time rejection of unknown types/properties) unchanged.

3. Neutral command model and catalog dispatch

Unchanged from v2: ParsedIndexSpec → resolve catalog/database/table → InternalIndexProvider or LanceIndexProvider → provider validation → internal schema change or Lance external-index job. Lance definitions never enter Doris internal catalog.Index serialization.

4. Durable job framework and SHOW/CANCEL BUILD INDEX integration

Job record (persisted via edit log)

job id, operation kind (CREATE / REPLACE / BUILD / DROP), catalog/database/table identity, dataset URI or Namespace table id;
logical index name and normalized definition;
operation_id (client-generated UUID), persisted before any dispatch;
starting dataset version; expected/accepted segment UUID set; old segment UUIDs (replace);
executor identity, worker task ids, or Namespace transaction id (persisted when returned);
state, progress, timestamps, sanitized failure information;
no credentials — workers obtain or refresh catalog credentials when a task or retry is dispatched.

States

Unchanged from v2: PENDING → PREPARING → BUILDING → COMMITTING → REFRESHING → FINISHED, plus FAILED / CANCELLING / CANCELLED / COMMITTED_REFRESH_PENDING / OUTCOME_UNKNOWN. DROP traverses the same machine (typically PENDING → COMMITTING → REFRESHING → FINISHED in one cycle). FE failover reconstructs unfinished jobs from the edit log and resumes polling, reconciliation, commit, or refresh; it does not blindly restart native training.

Resource admission

Unchanged from v2, restated here because it gates the production path: jobs are admitted through explicit per-catalog and cluster-wide concurrency limits; BE execution is associated with workload-management accounting where the executor supports it.

Integration with the existing job commands

The grammar already accepts the required shapes on both master and branch-4.1: SHOW BUILD INDEX ((FROM|IN) db)? wildWhere? sortClause? limitClause? and CANCEL BUILD INDEX ON <multipart table> (job_ids) (DorisParser.g4: master fe-sql-parser :392-393/:644-646; branch-4.1 fe-core :390-391/:640-642). The current implementations are internal-only on both branches: CancelBuildIndexCommand.validate calls Util.prohibitExternalCatalog (:80) and resolves an OlapTable + SchemaChangeHandler.IndexChangeJob; ShowBuildIndexCommand builds the internal proc path /jobs/<dbId>/build_index (:188-191) with internal-specific columns (PartitionName, AlterInvertedIndexes, TITLE_NAMES :68-74). The design:

Registry: a catalog-agnostic external-index job registry in FE, edit-log persisted and rebuilt on replay, sibling to the internal IndexChangeJob registry. Lance jobs register here; the internal path is untouched.
Proc/SHOW source: SHOW BUILD INDEX unions internal and external jobs. A new JobType column (INTERNAL / EXTERNAL_LANCE) distinguishes rows. Internal-only columns stay empty for external rows (Lance jobs are table-level, not partition-level). External rows populate the generic columns: JobId, DbName (catalog-qualified), TableName, IndexName, State, Progress, CreateTime, FinishTime, Msg.
Catalog qualification: CANCEL's multipart table name already parses catalog.db.tbl (grammar + LogicalPlanBuilder.visitCancelBuildIndex are catalog-aware). SHOW's FROM clause currently takes a single identifier only; it is extended to accept a catalog-qualified catalog.db (preferred, consistent with the table qualification) — exact grammar detail in the PR.
Filtering and privileges: external rows are visible with table-level SHOW on the resolved external table; CANCEL requires table-level ALTER.
State mapping: external-index job states are displayed verbatim (PENDING … OUTCOME_UNKNOWN); WHERE filtering on State uses these same job-state values; documentation maps them to the internal lifecycle.
Cancellation semantics: unchanged from v2 (persisted before workers are signaled; fences COMMIT; generation-checked results). For REST-tracked operations there is no Namespace cancel operation: cancellation fences the Doris-side commit/refresh and stops polling; a later external completion is handled by the provenance rules in §10.
Conflict reporting: automatic conflict retry decisions are never made by parsing native exception strings; typed conflict results are used where the provider exposes them (unchanged from v2).

5. Execution model

Directory Catalog: BE pipeline + segment contract

Production execution requires extending lance-c (prerequisite, §12). FE orchestrates; BE workers build; FE performs the single atomic logical-index commit through the pinned Java SDK.

Commit path (verified against pinned sources): provenance-carrying commits go through the general transaction path — Transaction.Builder.transactionProperties(Map) (Transaction.java:176-179) carrying the job's operation_id, with operation.CreateIndex, executed via CommitBuilder (JNI wiring: lance-jni/src/transaction.rs:830-846). The one-shot Dataset.createIndex(IndexOptions), commitExistingIndexSegments, and dropIndex paths do not carry transaction properties (they build the commit internally with None properties; create/commit-existing: rust/lance/src/index.rs:1538-1547, drop: rust/lance/src/index.rs:1177-1194) and are therefore not used for provenance commits.

Segment contract:

Compatibility: the pinned lance-c and the pinned Java SDK must be verified against the same Lance core index-metadata format. The commit-side API surface is verified in the pinned sources: Index.Builder (Index.java:187-258), commitExistingIndexSegments (Dataset.java:1176), IndexOptions.withFragmentIds/withIndexUUID (IndexOptions.java:135-157), and the commit path's own validation (non-empty set, unique UUIDs, non-overlapping fragment coverage; rust/lance/src/index.rs:100-131). The full loop is additionally verified end to end by a PoC on lance-core 9.1.0-beta.3: fragment-scoped uncommitted builds over disjoint fragment subsets → single coordinator commit → dataset version increment by exactly one → k-NN query consumes the resulting multi-segment logical index. An end-to-end Doris test will prove the same for a BE-built (lance-c) segment committed via the Java SDK.
BE→FE segment metadata schema: per segment, workers report: uuid, field IDs, name, datasetVersion, indexVersion, indexType, fragment coverage, and the indexDetails protobuf bytes (createdAt/baseId optional; createdAt is filled by the commit side). This is the exact field set the commit path requires — validated statically (JNI index_metadata_to_segment, lance-jni/src/blocking_dataset.rs:1218-1243, rejects segments missing fragment coverage or index details; segment.fields is validated against the indexed column's field id, rust/lance/src/index.rs:1469-1475) and empirically (PoC: a Builder-reconstructed Index missing any of these fails before commit). Bare UUIDs are not sufficient.
Segment identity: every segment carries its own UUID — commitExistingIndexSegments rejects duplicate segment UUIDs (PoC-verified); the "shared UUID" wording in withIndexUUID's javadoc belongs to the separate merge flow. FE pre-assigns per-worker UUIDs (via withIndexUUID where the executor supports it) so the expected UUID set is known before dispatch; this set backs the provenance rule in §10.
Worker result acceptance: before accepting a worker result, FE validates per segment: field IDs, physical index type and parameters, starting dataset version, exact assigned fragment coverage (disjoint across workers and equal to the assignment), UUID uniqueness, and artifact location beneath the dataset index directory.
Full final segment set: CREATE/REPLACE commit the complete segment set for the starting snapshot. Incremental BUILD commits the complete final logical segment set — existing compatible segments plus newly built ones — not only the new segments (PoC-verified; note that a follow-up commit on an already-committed name uses replace semantics in the pinned SDK).
IVF model strategy (Phase 1): shared model. A dedicated training task derives IVF centroids and PQ codebooks once from a sample of the starting snapshot; all build workers use the shared model — concretely, the pinned Java SDK supports exactly this split (centroids trained once via VectorTrainer.trainIvfCentroids, distributed to builders through IvfBuildParams.setCentroids; PoC-verified across workers); on the BE path this depends on lance-c extension ① accepting caller-supplied centroids/codebooks (§12). This keeps segments mutually compatible for a future physical merge and yields uniform recall across segments. Independent per-segment models (valid for fan-out query) are rejected for Phase 1 because they preclude later merge. Physical segment merge itself is not performed in Phase 1.
Credentials: never persisted in the job record; obtained/refreshed at task dispatch (§4).
Local/file restriction (restored from v2): a local/file dataset is mutable only when the deployment guarantees that the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation; otherwise mutation is rejected. Local filesystem support is intended primarily for single-node development and tests; it must not silently depend on the current master FE's local disk.

FE experimental fallback: unchanged gating (experimental, disabled by default, size/fragment/concurrency limits). Implementation uses the Java SDK's existing uncommitted primitives; no lance-c dependency.

REST Namespace Catalog: capability matrix

Capability detection is behavioral: the pinned Java interface represents unsupported operations as default methods throwing a typed UnsupportedOperationException bound to spec error code 0 (LanceNamespace.java:402-448, errors/ErrorCode.java:22); there is no capability-advertisement API. Unsupported combinations fail during analysis where statically known, otherwise with a clear capability error. Doris never falls back to direct Dataset mutation with vended credentials. All REST mutations use the durable job record.

Operation	REST Phase-1 (pinned Namespace 0.7.7)
`SHOW INDEX`	Yes — via `ListTableIndices` + `DescribeTableIndexStats`. Fields limited to the stats model: name, UUID, columns, status; index type, distance, indexed/unindexed row counts, physical index count (spec.yaml:3149-3234). Fragment counts and creation properties are unavailable → corresponding SHOW fields are empty.
`CREATE INDEX`	Restricted — only what `CreateTableIndexRequest` carries: column, index_type, name, distance_type (FTS fields unused; spec.yaml:3049-3108). IVF build properties (`num_partitions`, `num_sub_vectors`) and BTREE `zone_size` are rejected for REST tables.
`CREATE OR REPLACE INDEX`	Rejected — the request model has no replace semantics and the spec defines `TableIndexAlreadyExists` (error code 7); a non-atomic drop+create pair is not offered under OR REPLACE.
`BUILD INDEX`	Rejected — no incremental/uncovered-fragment operation exists in the Namespace API.
`DROP INDEX`	Yes — via `DropTableIndex` (LanceNamespace.java:445); still uses the durable job record.
`lance_index_segments` TVF	Rejected — the stats model exposes no per-segment metadata.

Monitoring and provenance: progress is polled via ListTableIndices / DescribeTableIndexStats (the spec-sanctioned channel). If the server returns a transaction_id, it is persisted as the REST job's only server-side evidence (§10). The pinned request model cannot carry a client operation ID or transaction properties; adding a namespace-level idempotency/provenance key is listed as a spec-evolution follow-up.

(Alternative considered: make Namespace spec/API changes — replace semantics, build parameters, incremental build — an explicit prerequisite and offer uniform REST behavior. The matrix above is the Phase-1 proposal; the spec-evolution path can be a follow-up with the Lance community.)

6. Initial type and property matrix

IVF_PQ: SQL category ANN + index_type=IVF_PQ; exactly one fixed-size-list column; element types FLOAT16, FLOAT32 (others deferred, see corrections); field and elements non-null; metrics L2/COSINE/DOT; num_partitions, num_sub_vectors optional positive integers (num_sub_vectors must divide the dimension); num_bits fixed to 8.
BTREE: one column; integral/floating numeric, DECIMAL, STRING, DATE/DATETIME/TIMESTAMPTZ per the existing Lance pushdown support; nullable columns gated by the NULL-predicate tests; optional zone_size positive integer (Lance default 4096, DEFAULT_BTREE_BATCH_SIZE, rust/lance-index/src/scalar/btree.rs:91).
BITMAP: one column; BOOLEAN, integral numeric, STRING, DATE; explicit NULL predicate tests; no user-visible properties.
Deferred: IVF_FLAT (end-to-end query verification), Lance INVERTED (needs Doris FTS SQL semantics; to be exposed as FTS), UINT8/INT8/FLOAT64 (see corrections).

7. Logical and physical metadata surfaces

SHOW INDEX: unchanged 13-column schema (ShowIndexCommand.java:52-66, identical on master and branch-4.1). One row per logical index/column: the pinned Lance list APIs return one entry per physical segment — multiple same-named entries for a multi-segment logical index (PoC-verified) — so SHOW INDEX groups by logical name. Index_type reports the physical Lance type (IVF_PQ, BTREE, BITMAP) rather than the SQL category. Properties carries bounded logical details only (physical type, metric, parameters, aggregate indexed/unindexed rows and fragments from the logical description) — never segment arrays. The logical index count is the number of user-visible logical index descriptions (i.e., distinct grouped names), excluding Lance system indexes, and is not derived from physical manifest entries; no separate count statement is added.
lance_index_segments TVF: catalog, database, table, logical index name, segment UUID, physical index type, dataset version, creation timestamp when available. Per-segment row/fragment counts are deferred (see corrections): the lance-c list JSON carries per-entry {name, uuid, columns, type, dataset_version} and no fragment coverage or row counts; this becomes lance-c extension item ⑥.
REST: SHOW fields are capability-dependent per §5; the TVF is rejected.

8. Authorization and identifier semantics

Unchanged from v2: mutation + CANCEL require table-level ALTER; SHOW surfaces require table-level SHOW. Doris resolves user-supplied index names case-insensitively (quoted and unquoted identifiers alike) and preserves display case; new case-only duplicates are rejected; pre-existing external collisions surface an explicit ambiguity error on mutation. Note the layered reality (corrected research): Lance itself is case-sensitive and exact-matching (create.rs:202, api.rs:211), so Doris's case-insensitive resolution is a Doris-layer policy over exact-match storage — which is exactly why external case-only collisions are a real state that must be handled explicitly.

9. Versioning, concurrency, and visibility

Successful create/replace/build/drop commits a new dataset version atomically; queries pinned to older versions are unaffected; historical-version mutation is rejected.
Concurrent same-name CreateIndex (external writers): the format documentation defines last-committer-wins; the pinned implementation reaches it via retryable conflict + rebase (see corrections). Doris serializes same-name jobs internally (case-insensitive) but does not control external writers, which may retry and therefore replace; reconciliation never treats a name/definition match as proof of attribution (§10).
Conflicts (per transaction.md:175-194): Overwrite/Restore/UpdateMemWalState conflict with index creation. Rewrite conflicts only with overlapping fragments and no stable row ids and no fragment reuse index; DataReplacement conflicts only with overlapping fragments when the replaced column is being indexed — these are surfaced as retryable. Concurrent append is compatible; appended fragments are reported as unindexed and covered by a later BUILD.
Doris does not hold table metadata locks for the lifetime of a build.
Conflict-retry decisions are never made by parsing native exception strings (typed conflict results only).
Queries over a partially indexed table must combine indexed results with scans of unindexed fragments; end-to-end tests must prove result equivalence with index use disabled.

10. Failure recovery, provenance, and metadata refresh

The three outcome classes from v2 are retained, with a strengthened and honestly-bounded success criterion.

Confirmed pre-commit failure

Unchanged: job FAILED; no post-DDL refresh; unreferenced artifacts reclaimed by Lance cleanup per retention.

Confirmed commit — requires job-specific evidence

Directory (create/replace/build): the logical index in the reopened latest snapshot is backed by exactly the job's expected segment UUID set — pre-assigned by FE before dispatch where the executor supports it (§5), so the expectation is deterministic rather than observed; the commit itself carries the job's operation_id in the Lance transaction properties via the general Transaction path (§5; runtime-verified readable back from the new version's transaction).
Directory (drop): the pinned SDK has no DropIndex operation class, and the one-shot Dataset.dropIndex commits internally with None properties (rust/lance/src/index.rs:1177-1194), so it is not used for provenance commits. A provenance-carrying drop goes through the same general Transaction path, expressed as operation.CreateIndex with an empty new-index list and removedIndices set to the index metadata loaded for the name before dispatch (withRemovedIndices, operation/CreateIndex.java:92-94) — mirroring the Rust one-shot's internal operation — with transactionProperties carrying the job's operation_id. That recorded operation_id is the job-specific evidence; the exact removed segment UUID set is further corroboration. The postcondition "name is absent" is corroboration only — an external writer can independently drop the same name (false positive) or recreate it after our drop (false negative). Without transaction-properties evidence, a Directory drop follows the unknown-outcome rule below.
REST: the pinned Namespace API cannot carry a client operation ID in index requests. The only job-specific evidence is the server-returned transaction_id (persisted when returned), matched via the generic transaction API (DescribeTransaction, LanceNamespace.java:754) where the server supports it. A matching logical definition observed through list/stats is corroboration only.

On a confirmed commit, Doris invalidates/refreshes metadata and emits the external-DDL refresh edit log; a refresh failure transitions the job to COMMITTED_REFRESH_PENDING with retry (unchanged).

Unknown or post-commit failure

Reconcile by reopening the latest snapshot / querying the Namespace state, as in v2 — but if no job-specific evidence is available (expected: FE failover before persisting the returned transaction_id; a drop without transaction-properties evidence; any REST operation on a server that returns no transaction_id), the job remains OUTCOME_UNKNOWN with a diagnostic. It is never marked successful from a matching postcondition alone — this is required precisely because same-name CreateIndex is last-committer-wins for external writers.

A reconciled IF NOT EXISTS / IF EXISTS no-op still invalidates/refreshes Doris metadata when newer external state is observed (unchanged).

11. Test plan

v2 coverage, plus:

capability matrix: every rejected REST combination fails with the designed error (analysis-time vs capability error);
REST IF NOT EXISTS compares only stats-observable fields (name, columns, type, distance);
provenance: Directory commit carries operation_id in transaction properties (read back from the new version's transaction); commit-with-lost-response reconciled via UUID set; drop commits are expressed as operation.CreateIndex with removedIndices over the general Transaction path (verified through JNI) and reconciled only via transaction-properties evidence; matching-postcondition-without-evidence stays OUTCOME_UNKNOWN (including REST with no transaction_id returned);
external same-name replacement during a Doris job (retried create replacing ours) is detected and attributed correctly;
DROP failover: drop commits externally, FE fails before refresh → resumed from the durable record;
incremental BUILD commits the full final segment set (existing + new), validated against duplicate-UUID / overlapping-coverage rejection;
BE→FE segment metadata schema: segments missing fragment coverage or index details are rejected before commit (mirroring the JNI conversion requirements);
shared-model build: segments built by different workers are merge-compatible and fan-out query recall is uniform;
SHOW/CANCEL BUILD INDEX: union display, JobType column, catalog qualification, privilege filtering, external-index job states, cancellation fencing;
resource admission: per-catalog and cluster-wide concurrency limits enforced;
local/file visibility restrictions (mutation rejected when the path is not uniformly visible);
job record contains no credentials (edit-log content test);
REST monitoring via list/stats polling; transaction_id persisted when present.

12. Confirmed design decisions and implementation decomposition

Execution: Directory = BE via extended lance-c, production-gated until the required APIs land; REST = Namespace capability matrix; no external build service (deployment/auth/ops dependency for a core feature is not justified); FE sync build = experimental fallback only.
lance-c extensions (prerequisite): ① fragment-scoped uncommitted index builds, including caller-supplied shared IVF centroids / PQ codebooks (so the §5 shared-model strategy reaches BE workers); ② segment metadata serialization/transfer (BE→FE per §5 schema); ③ progress reporting; ④ cooperative cancellation; ⑤ segment merge / commit-existing; ⑥ per-segment coverage/stats exposure (feeds the TVF deferral). Upstream engagement starts now — lance-c is actively landing related distributed APIs (v0.1.3 added segment-selection for distributed search).
Segment contract: pinned compatibility, BE→FE metadata schema, validation checklist, full-final-set commits, shared IVF model, no persisted credentials.
Commit path: provenance commits use the general Transaction path (Transaction.Builder.transactionProperties + operation.CreateIndex + CommitBuilder); one-shot convenience APIs are not used where provenance is required.
Provenance: operation_id persisted pre-dispatch; success requires job-specific evidence; otherwise OUTCOME_UNKNOWN. REST boundary per §10 is stated explicitly.
DROP durable; SHOW/CANCEL BUILD INDEX integration per §4; resource admission per §4.
SQL: USING ANN + index_type=IVF_PQ; OR REPLACE = full rebuild; BUILD = incremental; SHOW Index_type = physical type.
Types: IVF_PQ (FLOAT16/FLOAT32) + BTREE + BITMAP only.
REST: capability matrix; mutation never bypasses the service.

Work items (dependency-ordered; deliverable as separate PRs):

lance-c extensions (upstream; tracked prerequisite subtask, then pin bump);
FE neutral command model (ParsedIndexSpec + providers; internal behavior unchanged);
Durable job framework + SHOW/CANCEL integration (registry, edit log, proc union, privileges, resource admission);
BE build pipeline (training task, build workers, result validation, credential handling);
Commit, provenance, reconciliation (general Transaction path, segment contract enforcement, refresh retry);
Metadata surfaces (SHOW INDEX physical type, TVF, REST capability matrix);
Tests and documentation per §11 + lifecycle guarantees/limitations doc.

Questions for reviewers

Q1 — target branch: [feature](lance) support lance catalog in doris #65730 landed on branch-4.1 only; master currently has no Lance catalog/FE code, and the branches have diverged structurally (the SQL grammar lives in fe/fe-sql-parser on master vs fe/fe-core on branch-4.1; both verified). Should [Feature](lance) Add vector and scalar index lifecycle management #66497 implementation target branch-4.1 only, master-first with a 4.1 backport, or dual-track?
Q2 — metric property key: since vector creation reuses the Doris category surface (USING ANN), I propose adopting the internal ANN property key metric_type (verified required by AnnIndexPropertiesChecker). One caveat: internal ANN currently accepts only l2_distance and inner_product — COSINE would be a newly introduced value for this key, and Lance value names differ (l2/cosine/dot). Options: (a) metric_type with Lance value names (l2/cosine/dot), documented as Lance-table-specific values; (b) metric_type with Doris value names mapped to Lance (l2_distance→L2, inner_product→DOT, plus new cosine→COSINE); (c) fully Lance-native key metric. My preference is (a) for key consistency with internal ANN while keeping Lance-native values. Build parameter names stay Lance-native (num_partitions, num_sub_vectors, num_bits) in all options.
Q3 — lance-c packaging: confirm the lance-c extension is tracked as an explicit prerequisite subtask (upstream issue + PRs, then a pin bump in Doris), with [Feature](lance) Add vector and scalar index lifecycle management #66497 remaining the parent lifecycle issue until the native and Doris work is integrated end to end. Related: the TVF per-segment coverage columns (see corrections) ride on extension item ⑥ — please confirm the Phase-1 TVF ships without them.

References

v2 proposal and round-1/round-2 review threads on this issue
[feature](lance) support lance catalog in doris #65730, [Tracking] Lance feature matrix for Doris 4.2 #66340
Lance transaction semantics: https://lance.org/format/table/transaction/ (pinned: docs/src/format/table/transaction.md:162-194); implementation conflict path: rust/lance/src/io/commit/conflict_resolver.rs:577-617 (pinned v9.1.0-beta.3)
Lance distributed indexing: https://lance.org/guide/distributed_indexing/
Lance Namespace operations + spec: https://lance.org/format/namespace/operations/ (pinned: spec.yaml:1633-1634, 3049-3234)
Pinned Lance Java SDK v9.1.0-beta.3: Dataset.java (:590/:607/:1106/:1176), IndexOptions.java (:135-157), Index.java (:187-258), Transaction.java (:176-179), operation/CreateIndex.java (:92-94), CommitBuilder.java (:75,:273); JNI: lance-jni/src/transaction.rs:830-846, lance-jni/src/blocking_dataset.rs:1218-1243
Pinned lance-c v0.1.2 (include/lance/lance.h); upstream lance-c v0.1.3 segment-selection APIs
Doris (master + branch-4.1): CancelBuildIndexCommand.java (:80), ShowBuildIndexCommand.java (:68-74,:188-191), ShowIndexCommand.java (:52-66), DorisParser.g4 (master fe-sql-parser :392-393/:644-646; 4.1 fe-core :390-391/:640-642), AnnIndexPropertiesChecker.java (:42-46)

Thanks for the v3 revision. After checking the confirmed 4.2 release boundary and re-reading the baseline implementation in #65730, I need to correct part of my previous review and narrow the proposal.

There are two confirmed product constraints:

Doris 4.2 is based on branch-4.1, not master. Therefore this work should target branch-4.1. There is no need for a master-first or dual-track implementation in this issue.
No functionality that requires extending lance-c will be included in this release. Such work is explicitly outside the 4.2 scope.

These constraints materially change the appropriate Phase-1 design.

Query syntax: keep the existing vector_search() contract

I rechecked #65730. The baseline deliberately exposes Lance vector search only through the vector_search() TVF:

#65730

The implementation and regression tests use:

SELECT row_id, _distance
FROM vector_search(
    "table" = "catalog.db.table",
    "column" = "embedding",
    "query_vector" = "[0,0,0,0]",
    "metric" = "l2",
    "top_k" = "5"
);

There is no implementation or stated plan in the baseline to translate internal-table ANN query forms such as l2_distance_approximate(...) or inner_product_approximate(...) into a Lance search.

Therefore:

Lance queries should continue to use vector_search().
This index-management issue should not add compatibility with the internal-table ANN query syntax.
Reusing USING ANN as a category in CREATE INDEX is only DDL/parser reuse; it does not imply query-syntax reuse.
For Q2, I now recommend the Lance-native property already used by the existing query surface: "metric" = "l2|cosine|dot" (option c), rather than introducing metric_type with internal-table values.

Query behavior is relevant to this issue only as end-to-end verification: an index created by the lifecycle API must be consumed correctly by the existing vector_search(..., "use_index"="true") path, with a compatible metric and correct results. This issue should not redesign the query syntax or optimizer rules.

Scope the implementation to the existing lance-c 0.1.2 APIs

The pinned lance-c 0.1.2 already exposes the one-shot lifecycle APIs required by the original issue:

lance_dataset_create_vector_index
lance_dataset_create_scalar_index
lance_dataset_drop_index
lance_dataset_index_count
lance_dataset_index_list_json

https://github.com/lance-format/lance-c/blob/v0.1.2/include/lance/lance.h#L490-L541

The 4.2 design should use those existing APIs, with the heavy direct-Dataset operation executed on a selected BE for Directory Catalog, and use the existing Namespace operations for the supported REST cases. The limitations of the one-shot API—no native progress reporting, cooperative cancellation, distributed fragment build, or job-specific transaction provenance—should be documented honestly rather than solved by expanding lance-c in this release.

The following v3 items depend on new lance-c capabilities and should be removed from the 4.2 implementation scope:

fragment-scoped distributed builds;
shared IVF centroid/PQ-codebook distribution to BE workers;
BE-to-FE uncommitted segment metadata transfer;
coordinator commit of worker-built segments;
physical segment merge;
native progress and cooperative cancellation;
per-segment coverage/statistics extensions;
incremental BUILD over uncovered fragments.

In particular, the original #66497 scope requires replacing or rebuilding an index with the same name. The existing replace=true path provides a full rebuild. Incremental BUILD INDEX was added during review, but it is not required by the original issue and should be deferred because the intended distributed/incremental implementation requires APIs outside the 4.2 boundary.

Revised Phase-1 lifecycle

For 4.2, I recommend limiting the lifecycle to:

CREATE INDEX for the verified vector and scalar types;
CREATE OR REPLACE INDEX mapped to the existing full-build replace=true behavior;
SHOW INDEX, using the existing list/count metadata and grouping physical entries by logical name where necessary;
DROP INDEX;
restricted REST behavior based only on the pinned Namespace request/response models;
metadata refresh, privileges, version rules, concurrency limitations, and failure behavior appropriate to these existing one-shot operations.

Do not promise a reliable native CANCEL operation if the underlying call cannot be interrupted. Likewise, SHOW should not promise coverage fields that require a new lance-c API. UUID, columns, type, and dataset version are already available and satisfy the original metadata scope.

The query-side acceptance test should be limited to proving that a created IVF_PQ index is used by the existing vector_search() path and that scalar indexes are consumed by the existing Lance predicate path. FTS SQL design, internal ANN query syntax, distributed recall/model strategy, and future physical merge behavior are outside this issue.

Answers to the v3 questions

Q1: target branch-4.1 only. This is the confirmed base for Doris 4.2.
Q2: use the existing Lance-facing metric key and values (l2, cosine, dot), matching vector_search(). Do not require internal ANN query/property syntax compatibility.
Q3: lance-c extensions are not prerequisites for this version; they should be tracked as future work. Any feature that requires such an extension, including per-segment coverage, distributed BUILD, progress, cancellation, or segment merge, must be deferred rather than gating [Feature](lance) Add vector and scalar index lifecycle management #66497.

With these scope corrections, v3 should be simplified substantially. The goal of this issue is index lifecycle management using the capabilities already present in the 4.1 baseline and pinned SDKs, not a distributed index-build framework or a unified internal/external vector-query syntax.

Quick update on the prerequisite track (v3 §12 decision 2 / Q3): the upstream engagement has started — I've filed lance-format/lance-c#55, an RFC covering all six lance-c extensions, split into three independently-mergeable waves:

Wave 1 (blocking): fragment-scoped uncommitted segment builds with shared-model injection (caller-supplied IVF centroids / PQ codebook) + segment metadata serialization (pb::IndexMetadata bytes as the BE→FE transfer format).
Wave 2 (production quality): build progress reporting + cooperative cancellation.
Wave 3 (symmetry/observability): commit-existing-segments C API + per-segment coverage/stats in listing.

One finding worth surfacing here, since it de-risks the schedule: the Rust core (9.1.0-beta.3) already implements nearly everything behind the C boundary. CreateIndexBuilder already carries fragments / index_uuid / train / progress / transaction_properties and execute_uncommitted() -> IndexMetadata (rust/lance/src/index/create.rs:50-141), with scalar fragment-scoped builds supported as well (:270); the injection points IvfBuildParams.centroids and PQBuildParams.codebook both exist. The only genuine core-side gaps are the cancellation hook and per-segment listing stats — the rest is FFI exposure.

This does not presume the Q3 answer: if round-3 prefers a different tracking form, the upstream issue adapts. Round-3 feedback welcome whenever convenient — no urgency imposed by this step.

Thanks for starting the upstream discussion. The Rust-core findings are useful for future distributed-index work, and lance-format/lance-c#55 can remain as that future track.

However, this does not change the confirmed scope for Doris 4.2, and Q3 is no longer open: lance-c extensions are not a prerequisite for #66497 in 4.2. Features that depend on those extensions must be deferred. Therefore, Wave 1 should not be described as blocking the 4.2 implementation or as de-risking its current schedule; it is outside the release's critical path.

For #66497, the next design revision should be simplified around the existing branch-4.1 baseline and the pinned lance-c 0.1.2 one-shot APIs:

lance_dataset_create_vector_index and lance_dataset_create_scalar_index;
full rebuild/replacement through the existing replace=true path;
lance_dataset_index_count and lance_dataset_index_list_json for the currently available metadata;
lance_dataset_drop_index;
execution of the heavy direct-Dataset call on a selected BE for Directory Catalog.

The 4.2 design should explicitly exclude fragment-scoped distributed builds, shared-model distribution, BE-to-FE uncommitted segment transfer, coordinator segment commit, incremental BUILD INDEX, physical merge, native progress, cooperative cancellation, and per-segment coverage/statistics extensions. It should also avoid promising reliable CANCEL BUILD INDEX semantics when the underlying one-shot native call cannot be interrupted.

The REST path is separate: lance-c additions do not expand REST Catalog capabilities. REST index lifecycle operations must continue to use the pinned Namespace request/response models and the server's implemented capabilities. In particular, newer lance-c APIs would not provide REST OR REPLACE, incremental build, cancellation, or additional request parameters.

One future-facing note for lance-c#55: pb::IndexMetadata bytes are not directly accepted by the pinned Java SDK. Index.Builder takes the metadata as individual fields and exposes no public parseFrom(byte[]) entry point, so the cross-SDK design still needs an explicit Java decoder/adapter and a version-compatibility contract. This does not need to be solved for #66497 because that distributed path is out of scope for 4.2.

Please make the next revision a 4.2 design based only on the existing APIs, covering BE selection/execution, supported index types and properties, replace=true semantics, privileges, concurrency, versioning, failure behavior, metadata refresh, local-path visibility, and the restricted Namespace 0.7.7 REST capability matrix. Query-side work should remain limited to validating index consumption through the existing vector_search() and scalar predicate paths.

@Gabriel39

Revised Design Proposal v4 — Lance Index Lifecycle, scoped to the 4.2 boundary

Draft for #66497, prepared 2026-08-07. Incorporates @Gabriel39's round-3 review (issuecomment-5211673675) and its follow-up (issuecomment-5212293290): target branch-4.1 only (the confirmed 4.2 base), no functionality requiring lance-c extension in this release, query consumption stays on the existing vector_search() TVF, and the v3 distributed-build machinery moves to a separately tracked future-work item. v4 is intentionally a subtraction from v3: everything removed is named in §9 with its future-work tracking point.

/cc @Gabriel39 @zhangstar333

Major changes from v3

Scope cut to the one-shot pinned APIs. Directory builds execute the existing lance-c v0.1.2 one-shot lifecycle calls on a selected BE (lance_dataset_create_vector_index / create_scalar_index / drop_index / index_count / index_list_json, lance.h@v0.1.2 L490-541); REST keeps the v3 capability matrix on the pinned Namespace 0.7.7 model. No new native capability is required anywhere.
BUILD INDEX (incremental) removed from 4.2 — it was added during review and is not required by the issue; its intended implementation needs APIs outside the 4.2 boundary. The statement is rejected for Lance tables with an explicit error (§2).
Provenance rewritten honestly (§6). The one-shot C calls commit internally and carry no client operation ID and no pre-assigned segment UUID set. v3's deterministic Directory provenance is unavailable in 4.2; reconciliation after a lost response is explicitly bounded to OUTCOME_UNKNOWN + diagnostics. This is the proposed 4.2 trade-off, with the upgrade path tracked in §9.
CANCEL is best-effort and documented as such (§5): it fences Doris-side tracking and refresh; it does not interrupt the native build.
Query consumption is verification-only: acceptance proves an index built by this lifecycle is consumed by the existing vector_search(..., "use_index"="true") path (branch-4.1 VectorSearchTableValuedFunction.java:81,158-159) with compatible metric and correct results, and that scalar indexes are consumed by the existing Lance predicate path ([feature](lance) support lance catalog in doris #65730). No query-syntax or optimizer work.
Q2 resolved per round-3: property key is Lance-native metric with values l2|cosine|dot, matching the existing query surface (option c). USING ANN remains DDL category reuse only.
lance_index_segments TVF deferred: its distinct value was per-segment coverage, which is exactly the deferred capability; the remaining columns duplicate SHOW INDEX's grouped output. It returns with the future-work track (§9).
FE-side experimental build fallback removed: it existed to de-risk the distributed path. With the one-shot BE path as the production form, keeping it would add a second, non-production execution mode against the "simplified substantially" direction.

1. Goals and Phase-1 boundary (4.2)

Provide the issue's lifecycle over the pinned SDKs, nothing more:

CREATE INDEX for IVF_PQ vector indexes and BTREE/BITMAP scalar indexes (Directory full; REST per §4 matrix);
CREATE OR REPLACE INDEX = full rebuild via the existing replace=true behavior;
SHOW INDEX — logical definitions, names, UUIDs, columns, types, dataset versions (grouped by logical name where a table carries multiple physical entries per name; REST fields limited to the §4 stats model);
DROP INDEX by name (Directory and REST);
durable job status, admission limits, best-effort cancellation, metadata refresh, privileges, and honestly-bounded failure recovery;
Lance dataset manifests / the Namespace service remain the authoritative index metadata; Doris persists only its own job state.

Explicit non-goals for 4.2 (each mapped in §9): fragment-scoped distributed builds, shared-model distribution, uncommitted segment transfer, coordinator commit of worker segments, physical segment merge, native progress/cancellation, per-segment coverage/statistics, incremental BUILD, FTS SQL semantics, internal-table ANN query syntax compatibility.

Historical-version tables remain read-only for index mutation; inspection uses the selected snapshot, mutation targets only the latest writable state.

2. SQL surface

Vector: CREATE INDEX idx ON t (embedding) USING ANN PROPERTIES ("index_type"="IVF_PQ", "metric"="l2", "num_partitions"="256", "num_sub_vectors"="16").
- "metric" accepts l2|cosine|dot (Lance-native values, matching vector_search()). Unknown keys/values are rejected at analysis.
Scalar: USING BTREE / USING BITMAP (optional "zone_size" for BTREE per the pinned Lance default, unchanged from v3).
Type and constraint matrix unchanged from v3 §6: IVF_PQ on exactly one fixed-size-list column; element types FLOAT16/FLOAT32 only (UINT8/INT8/FLOAT64 deferred per the v3 corrections); field and elements non-null; num_sub_vectors must divide the dimension; num_bits fixed to 8; BTREE/BITMAP column-type lists per v3 §6.
CREATE OR REPLACE INDEX = full rebuild + atomic same-name replacement via replace=true.
SHOW INDEX FROM t; DROP INDEX [IF EXISTS].
SHOW BUILD INDEX FROM db WHERE ...; CANCEL BUILD INDEX ON ctl.db.tbl (job_id) — retained (builds are async jobs), semantics per §5.
BUILD INDEX is rejected for Lance tables in 4.2 with an explicit unsupported-operation error naming the deferral — not silently accepted, not partially implemented.

Statement semantics: create/replace/drop each return after a durable job is accepted. IF NOT EXISTS / IF EXISTS reconciliation compares name + column + physical type (Directory) — definition fields not readable back through the pinned list JSON (metric, build parameters) do not participate, so a same-name/column/type index with different parameters is treated as existing; this limitation is documented; REST compares only stats-observable fields per §4. OR REPLACE remains convergent-but-not-idempotent. Single column per index. USING ANN is DDL/parser category reuse only — it implies nothing about query syntax. Carried over unchanged from prior revisions: conflict-retry decisions are never made by parsing native exception strings; IF NOT EXISTS and OR REPLACE are mutually exclusive; USING is required for Lance tables (no implicit default index type).

3. Neutral command model and dispatch

Unchanged from v3: ParsedIndexSpec → resolve catalog/database/table → InternalIndexProvider or LanceIndexProvider → provider validation → internal schema change or Lance external-index job. Lance definitions never enter Doris internal catalog.Index serialization. Implementation targets branch-4.1 only.

4. Execution model and REST capability matrix

Directory Catalog: one-shot build on a selected BE

FE admits the job (§5), selects a BE (any healthy BE for object-storage datasets; the local/file restriction below constrains the choice for local datasets), and dispatches one task. The BE opens the dataset through lance-c v0.1.2 and executes the matching one-shot call, which trains, builds, and commits internally as a single dataset version. FE then refreshes metadata (§6). There is no FE-side commit in 4.2.

Documented limitations of this execution form (honest, not solved in 4.2): no native progress reporting; no cooperative interruption; whole-dataset build in one BE process (memory/compute on a single node); commit carries no job-specific provenance (§6). These are exactly the capabilities tracked upstream in §9.

Local/file restriction (unchanged from v3): a local/file dataset is mutable only when the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation; otherwise mutation is rejected.

REST Namespace Catalog: capability matrix (rows unchanged from v3; the TVF row is subsumed by the global TVF deferral, change 7)

Behavioral detection via typed UnsupportedOperationException (spec error code 0); unsupported combinations fail at analysis where statically known, otherwise with a clear capability error; Doris never falls back to direct Dataset mutation with vended credentials.

Operation	REST 4.2 (pinned Namespace 0.7.7)
`SHOW INDEX`	Yes — `ListTableIndices` + `DescribeTableIndexStats`; fields limited to the stats model (name, UUID, columns, status; type, distance, row counts where offered).
`CREATE INDEX`	Restricted — only what `CreateTableIndexRequest` carries (column, index_type, name, distance_type; FTS fields unused). IVF build properties and BTREE `zone_size` are rejected for REST tables.
`CREATE OR REPLACE INDEX`	Rejected — no replace semantics in the request model; a non-atomic drop+create pair is not offered.
`BUILD INDEX`	Rejected (also deferred globally, §2).
`DROP INDEX`	Yes — via `DropTableIndex`; durable job record applies.

Monitoring via the spec-sanctioned ListTableIndices / DescribeTableIndexStats polling; transaction_id persisted when returned (§6).

5. Durable jobs, SHOW/CANCEL integration, admission

Job record (edit-log persisted; simplified from v3): job id, operation kind (CREATE / REPLACE / DROP), catalog/db/table identity, dataset URI or Namespace table id, logical index name and normalized definition, starting dataset version, state, timestamps, sanitized failure information, Namespace transaction id when returned. No credentials — obtained/refreshed at dispatch.

States (simplified — the one-shot call fuses build and commit): PENDING → RUNNING → REFRESHING → FINISHED, plus FAILED / CANCELLING / CANCELLED / COMMITTED_REFRESH_PENDING / OUTCOME_UNKNOWN. CANCELLING → CANCELLED completes once the fence is persisted. FE failover reconstructs unfinished jobs from the edit log and resumes polling or refresh; it does not blindly re-dispatch a build (§6).

SHOW/CANCEL BUILD INDEX integration as in v3 §4 (registry, proc union with JobType, catalog qualification, privileges: SHOW requires table-level SHOW, CANCEL requires table-level ALTER), minus any field that would require a new lance-c API; the Progress column for one-shot jobs is a coarse mapping from job state — no native build progress exists in 4.2.

Cancellation semantics (honest): CANCEL persists intent and fences Doris-side tracking, polling, and refresh. It cannot interrupt the native one-shot call. A fenced build may still complete and commit externally; the job's terminal state remains CANCELLED, and Doris performs no further observation of the fenced job — the externally committed index is picked up by the next reconciliation or metadata refresh (a subsequent DDL statement, manual REFRESH, or cache invalidation on access), and the DROP INDEX the operator issues performs that reconciliation itself. This is documented as best-effort cancellation, not abort.

Resource admission: per-catalog and cluster-wide concurrency limits on running build jobs, unchanged in intent from v3.

6. Failure recovery, provenance, and metadata refresh (rewritten for 4.2)

The v3 outcome classes are retained; the evidence standard is restated for what the pinned one-shot APIs can actually provide.

Confirmed pre-commit failure (BE/REST call returned an error): job FAILED; no refresh; unreferenced artifacts reclaimed by Lance cleanup per retention.
Confirmed success: Directory — the one-shot call returned success. REST — the request was accepted and completion observed through the §4 list/stats polling (REST index creation is asynchronous, spec.yaml:1633-1634: an accepted request is not a finished build). On confirmed success: invalidate/refresh metadata, emit the external-DDL refresh edit log; refresh failure → COMMITTED_REFRESH_PENDING with retry. In the failover flow (completion observation lost before the terminal state was persisted), REST confirmation additionally requires the persisted transaction_id; a list/stats postcondition match alone remains corroboration only, per the rule below.
Lost response / post-dispatch failure: reopen the latest snapshot / query the Namespace state. Because the one-shot commit carries no operation_id and Doris pre-assigns no segment UUID, a matching postcondition (index with expected name/column/type at a newer dataset version) is corroboration only — an external writer can independently create or replace the same name (Lance same-name CreateIndex is last-committer-wins). Without job-specific evidence the job remains OUTCOME_UNKNOWN with a diagnostic naming the ambiguity; it is never marked successful from a matching postcondition alone. For REST, the server-returned transaction_id (when present) remains the only job-specific evidence, matched via DescribeTransaction where supported.
DROP follows the same rule: postcondition "name absent" is corroboration only.
Refresh-on-observation: any reconciliation that observes newer external state — including a reconciled IF NOT EXISTS / IF EXISTS no-op or an OUTCOME_UNKNOWN diagnosis — still invalidates/refreshes Doris metadata, so a diagnosis never leaves a stale cache behind.
Operator exit for OUTCOME_UNKNOWN: such a job never retries automatically. The operator inspects actual state via SHOW INDEX (or the dataset directly), then either issues DROP INDEX if the mutation landed or re-issues the statement if it did not — CREATE OR REPLACE INDEX is the recommended re-issue form for create/replace ambiguities, converging manually to a known state.

This is a deliberate, stated weakening relative to v3 (which carried deterministic UUID-set + transaction-properties evidence). It follows from the confirmed 4.2 scope: the only remedies live in the deferred capabilities, and §9 names the upgrade path.

7. Metadata surfaces and authorization

SHOW INDEX: unchanged 13-column schema (branch-4.1 ShowIndexCommand.java:52-66); one row per logical index/column, grouping same-named physical entries; Index_type reports the physical Lance type (IVF_PQ/BTREE/BITMAP); Properties carries bounded logical details only — physical type; segment count; metric and build parameters where known from the creating job record (on Directory these are not readable back through the pinned list JSON, so for externally created indexes they stay empty; on REST, distance_type is populated from the stats model where offered, and build parameters are always empty — they are inexpressible there) — never segment arrays, no coverage fields. The logical index count is the number of distinct grouped names, excluding system indexes. Field provenance is stated per catalog kind: Directory — UUID, columns, type, and dataset version come from the existing index_list_json and satisfy the issue's metadata scope; REST — the pinned stats model exposes name, UUID, columns, status, and (where offered) type/distance/row counts only — no dataset version, so the corresponding SHOW fields stay empty (spec.yaml:3162-3184, :3206-3234).
Authorization and identifiers unchanged from v3: mutation + CANCEL require table-level ALTER; SHOW surfaces require table-level SHOW; Doris-layer case-insensitive name resolution over Lance's case-sensitive exact-match storage, with explicit ambiguity errors on pre-existing external case-only collisions.
Versioning/concurrency unchanged in the retained parts: mutation commits are atomic new dataset versions; historical-version mutation rejected; Doris serializes same-name jobs internally and does not hold table metadata locks for the lifetime of a build; queries over a table whose index predates appended data combine indexed results with scans of unindexed data — the acceptance suite proves result equivalence with use_index=false.

8. Test plan (4.2)

Create/replace/show/drop for IVF_PQ + BTREE + BITMAP on Directory; REST capability matrix rejections (analysis-time vs capability error) for every rejected combination;
Query consumption (acceptance): an IVF_PQ index created through this lifecycle is used by vector_search(..., "use_index"="true") with a compatible metric and returns correct results; scalar indexes are consumed by the existing Lance predicate path; equivalence with index use disabled;
IF NOT EXISTS / IF EXISTS reconciliation fields per §2/§4, including: same-name/column/type with divergent parameters treated as existing (documented Directory limitation); reconciled no-op refreshes metadata when newer external state is observed;
Job lifecycle: durable acceptance, failover reconstruction, refresh retry (COMMITTED_REFRESH_PENDING), admission limits enforced;
Cancellation honesty test: CANCEL fences tracking/refresh; a build that completes externally afterwards is picked up by the next reconciliation/refresh (§5) and is droppable;
Lost-response reconciliation stays OUTCOME_UNKNOWN (Directory and REST-without-transaction_id);
Local/file visibility restriction; job record contains no credentials (edit-log content test).

9. Moved to future work — tracked, not dropped

All removals land in one tracked place: lance-format/lance-c#55 (upstream RFC, filed 2026-08-07) plus a Doris-side follow-up issue to be opened when the 4.2 branch cuts. Every deferred item below is a Directory-path capability — per round-3 follow-up, lance-c additions do not expand REST Catalog capabilities: REST OR REPLACE, incremental build, cancellation, and additional request parameters require Lance Namespace spec evolution, tracked separately as a spec-level follow-up with the Lance community; nothing in lance-c#55 expands the §4 REST matrix. The mapping:

Deferred item (round-3 list)	lance-c#55 extension
Fragment-scoped distributed builds	E1 (fragment-scoped uncommitted build)
Shared IVF centroid/PQ-codebook distribution	E1 (training entries + model injection)
BE→FE uncommitted segment metadata transfer	E2 (`pb::IndexMetadata` bytes; cross-SDK consumption requires an explicit Java decoder/adapter from bytes to `Index.Builder` per-field setters, plus a version-compatibility contract — the pinned SDK has no `parseFrom(byte[])`; per reviewer note, issuecomment-5212293290)
Coordinator commit of worker-built segments	E5 / existing Java SDK path
Physical segment merge	E5 open question
Native progress / cooperative cancellation	E3 / E4
Per-segment coverage/statistics	E6 (also restores the TVF)
Incremental `BUILD INDEX`	Composition of E1+E2+commit

Correction to my earlier cross-link comment (issuecomment-5212075800), per round-3 Q3: nothing in lance-c#55 gates this issue — the RFC continues as the post-4.2 distributed track, and its wave split was designed for exactly this decoupling. When the deferred track resumes, v3's provenance design (pre-assigned UUID sets + transaction-properties commits) is reinstated on top of E1/E2, closing the §6 gap.

10. Work items (4.2; branch-4.1; deliverable as separate PRs)

FE neutral command model + property validation (metric key; rejection of REST-inexpressible properties) + explicit BUILD INDEX rejection for Lance;
Durable job framework + SHOW/CANCEL integration (§5 states, admission, privileges);
BE one-shot execution task for Directory (lance-c v0.1.2 calls, selected-BE dispatch, credential handling at dispatch);
REST Namespace path per §4 matrix (create/drop/poll, transaction_id persistence);
Metadata surfaces (§7: SHOW INDEX grouping, physical type, bounded properties);
Failure recovery + refresh per §6;
Tests per §8 + documentation (syntax, lifecycle semantics, limitations: best-effort CANCEL, no native progress, OUTCOME_UNKNOWN boundaries, single-node whole-dataset build).

Round-3 answers adopted

Q1: branch-4.1 only. ✔
Q2: option (c) — metric with l2|cosine|dot, matching vector_search(); no internal ANN query/property syntax compatibility introduced. ✔
Q3: lance-c extensions are not prerequisites; tracked as future work (§9). ✔

One clarification on scope wording: v3 had gated the production path on extended lance-c because its target was the distributed build form; with 4.2's production form redefined as the one-shot selected-BE execution, no gating remains. The §6 provenance weakening is the only deliberate technical downgrade, and it is stated rather than hidden.

branch-4.1: [feature](lance) Add Lance index admission, job inspectio…

[fix](lance) Settle job reports atomically and refresh fairly

After reconsidering the implementation against the execution model of Iceberg rewrite_data_files, please simplify the design and implementation for this issue. This changes the previously accepted v5.1 direction: a durable asynchronous Lance index job framework should no longer be a requirement for the initial implementation.

1. Use synchronous SQL with bounded worker execution

The SQL statement should validate the request, dispatch resource-isolated work to BE, wait for completion, refresh metadata after a confirmed commit, and return the result. Internal parallel execution does not require an asynchronous SQL/job interface.

Keep heavy Lance work outside FE and the BE main process, with hard worker resource limits, bounded concurrency/queueing, deadlines, and reliable child cleanup. These protections are independent of durable job management and should remain.

2. Revert the job infrastructure already introduced

Please prepare an explicit cleanup/revert on branch-4.1, rather than leaving the old framework disabled or maintaining two execution paths:

Revert the durable Lance index job state machine, manager, persistent same-name fences, unresolved-job quotas, and job replay infrastructure introduced by branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235.
Remove the job-specific parts of branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630: durable job creation and JobId results, SHOW LANCE INDEX JOB(S), and catalog guards tied to unresolved jobs. Preserve and adapt the useful privilege, schema, parameter, snapshot, and IF-condition validation for synchronous execution.
Remove the associated unused configuration, serialization/RPC fields, tests, and documentation as appropriate. Review journal/image compatibility before removing persisted readers or operation codes; do not make existing metadata unreadable or reuse old codes. Any necessary compatibility handling should be minimal and should not preserve an active job framework.
Stop pursuing branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978 and branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC #67754 in their current form. Rework branch-4.1: [feature](lance) Add the cgroup-isolated one-shot worker for Lance index mutations #68668/branch-4.1: [test](lance) Prove the IVF_PQ tracer and worker-fault convergence through the isolated worker #68669 around the synchronous worker lifecycle and its failure semantics, retaining useful isolation and query-consumption tests.

Keep the read-only SHOW INDEX and physical-index inspection functionality, and the reusable target-aware DDL validation. Please update the issue design and PR roadmap to reflect this scope change.

3. State the simpler failure contract explicitly

The initial version does not promise persistent job history, FE failover recovery, resumable builds, or durable same-name serialization across failures.

A timeout, disconnect, worker loss, or lost response after dispatch must not be presented as proof that nothing committed. Return an explicit indeterminate-outcome error when applicable, do not automatically retry a potentially committed mutation, and let the user inspect authoritative index metadata. Observing a same-name index does not prove which request created it.

Cancellation cannot undo an existing commit. Worker capacity must remain accounted for until termination is confirmed. A confirmed commit followed by metadata-refresh failure must be reported distinctly from a failed build.

4. Keep a path to distributed construction

Synchronous SQL does not preclude distributed index construction. The desired extension is:

pin snapshot -> partition fragments -> parallel BE builds of uncommitted segments -> validate/collect outputs -> one coordinated final commit -> refresh -> return

Please keep build and commit responsibilities separable. Verify the APIs available in the pinned Lance bindings and add the necessary adapter support before implementing this distributed path; do not run several complete, independently committing create_index calls against the same logical index name.

The initial implementation may use one worker. Multi-BE construction can follow without introducing durable jobs; persistent recovery should be a separate, justified follow-up requirement rather than a prerequisite for index lifecycle support.

After reconsidering the implementation against the execution model of Iceberg rewrite_data_files, please simplify the design and implementation for this issue. This changes the previously accepted v5.1 direction: a durable asynchronous Lance index job framework should no longer be a requirement for the initial implementation.

1. Use synchronous SQL with bounded worker execution

The SQL statement should validate the request, dispatch resource-isolated work to BE, wait for completion, refresh metadata after a confirmed commit, and return the result. Internal parallel execution does not require an asynchronous SQL/job interface.

Keep heavy Lance work outside FE and the BE main process, with hard worker resource limits, bounded concurrency/queueing, deadlines, and reliable child cleanup. These protections are independent of durable job management and should remain.

2. Revert the job infrastructure already introduced

Please prepare an explicit cleanup/revert on branch-4.1, rather than leaving the old framework disabled or maintaining two execution paths:

Revert the durable Lance index job state machine, manager, persistent same-name fences, unresolved-job quotas, and job replay infrastructure introduced by [branch-4.1: feature Durable Lance index job infrastructure with fence, quota and replay branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235](branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235).

Remove the job-specific parts of [branch-4.1: feature Add Lance index admission, job inspection SQL and catalog DDL guard branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630](branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630): durable job creation and JobId results, SHOW LANCE INDEX JOB(S), and catalog guards tied to unresolved jobs. Preserve and adapt the useful privilege, schema, parameter, snapshot, and IF-condition validation for synchronous execution.

Remove the associated unused configuration, serialization/RPC fields, tests, and documentation as appropriate. Review journal/image compatibility before removing persisted readers or operation codes; do not make existing metadata unreadable or reuse old codes. Any necessary compatibility handling should be minimal and should not preserve an active job framework.

Stop pursuing [branch-4.1: feature Add Lance index job dispatcher with thrift dispatch boundary branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978](branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary #67978) and [branch-4.1: feature Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC #67754](branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC #67754) in their current form. Rework [branch-4.1: feature Add the cgroup-isolated one-shot worker for Lance index mutations branch-4.1: [feature](lance) Add the cgroup-isolated one-shot worker for Lance index mutations #68668](https://github.com/apache/doris/pull/68668)/[branch-4.1: test Prove the IVF_PQ tracer and worker-fault convergence through the isolated worker branch-4.1: [test](lance) Prove the IVF_PQ tracer and worker-fault convergence through the isolated worker #68669](branch-4.1: [test](lance) Prove the IVF_PQ tracer and worker-fault convergence through the isolated worker #68669) around the synchronous worker lifecycle and its failure semantics, retaining useful isolation and query-consumption tests.

Keep the read-only SHOW INDEX and physical-index inspection functionality, and the reusable target-aware DDL validation. Please update the issue design and PR roadmap to reflect this scope change.

3. State the simpler failure contract explicitly

The initial version does not promise persistent job history, FE failover recovery, resumable builds, or durable same-name serialization across failures.

A timeout, disconnect, worker loss, or lost response after dispatch must not be presented as proof that nothing committed. Return an explicit indeterminate-outcome error when applicable, do not automatically retry a potentially committed mutation, and let the user inspect authoritative index metadata. Observing a same-name index does not prove which request created it.

Cancellation cannot undo an existing commit. Worker capacity must remain accounted for until termination is confirmed. A confirmed commit followed by metadata-refresh failure must be reported distinctly from a failed build.

4. Keep a path to distributed construction

Synchronous SQL does not preclude distributed index construction. The desired extension is:

pin snapshot -> partition fragments -> parallel BE builds of uncommitted segments -> validate/collect outputs -> one coordinated final commit -> refresh -> return

Please keep build and commit responsibilities separable. Verify the APIs available in the pinned Lance bindings and add the necessary adapter support before implementing this distributed path; do not run several complete, independently committing create_index calls against the same logical index name.

The initial implementation may use one worker. Multi-BE construction can follow without introducing durable jobs; persistent recovery should be a separate, justified follow-up requirement rather than a prerequisite for index lifecycle support.

Agreed — accepted, I'll rework this to the synchronous model.

The synchronous shape was my starting point when drafting the original design, simply because it is easier to implement and verify. The durable async framework came in only to keep post-dispatch ambiguity safe across failover, and its complexity — fences, quotas, replay, force-release — has outgrown that value. The synchronous path is also a much smaller diff to review and leaves fewer corners for hard-to-debug failure modes.

Concretely:

revert the branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235 job infrastructure and the job-specific parts of branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630 on branch-4.1, keeping the privilege/schema/parameter/snapshot/IF validation, SHOW INDEX, and the inspection TVF; journal/image compatibility reviewed before any persisted reader or op code is removed
close the open dispatcher and force-release PRs, and rework the worker PR around the synchronous lifecycle and its failure semantics
update the design and roadmap with the explicit failure contract, and keep build and commit responsibilities separable for the later distributed path

[refactor](lance) Remove the durable Lance index job framework (apach…

branch-4.1: [refactor](lance) Remove the durable Lance index job fram…

This supersedes the v5.1 proposal following the synchronous-direction review. The baseline facts — the commit-resolved pinned revisions, panic behavior at the FFI boundary, the binding entry points, and the Rust-to-C error mapping — were re-verified against the pinned sources, and every execution parameter the direction comment left open was resolved to the reading most consistent with it. Section 11 records that evidence and the single remaining decision (client-disconnect semantics), and asks for confirmation of the 4.2 boundary.

Design Proposal v6 — Synchronous Lance Index Lifecycle for Doris 4.2

Prepared for publication: 2026-10-08 (Asia/Shanghai); revised 2026-10-09

Target: Apache Doris branch-4.1 / Doris 4.2

Issue: apache/doris#66497

Status: v6 publication candidate (revised 2026-10-09 after local verification of the pinned ABI facts and the failure-contract review); supersedes v5.1 following the synchronous-direction comment (2026-10-08)

Provenance: v5/v5.1 remain issue-history provenance only; they provide no implementation guidance here

0. Authority and revision summary

This standalone document defines the capability and safety contract for Lance index lifecycle management in Doris 4.2 under a synchronous execution model. If accepted, it supersedes the v5.1 minimal one-shot proposal.

Context for this revision

On 2026-10-08, after reconsidering the implementation against the execution model of Iceberg rewrite_data_files, the reviewer changed the previously accepted v5.1 direction: a durable asynchronous Lance index job framework is no longer a requirement for the initial implementation. The SQL statement itself validates the request, dispatches resource-isolated work to a BE, waits for completion, refreshes metadata after a confirmed commit, and returns the result. Internal parallel execution does not require an asynchronous SQL/job interface.

The job infrastructure already introduced on branch-4.1 — the durable job state machine, manager, persistent same-name fences, unresolved-job quotas, and job replay of #67235, plus the job-specific parts of #67630 (durable job creation and JobId results, SHOW LANCE INDEX JOB(S), catalog guards tied to unresolved jobs) — is reverted explicitly rather than left disabled or maintained as a second execution path. The reusable privilege, schema, parameter, snapshot, and IF-condition validation, the read-only SHOW INDEX, and the physical-index inspection are preserved and adapted for synchronous execution. The worker resource protections — heavy Lance work outside FE and the BE main process, hard worker resource limits, bounded concurrency/queueing, deadlines, and reliable child cleanup — are independent of durable job management and remain requirements.

Relative to v5.1, this revision retains: the SQL surface and type/property matrix, the authoritative one-snapshot metadata contract, admission-time authoritative preflight and IF behavior, dataset-version and schema-contract revalidation before the mutating native invocation, hard worker process isolation, typed provider-result classification, and secret handling. It removes: durable job records, the compact PENDING -> RUNNING -> ... lifecycle, same-name fences, unresolved-job quotas, FORCE_RELEASE, job inspection SQL, and replay/failover recovery. The UNKNOWN concept survives only as a client-visible indeterminate-outcome error class, never as a persisted state.

Baseline

The baseline is resolved at commit level, because package versions and release tags alone do not determine API or error semantics: Doris branch-4.1 at c8c0ddca4d; FE Java binding lance-core 12.0.0; BE C binding lance-c at commit 98468344 (its package version string still reads 0.1.9), whose Cargo manifest pins the Rust core at 68c12dfd. The BE binding is identified by the exact source URL recorded in thirdparty/vars.sh at the stated Doris commit — not by its version string or tag. Any future statement about "what the pinned bindings do" resolves the vars.sh-to-commit-to-manifest chain before asserting a fact.

No 4.2 guarantee depends on upgrading or extending those APIs; Section 8 records which extension capabilities the pinned bindings do and do not expose.

Normative terms use MUST, MUST NOT, SHOULD, and MAY in their ordinary requirements sense. Sections explicitly marked future work are non-normative for 4.2.

Review focus: Section 11 records the verified baseline evidence and the single open decision (client-disconnect semantics), and asks for confirmation of the 4.2 boundary.

1. Scope and release boundary

1.1 Core safety contract

Doris 4.2 executes one Lance mutation inside a single synchronous SQL statement without retrying after dispatch may have occurred, fabricating an outcome, or implying that nothing committed when the outcome is uncertain. Mutation outcome and metadata refresh are independent dimensions. A mutation statement returns exactly one of four outcome classes:

success — including an authoritative-preflight no-op, which returns only after its required bounded cache invalidation completes (Section 6.4); a committed success is followed by the applicable refresh obligations;
confirmed failure — a complete trusted result proves this invocation did not commit;
indeterminate outcome — an explicit typed error stating the mutation may or may not have committed, directing the user to authoritative index metadata;
committed, refresh incomplete — a complete trusted success was received and the required refresh did not complete (Section 6.4); the build outcome is known and is never reported as a build failure.

A client that cannot receive a result packet — a timeout with a lost response, a disconnect, or FE loss — treats the outcome as indeterminate. This is a documentation and driver rule (Sections 5.5 and 6.2), because no SQL error can be delivered on a closed connection.

1.2 Required in 4.2

The 4.2 deliverable includes:

Directory Catalog CREATE INDEX, CREATE OR REPLACE INDEX (full same-name rebuild), and DROP INDEX by logical name, for the initially verified vector and scalar index types, executed synchronously within the statement;
SHOW INDEX for logical index metadata and the lance_index_entries TVF for physical index UUID and dataset version (both unchanged from the merged read-only work);
basic IF NOT EXISTS and IF EXISTS behavior based on authoritative Lance metadata;
table-level SHOW and ALTER authorization;
admission validation preserved from the merged work: privilege, target, schema, parameter, snapshot, and IF-condition checks before any dispatch, with the final authoritative preflight inside the master's name entry (Section 4.3);
one resource-isolated worker process on one selected BE through a verified hard boundary: memory limit, PID/thread limit, wall-clock runtime limit, reliable termination detection and child reaping across the whole isolated execution domain;
bounded concurrency and bounded queueing for mutation statements, and one total time budget per statement covering admission reads, queueing, dispatch, execution, and refresh (Section 5.5);
worker capacity accounting held in memory by the BE supervisor until termination of the isolated execution domain is confirmed (Section 5.4);
typed result classification per Section 6, including the distinct indeterminate-outcome and committed, refresh incomplete error classes;
the refresh obligations of Section 6.4, with refresh failure reported distinctly from build failure;
focused crash, OOM, panic, privilege, fault-injection, and end-to-end tests;
user documentation for syntax, synchronous semantics, the four outcome classes, limitations, and inspection.

Lance manifests remain the authoritative index metadata. Doris persists no mutation control state: no job record, no fence, no quota book, no replay.

1.3 Explicitly deferred from 4.2

These exclusions are normative and cannot gate 4.2:

persistent job history, FE failover recovery for in-flight mutations, resumable builds, and durable same-name serialization across failures; persistent recovery, if ever needed, is a separate issue that must justify itself rather than a prerequisite for index lifecycle support;
job inspection and release SQL and every persisted-UNKNOWN operational protocol — SHOW LANCE INDEX JOB(S), RESOLVE ... FORCE_RELEASE, audited release notes, and late-commit warnings attached to records;
automatic UNKNOWN watchers or reconciliation daemons, idempotency protocols, automatic retry of dispatched mutations, incremental BUILD INDEX, coverage repair, build progress, and continuously appended-table freshness SLAs;
running-cancellation rollback: cancellation is best-effort termination only and never undoes a commit (Section 6.3);
distributed index construction as a 4.2 capability (Section 8 states the extension path; the initial version uses one worker);
REST inspection and mutation surfaces;
creating additional index types, composite indexes, or nullable indexed fields, and richer physical metadata (dropping existing composite, nested, or legacy indexes by name remains in scope);
provider-backed table/dataset incarnation registries, external same-URI replacement detection, normalization migration, and mixed-version fence protocols;
a duplicate logical TVF, server-side exact logical-index counts, and countRows() surfaces;
broad catalog-wide external-DDL transaction machinery beyond the in-memory guards of Section 4.

2. User-visible contract

2.1 SQL surface

CREATE INDEX [IF NOT EXISTS] idx
ON lance_ctl.db.tbl (embedding)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    "metric" = "l2",
    "num_partitions" = "256",
    "num_sub_vectors" = "16"
);

CREATE INDEX idx_btree
ON lance_ctl.db.tbl (event_time)
USING BTREE;

CREATE INDEX idx_bitmap
ON lance_ctl.db.tbl (category)
USING BITMAP;

CREATE OR REPLACE INDEX idx
ON lance_ctl.db.tbl (embedding)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    "metric" = "cosine",
    "num_partitions" = "256",
    "num_sub_vectors" = "16"
);

SHOW INDEX FROM lance_ctl.db.tbl;

SELECT *
FROM lance_index_entries("table" = "lance_ctl.db.tbl")
WHERE IndexName = "idx";

DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;

USING ANN is a neutral DDL category. It does not add or imply Doris internal ANN query syntax for Lance. Lance vector queries continue to use vector_search().

CREATE OR REPLACE maps to replace=true and fully rebuilds the logical name. It converges by name but is not idempotent: another invocation may produce a different UUID and dataset version. It is mutually exclusive with IF NOT EXISTS; USING, target column, and required properties are validated before dispatch.

Only top-level CREATE, REPLACE, and DROP mutate Lance indexes in 4.2; ALTER TABLE ... ADD/DROP INDEX and BUILD INDEX remain unsupported. Internal tables are unchanged.

2.2 Synchronous execution semantics

All Lance index mutations are forwarded to and executed on the active master FE. An admitted mutation executes within the statement and returns when one of the Section 1.1 outcome classes is known:

static validation (any FE)
  -> forward to master; forwarding preserves single-send (Section 4.3)
  -> acquire the in-memory active-name entry; same-name busy fails fast
  -> final authoritative IF preflight, inside the entry (Sections 2.3, 4)
  -> bounded admission: concurrency limits, finite queue, total budget check (Section 5.5)
  -> select one BE; the BE supervisor reserves worker capacity (Section 5.4)
  -> single dispatch request carrying invocation identity, remaining budget, and its verifiable expiration
  -> BE supervisor launches the resource-isolated worker
  -> one pinned lance-c mutating invocation against the admitted dataset version
  -> complete typed result classified (Section 6.5)
  -> refresh obligations (Section 6.4)
  -> release worker capacity and the name entry on confirmed termination of the execution domain
  -> return to the client

The statement returns no job id and exposes no job surface. A client that loses the connection, times out, or kills the statement follows Sections 5.5 and 6.2 and MUST NOT blindly resubmit.

2.3 Submission and `IF` behavior

Admission reads authoritative metadata before any dispatch:

plain CREATE fails if the normalized logical name already exists — existence is determined from the physical list together with available logical descriptions, so a legacy index omitted from logical descriptions still occupies its name (Section 3.1);
CREATE IF NOT EXISTS returns an immediate no-op only when the existing authoritative normalized name, indexed columns, physical family, and algorithm match. A property is compared only when the same bounded logical metadata already used by SHOW INDEX exposes a stable value; the exposed mappings are metric against metric_type, num_sub_vectors against compression.num_sub_vectors, and the effective num_bits=8 against compression.num_bits. num_partitions is not reliably exposed and is therefore not compared, and no substitute value (for example target_partition_size) may stand in for it. No additional canonicalization or metadata read is a prerequisite;
CREATE IF NOT EXISTS fails on an authoritative mismatch rather than treating a different same-name definition as success;
DROP fails if the authoritative name is absent;
DROP IF EXISTS returns an immediate no-op when the authoritative name is absent;
REPLACE requires an existing or creatable unambiguous target name and always represents a new one-shot mutation when admitted.

Two ordering rules close the preflight races. First, the final authoritative preflight runs after the active-name entry is acquired on the master, immediately before dispatch admission: a preflight taken earlier — before forwarding, before queueing, or before the entry — is not sufficient to dispatch, so a request that queued behind another same-name statement cannot proceed on a stale snapshot. Second, an authoritative no-op completes its required bounded cache invalidation before returning (Section 6.4), so a stale cache cannot contradict the returned state; the invalidation closure and its residual failure class are defined there.

Authoritative preflight is not an external lock. A post-preflight race against an external writer is classified from the result rules in Section 6.5; Doris never turns a generic post-invocation error into an IF no-op by inspecting message text.

2.4 Supported type and property matrix

Type	Indexed-column contract	User properties and checks
`IVF_PQ`	Exactly one non-null fixed-size-list field with `FLOAT16` or `FLOAT32` elements	`index_type=IVF_PQ`; `metric` is `l2`, `cosine`, or `dot`; positive required `num_partitions` and `num_sub_vectors`; subvectors divide the vector dimension; configured static bounds apply; `num_bits=8` is fixed
`BTREE`	Exactly one non-null scalar field from the pinned predicate-pushdown set: integral, floating, decimal, string, date, or timestamp	No user build properties; C `params_json=NULL`
`BITMAP`	Exactly one non-null boolean, integral, string, or date field	No user build properties; C `params_json=NULL`

For CREATE and REPLACE, Hamming distance, nullable/unsupported nested fields, FLOAT64, UINT8, or INT8 vector elements, composite indexes, arbitrary scalar JSON, and unknown properties or values fail before dispatch — these admission limits never restrict what can be dropped by name. FE enforces static syntax, type, and bounds; the worker repeats the independently checkable facts against the admitted snapshot before the mutating native invocation.

Vector element nullability is not a schema-level admission check: the manifest has no slot for element nullability, and the pinned Java SDK reconstructs every fixed-size-list element as nullable regardless of what was written, so element nullability is unobservable at the schema level. The two validation layers are kept separate: facts independently checkable before the call — the element type and the non-null top-level field — are checked before the mutating native invocation; validation of actual element values happens inside the provider's build, within the worker's budget, and its failures classify under the post-invocation rules of Section 6.5, never as provable pre-invocation failures.

3. Authoritative metadata contract

3.1 One pinned Directory snapshot

Every SHOW, physical inspection, IF check, and post-commit refresh uses one opened Directory Dataset snapshot. One snapshot does not mean one native call: the read chain issues several calls against the same opened dataset and keeps one overall deadline for the chain.

raw index names come from the index listing; reserved system-entry names (__lance_frag_reuse, __lance_mem_wal) are filtered and the raw list is deduplicated by exact name;
logical descriptions are fetched per name with an exact-name criteria — a criteria-free bulk description is not used, because converting the details of legitimate system entries can fail the whole call before Doris filters anything. A per-name lookup returns exactly one non-null description carrying the requested name, or — only in the legacy case below, where every physical segment of that index lacks readable details — an empty result that leaves the name to the physical list; a null, duplicate, or wrong-name result fails closed;
the same snapshot schema resolves indexed field IDs;
the physical listing supplies physical UUIDs and dataset versions.

The chain bounds raw entries and logical names separately, both finite. Name occupancy for mutation is determined from the physical list together with the available logical descriptions. A legacy index whose segments all lack readable details may be omitted from logical descriptions while its name still appears in the physical list: plain CREATE rejects such a name as occupied, CREATE IF NOT EXISTS does not claim a match without a reliable definition, and DROP IF EXISTS does not claim absence from an incomplete logical list alone. Missing legacy details are not malformed metadata; malformed or oversized data still fails closed with a typed bounded error, and the two cases are never conflated. Whether SHOW continues to omit such legacy logical descriptions is a fixed, documented behavior, not an accident of the read path.

Normal inspection and FE admission MUST NOT call Dataset.countRows(). Dataset.getIndexStatistics() is also forbidden because the pinned version may migrate metadata and write a manifest. Provider-local build validation stays in the isolated worker and creates no Doris row-count surface.

FE metadata reads have bounded results and deadlines through the existing bounded inspection executor (LanceIndexInspectionExecutor, 4 concurrent / 16 queued on the baseline), which remains the read-side owner: admission reads go through it, and a statement exiting does not reclaim capacity of a read still inside JNI. Timeout does not imply JNI cancellation; ownership remains until the native call returns. A separate read helper process is not a 4.2 requirement.

3.2 Logical `SHOW INDEX`

SHOW INDEX keeps the existing MySQL-compatible 13-column schema and returns one row per logical index/column. For Lance:

Key_name is the exact display name;
Column_name comes from the same-snapshot field-ID resolution;
Index_type is the logical Lance algorithm such as IVF_PQ, BTREE, or BITMAP, not the SQL category ANN;
Properties is deterministic, valid, allowlisted JSON under a reviewed finite UTF-8 bound.

There is no second logical TVF or server-side exact-count API; callers may count distinct Key_name values in the bounded SHOW result.

3.3 Minimal physical inspection

lance_index_entries("table"="ctl.db.tbl") returns one row per physical entry:

CatalogName, DatabaseName, TableName, IndexName, IndexUuid, DatasetVersion.

The TVF exists only because UUID and dataset version do not fit the established SHOW schema. It exposes no fragments, coverage, provider/consistency state, opaque details, or transaction properties.

The TVF requires one table argument, table SHOW, and an optional ordinary predicate over the bounded result. It is all-or-error and never silently truncated.

3.4 Pinned type vocabulary, system entries, and bounds

The pinned Java SDK has different logical and physical vocabularies:

per-name logical descriptions derive a concrete vector algorithm such as IVF_PQ, while the physical listing may expose the umbrella manifest type VECTOR;
scalar values differ in case and separators, for example BTREE versus BTree and LABEL_LIST versus LabelList.

Authoritative IF and internal joins use one normalization rule:

normalize case and ignore underscores for family comparison;
accept physical VECTOR only for supported vector algorithms;
accept physical SCALAR only for supported scalar algorithms;
never match an internal Doris index family to a Lance family.

Logical SHOW displays the concrete per-name description value. The minimal physical TVF does not expose the umbrella type.

Duplicate logical names or UUID ownership, unknown field IDs, malformed or oversized data, and required logical/physical mismatches fail closed with a typed bounded error.

Directory inspection caps logical names, physical entries, external strings, fields per index, and aggregate field-name bytes. Exact values follow Doris configuration conventions. Because these post-JNI checks do not bound native allocation, focused FE stress/failure tests cover accepted and over-limit data.

4. Target identity, admission, and Doris-side guards

4.1 Stable target and logical name

The target key used for validation, DDL guards, and in-memory concurrency is:

(
  persisted catalog identity,
  provider = DIRECTORY,
  normalized stable dataset locator,
  normalized logical-index-name bytes
)

The catalog identity is persisted configuration. The request's display name and normalized name bytes live only in bounded invocation memory for the duration of the statement and its cleanup — this contract adds no mutation persistence. Normalization v1 is the UTF-8 result of Java toLowerCase(Locale.ROOT). Doris preserves display case, rejects new case-only duplicates, resolves a unique match to its stored name, and fails mutation on ambiguous external case-only collisions.

There is no durable fence keyed on this tuple. Credential-bearing URLs are never identity; URI aliases are external writers. Replacing the dataset at the same URI concurrently with a mutation is unsupported and lands in the indeterminate class of Section 6.

4.2 Dataset-version and schema-contract revalidation

CREATE and REPLACE admission records the exact selected/latest dataset version and the following ordered contract:

schema_contract_version = 1

indexed_field = {
  field_id,
  normalized_name,
  normalized_type,
  nullable,
  fixed_size_list_dimension,     // when applicable
  vector_element_type,           // when applicable
  vector_element_nullable        // when applicable
}

normalized_type includes relevant parameters such as decimal precision/scale and timestamp unit/time-zone semantics. Unindexed fields are excluded.

Before the mutating native invocation, the worker reopens the admitted dataset version, independently recomputes contract v1, and compares the ordered representation. An unavailable version, mismatch, or unsupported contract produces a complete pre-invocation confirmed failure with typed STALE_ADMISSION or UNSUPPORTED_SCHEMA_CONTRACT; the worker never switches to latest. This revalidation confirms the admission snapshot itself; it does not observe whether storage-latest has since advanced, and any post-admission race is arbitrated by the provider's commit machinery and classified under Sections 4.3 and 6.5. No additional worker-start latest-equality gate is added: this design does not require storage-latest to still equal the admitted version when the worker starts, and such a gate would turn concurrent appends that the provider can safely arbitrate into early rejections — Section 4.3 promises no such failure — while still not guaranteeing equality at final commit. The worker may still read storage-latest for result diagnostics; such reads never gate the invocation.

Java admission and Rust execution share golden contract fixtures.

A DROP target is revalidated against the pinned admission snapshot: the worker reopens the admitted dataset version and resolves the normalized name uniquely to a stored name in that snapshot's bounded physical list, then issues one drop by that name. This confirms the admission snapshot only — it does not prove that storage-latest has not advanced since admission, and a concurrent external change is arbitrated by the provider's commit machinery and classified under Sections 4.3 and 6.5. No field-schema contract is built for DROP — CREATE's single-column and algorithm restrictions never limit what can be dropped, and composite, nested, or legacy indexes whose names appear in the physical list are droppable by logical name. A same-URI dataset replacement that reproduces the same version cannot be detected by this revalidation; no indeterminate-detection promise is made for it.

4.3 Same-name concurrency and admission serialization

With fences and quotas removed, same-name concurrency is handled by master-local admission plus provider-side arbitration, and nothing else:

all mutations execute on the active master FE. Statements arriving on a follower are forwarded, consistent with existing DDL forwarding conventions. Single-send must hold across every hop — follower-to-master forwarding, master-to-BE dispatch, and thrift reconnect or timeout paths — so that no path can produce a second execute request for one statement;
the master holds an in-memory active-name entry per target logical name. A same-name request arriving while an entry is active fails fast with a typed busy error; it does not queue behind the running statement and does not block a worker thread waiting for a lock;
the final authoritative preflight runs after the entry is acquired (Section 2.3), so a racing or queued request cannot dispatch against a stale preflight snapshot;
the entry is retained until termination of the isolated execution domain is confirmed — it outlives statement return, statement timeout, and client disconnect, because the worker may still be running. A later same-name request may therefore see busy until cleanup completes; releasing the client connection never releases the entry by itself. When the termination proof cannot be delivered or re-acquired (Section 5.3), busy persists until the coordinator epoch is rebuilt; the entry is bounded by the finite epoch cleanup total (Section 5.4) and is never released by a TTL or heartbeat guess;
Lance-side arbitration is a separate mechanism, not a classification promise. The provider's manifest arbitration may resolve concurrent commits internally (including rebase against concurrent appends). Doris does not rely on receiving a typed conflict: distinct provider conflict variants share the aggregate internal error code in the pinned C ABI, so a lost same-name race may surface as a generic internal error and classifies as indeterminate under Section 6.5. Pinning the build snapshot does not promise that the committed version equals the admission version plus one, and does not promise failure on concurrent append;
no entry survives an FE restart. Doris provides no durable same-name serialization across failover or against external writers; this window is an explicitly accepted exposure, and observing a same-name index never attributes creation (Section 6.2).

4.4 Credentials

No component persists credentials: dispatch, authoritative reads, and refresh resolve current catalog credentials when executed.

A credential-only catalog ALTER is not target-changing and MUST remain allowed; provider, stable-locator, and selected-version changes remain guarded (Section 4.5). Known-expired credentials fail before the mutating invocation as a confirmed failure. No credential-TTL or safety-margin protocol is required. Section 7 defines secret handling.

4.5 Minimal DDL guard boundary

While a mutation executes, an in-memory active-target token on the master guards target-changing DDL: a Doris DDL that changes the persisted catalog identity, Directory provider, normalized locator, or selected-version semantics fails fast with a busy error. Credential-only ALTER, ordinary cache refresh, unrelated properties, and external appends remain allowed. No broad catalog lock is held across execution.

Three distinct owners are involved and never conflated: the per-name admission entry (Section 4.3), the short FE metadata/DDL locks held only during bounded validation and admission (Section 5.1), and the active-target token (this section). FE restart loses the token; that window is accepted and is not serialized across failures.

Target-aware routing precedes internal catalog.Index translation. Internal validation/serialization is unchanged, and Lance algorithms do not extend the persisted internal IndexType enum.

5. Synchronous execution and resource boundary

5.1 Execution flow

The execution pipeline is the one stated in Section 2.2; this section fixes its locking and resource boundaries. FE metadata/DDL locks cover only bounded validation and admission — not dispatch, native execution, waiting, metadata reads, or refresh.

5.2 Hard process isolation

The pinned C binding catches Rust panics at the FFI boundary and reports them as a typed panic error, and its build configuration rejects unwinding-incompatible abort settings at compile time. Nevertheless OOM, stack overflow, and double panics can still terminate a process, and the native runtime creates threads outside Doris MemTracker control. Production mutation therefore never runs inside FE, the BE main process, or an ordinary RPC handler.

Before Lance/Rust initializes materially, the selected BE launches a same-build worker in a verified OS-enforced boundary providing:

a hard memory limit;
a hard PID/thread limit;
a wall-clock runtime limit;
reliable termination detection and child reaping across the whole isolated execution domain.

A deployment unable to verify the boundary rejects before the mutating invocation; there is no in-process or soft-RSS fallback. Launcher, containment, environment, diagnostic, and handshake details are not public contracts.

The BE supervisor and its queues are bounded. Capacity reservation and BE selection are strictly non-invoking: they do not send the execute request, launch a worker, initialize Lance/Rust, or make native calls. Once the single execute request is sent, a complete trusted identity-matched busy or other pre-invocation rejection is a terminal confirmed failure and is never automatically retried within the same statement.

Local/file:// mutation has a separate disabled-by-default operator assertion, requires exactly one FE and BE, normalized absolute paths, pre/post version and identity checks, and rejection on topology change before dispatch. These checks cannot prove a shared mount namespace; object storage is the production mode.

5.3 Invocation identity, result envelope, and termination evidence

FE records an immutable invocation identity (invocation id, selected BE process epoch, budget) before sending the single execute request, and every response path is matched against it;
the worker/supervisor result envelope contains only what is needed to classify the one invocation: the matching invocation identity and BE epoch, complete trusted pre-invocation-failure and native-return markers, the saved LanceErrorCode read before the consuming error message, a bounded sanitized message, and matching domain-termination proof when available;
only a complete identity-matched result proves confirmed success or confirmed failure; a missing marker never proves the mutating native call was not entered. EOF, signal, timeout, OOM, BE loss, malformed/partial protocol, or identity mismatch after dispatch yields the indeterminate class;
the adapter validates deterministic user errors before the mutating native invocation and classifies only saved typed codes, never message text. It reads the code before the consuming message, checks it before treating a zero count as empty, and releases returned strings with lance_free_string;
termination proof covers the whole isolated execution domain, not only the direct child: capacity is not released while surviving descendants may exist. A supervisor-level check that the containment holds no remaining processes may serve as the domain proof;
a complete result and a termination proof are two different kinds of evidence. A received, identity-matched result finally classifies the mutation and is never rewritten by a later termination proof. A termination proof arriving after a lost result only releases resources; it never converts an indeterminate outcome into an attributed one;
termination proof is handled separately from the mutation result. When a result or termination report is lost in transit, the current invocation's in-memory cleanup channel may re-acquire or confirm the proof — a bounded acknowledgment or a read-only status query are implementation details. The cleanup channel never re-sends the execute request, and a proof acquired this way never rewrites an indeterminate outcome;
when no proof can be obtained, admission stays fail closed. A same-name request may observe busy until the coordinator epoch is rebuilt, not merely until the worker happens to end. Paths that provably never dispatched, or that carry complete never-launched evidence, have independent release conditions and do not wait on a child that does not exist.

5.4 Capacity ownership and queueing

actual worker capacity is owned by the BE supervisor. A slot is reserved at dispatch and released only when termination of the whole isolated execution domain is confirmed (Section 5.3); the direction comment states this requirement outright — worker capacity must remain accounted for until termination is confirmed. The BE does not release capacity because the FE disappeared or a client connection ended. A BE restart may only assume old executors are gone when they actually terminated under the old containment, or when OS protections guarantee they die with the old supervisor; heartbeat timeout or address re-registration alone does not prove the old process epoch is dead;
FE-side limits are admission limits for the current coordinator epoch. The master enforces configurable concurrent-statement limits (cluster-wide and per catalog) in memory, with one finite admission queue; same-name busy or a full queue returns a typed busy error instead of an unbounded wait. Retained name entries, active-target tokens, and cleanup contexts that outlive their statement count against a finite current-epoch cleanup total — statement return never bypasses that accounting, so repeated statements against different names cannot accumulate unbounded historical entries; the sum of BE capacities does not bound these FE-side entries. These limits do not pretend to be a cross-restart hard worker cap, and ending a statement or connection does not release capacity belonging to a still-running worker;
the cluster-wide hard bound on simultaneously live workers is the sum of the finite per-BE supervisor capacities, independent of FE session lifecycle. Topology growth or configuration changes do not turn a smaller FE-side global parameter into that sum. A precise per-catalog hard quota that survives failover would need a separate capacity-allocation or BE-side aggregation mechanism and is out of scope;
queueing is two bounded tiers: the finite master admission queue in front of dispatch, plus a bounded receive/worker queue on each BE as a defense line. A dequeued request revalidates the target, the preflight, and its remaining budget before dispatch; time spent in either queue counts into the statement budget (Section 5.5). No cross-BE work stealing in 4.2.

5.5 Time budget, cancellation, and disconnect

one total time budget — the direct operationalization of the direction comment's wait-refresh-return flow — starts at statement entry, before the admission reads, and covers admission reads, queueing, dispatch, execution, and refresh, with a termination and refresh margin reserved. The execute request carries the remaining budget with verifiable expiration information under a declared cross-host clock-skew assumption (or a verified transit-time upper bound conservatively deducted); the BE rejects an already-expired request at receipt and again at dequeue, before the mutating native invocation, and converts the admitted remainder into a monotonic local runtime limit independent of FE connectivity — not reset by BE reselection, queueing, or reconnection. Follower-to-master forwarding carries the same original budget. The mutation budget is its own configured bound: a generic query timeout is not assumed to suffice, and insufficient configuration surfaces as rejection, not silent overrun;
a request dequeued with insufficient remaining budget is rejected before the mutating native invocation as a confirmed failure;
statement timeout and explicit cancellation request best-effort termination of the worker. Best-effort termination does not mean reaping completes before the SQL returns: capacity and the active-name entry wait for termination evidence (Sections 5.3, 5.4);
a complete trusted success already received is never rewritten by a later timeout or cancellation. If the refresh cannot complete within the budget, the statement returns committed, refresh incomplete (Section 6.4). Only the absence of a reliable mutation outcome yields indeterminate;
closing a socket cannot deliver any SQL error. A client that disconnects — voluntarily, by timeout with a lost response, or through FE loss — can receive no result packet and MUST treat the outcome as indeterminate and consult authoritative metadata. This is a documentation and driver rule;
disconnect after dispatch: the dispatched worker may run to its own completion or deadline within its budget; a live FE keeps a bounded completion, refresh, and cleanup continuation. This neither guarantees a commit nor releases capacity. Explicit KILL or statement timeout, by contrast, requests best-effort termination.

6. Failure contract

6.1 Promise and non-goals

The initial version does not promise persistent job history, FE failover recovery, resumable builds, or durable same-name serialization across failures. All in-flight control state is in memory. An FE restart or failover during a statement is observed by the client as a disconnect and follows Sections 5.5 and 6.2.

6.2 Indeterminate outcomes

A timeout, disconnect, worker loss, or lost response after dispatch, when no complete trusted outcome has been received, MUST NOT be presented as proof that nothing committed. Such a statement returns an explicit indeterminate-outcome error that:

states that the mutation may or may not have committed;
is never followed by an automatic retry of the potentially committed mutation — neither by Doris nor recommended to the client as a blind resubmission;
directs the user to inspect authoritative index metadata: SHOW INDEX FROM ... and lance_index_entries(...);
never attributes observed external state to this request: observing a same-name index does not prove which request created it.

6.3 Cancellation

Cancellation (query kill, statement timeout) triggers best-effort worker termination and returns control to the client; it cannot undo an existing commit. If a complete trusted success was already received before cancellation, the statement reports the committed, refresh incomplete class rather than indeterminate (Sections 1.1 and 5.5). Worker capacity and the active-name entry remain held until termination is confirmed (Sections 5.3, 5.4).

6.4 Refresh obligations independent of outcome

Mutation outcome and metadata refresh are independent. One bounded refresh action runs inside the statement where the table below requires it; there is no refresh job and no background retry framework. Refresh failure keeps the mutation classification and attaches a separate refresh diagnostic.

Fact already known at classification	Mutation attribution	Refresh responsibility
complete matching native success	committed by this invocation	refresh before returning success
preflight `IF` no-op — authoritative state already satisfies the condition	no mutation executed this statement	complete the required bounded invalidation-and-refresh initiation on the executing FE before returning the no-op (below); an optional authoritative reload failing afterwards is an independent diagnostic only
complete trusted commit-conflict result (saved typed code)	this invocation did not commit	refresh the externally advanced metadata
pre-invocation revalidation observed relevant version/metadata advancement	this invocation did not commit	refresh the observed advancement; no forced additional reads without evidence
post-invocation aggregate error or lost communication	unknown	refresh never attributes; the user may explicitly refresh or inspect

A confirmed commit followed by a refresh failure MUST be reported as a distinct typed error — committed, refresh incomplete — never as a build failure; the direction comment requires exactly this distinction. The commit is a fact, Lance metadata is authoritative, and the user may retry inspection or refresh through ordinary means. The build outcome is never rewritten by refresh failure.

The no-op branch closes the same way. Its required actions are a bounded local cache invalidation on the executing (master) FE and initiation of the ordinary external-table refresh propagation — the existing edit-log-and-replay mechanism every other external refresh already uses — both retried inside the statement and practically always completable. Reads served by the executing FE then re-resolve from authoritative metadata and never hit cache entries predating the invalidation; caches on other FEs converge through the existing refresh-propagation semantics, and this contract claims no new cross-FE freshness guarantee beyond that existing mechanism. An optional authoritative metadata reload that fails afterwards attaches an independent diagnostic and does not change the classification. In the residual pathological case where the invalidation or the propagation initiation cannot complete, the statement returns a typed refresh error in the confirmed-failure family: no mutation was executed, it is not a build failure, and it is neither committed nor indeterminate.

A native success proves that this invocation committed. It does not guarantee that another external writer has not since replaced or removed the index; "the index exists" is a statement about this commit, not a promise about the present external state.

6.5 Provider-result classification

Complete saved result	CREATE	REPLACE	DROP
rejection before the mutating native invocation, including version/schema/credential/resource revalidation	failure	failure	failure
`LANCE_OK` from the one native mutating invocation	success (+refresh)	success (+refresh)	success (+refresh)
complete trusted saved `LANCE_ERR_COMMIT_CONFLICT` (exact code)	failure (+refresh)	failure (+refresh)	failure (+refresh)
any other saved typed code after the mutating invocation began — including generic `NOT_FOUND`, `INVALID_ARGUMENT`, `NOT_SUPPORTED`, `INDEX`, `IO`, aggregate `INTERNAL`, and typed panic	indeterminate	indeterminate	indeterminate
no matching complete response, signal, OOM, panic, timeout, BE loss, or protocol ambiguity after dispatch	indeterminate	indeterminate	indeterminate

Three properties of the pinned ABI shape this table and are normative context, not implementation detail:

only the exact commit-conflict variant maps to the dedicated conflict code in the pinned C binding; other provider conflict variants share the aggregate internal code. A same-name race may therefore surface as a generic internal error, which classifies as indeterminate. Doris never parses retry hints or any message text to improve attribution;
dataset-not-found, generic not-found, and index-not-found share one code, and the drop path includes internal reads as well as the commit. A generic not-found received after the mutating invocation began therefore cannot prove a clean missing-target no-commit — for DROP as for every operation it classifies as indeterminate. A DROP whose authoritative preflight proved the name absent resolves as an IF EXISTS no-op before the mutating invocation; there is no post-invocation DROP no-op optimization in 4.2;
aggregate error classes lose information by design: some outcomes that did not commit will report indeterminate. This is a property of the pinned ABI and is accepted.

Duplicate CREATE may return coarse INDEX or INVALID_ARGUMENT, not a stable already-exists/no-commit result. CREATE IF NOT EXISTS therefore resolves before dispatch; a later matching index is only corroboration and the outcome stays indeterminate.

6.6 Error surface

Errors are typed, bounded, and sanitized. The four outcome classes of Section 1.1 are distinguishable by the client through distinct error codes/errClasses following Doris conventions; exact codes are implementation details. No secrets, raw provider payloads, unbounded text, or fabricated progress appear in any error.

7. Authorization, confidentiality, and cleanup

CREATE, REPLACE, and DROP require table ALTER.
SHOW INDEX and the physical inspection TVF require table SHOW.
Refresh actions run as a system action only after the initiating statement passed authorization, including the cleanup continuation that survives a client disconnect (Section 5.5).
There is no job lookup surface and no operator release path.
Secrets are excluded from durable state (none exists for mutations), child argv/environment, retry buffers, logs, core, SQL results, error messages, and test artifacts.

Failed native work may leave unreferenced index files. Doris never guesses and deletes physical paths. Lance cleanup/VACUUM and historical-version retention own physical reclamation.

8. Distributed construction extension (future work, non-normative)

Synchronous SQL does not preclude distributed index construction. The desired extension is:

pin snapshot
  -> partition fragments
  -> parallel BE builds of uncommitted segments
  -> validate/collect outputs
  -> one coordinated final commit
  -> refresh
  -> return

build and commit responsibilities MUST remain separable in the synchronous implementation so this path can be added without redesign;
the baseline bindings (Section 0) expose both halves of a two-phase build: lance_index_segment_builder_execute_uncommitted returns protobuf-encoded segment metadata without touching the manifest, and lance_dataset_commit_index_segments commits pre-built segments as one logical index in a single manifest commit; the FE binding exposes commitExistingIndexSegments. Entry-point existence is not sufficient evidence: this path still requires Doris-side orchestration, transport of the uncommitted segment artifacts, verification of the segment contract, object-storage visibility, model compatibility, and exactly-one coordinated commit before implementation;
the existing segment-commit entry carries coverage-driven replacement semantics and no plain-CREATE replace=false contract. The initial version therefore keeps the single one-shot mutating call, and any two-phase capability is designed and verified separately — CREATE/IF semantics are not changed for extensibility;
Doris MUST NOT run several complete, independently committing create_index calls against the same logical index name;
the initial implementation uses one worker; multi-BE construction can follow without introducing durable jobs; persistent recovery remains a separate, justified follow-up requirement rather than a prerequisite for index lifecycle support.

9. Delivery and release evidence

9.1 Reviewable delivery slices

explicit revert/cleanup on branch-4.1: remove the durable job framework of branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235 and the job-specific parts of branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630 (job creation, JobId results, job inspection SQL, unresolved-job guards), preserving and adapting the privilege/schema/parameter/snapshot/IF validation, SHOW INDEX, and the physical TVF. Compatibility verification targets: the removed operation code keeps its tombstone and is never reused; the journal reader still decodes an old record; edit-log replay discards it; old image modules are skipped; and the JobId removal is checked against ordinary MySQL, forwarded, and Flight SQL result schemas. Existing metadata must not become unreadable;
synchronous execution path: master forwarding and the active-name entry; final in-entry preflight; dispatch, wait, classify, refresh, and return; the indeterminate outcome class and the committed, refresh incomplete class; budget and capacity accounting;
reworked isolated worker around the synchronous lifecycle — isolation, supervisor, budget enforcement, and domain-wide cleanup carry over;
fault, isolation, and query-consumption tests rewritten against Section 6;
BTREE/BITMAP lifecycle and user documentation.

Mutation remains disabled until the evidence for its slice passes.

9.2 Gates

G1 — SQL and metadata: every CREATE/REPLACE/DROP/IF case; type/property rules at admission and exact-snapshot revalidation, verified against datasets that have round-tripped through a real manifest rather than only in-memory constructed schemas; DROP and DROP IF EXISTS success paths for composite, nested, and legacy indexes, including indexes with no readable logical description or schema-contract payload, and a case where the admitted version is still readable while storage-latest has advanced (the drop proceeds from the pinned snapshot and any race classifies per Section 6.5); one-snapshot SHOW/TVF behavior including normalization, system-entry filtering, bounds, and fail-closed malformed metadata; legacy name occupancy determined from the physical list, including the legacy per-name empty-result exception; IF no-op invalidation and refresh initiation including their failure and timeout paths — CREATE IF NOT EXISTS and DROP IF EXISTS no-op cases assert zero execute sends, zero workers, the correct SQL class, name/target owner release, that a repaired refresh is retried only through a fresh authoritative preflight, and that a follower-initiated statement followed by a read on another FE observes the existing refresh-propagation semantics; case-only collisions; privileges.
G2 — synchronous lifecycle correctness: the classification table exercised over fault permutations (lost response, timeout, OOM, panic, signal, BE loss, malformed protocol, identity mismatch); no automatic retry after possible dispatch; refresh-failure reported distinctly, and the no-op refresh closure classified in the confirmed-failure family (never committed, never indeterminate); only identity-matched responses classify outcomes. Every fault case asserts three things — what the client observes, whether caches were refreshed per the responsibility table (Section 6.4), and when the executor, its capacity slot, and the active-name entry are released; asserting only a SQL error or an execution count is insufficient. Two mandatory window cases: success received, then statement timeout must report committed, refresh incomplete (never indeterminate); timeout returned, then a new same-name admission must observe busy until the execution domain terminates, then proceed from a fresh preflight.
G3 — resource safety: unavailable isolation rejects before invocation; FE and BE survive worker OOM, panics, malformed protocol, and signals; domain-wide reaping including descendants; capacity ownership holds under admission, budget expiry, reap, and BE-epoch loss; bounded metadata reads; credential rotation; secret non-disclosure; two-tier queueing with dequeue revalidation; delayed delivery, queue dwell, FE loss, cancel arriving before execute, and clock-skew boundaries — no expired mutation ever starts, with insufficient budget plus complete no-invocation evidence a confirmed failure and without it indeterminate; a lost-termination-report case — the BE confirms domain termination while the FE loses every report, after which network recovery follows the cleanup rules, consecutive different-name executions respect the retained-owner total, and same-name admission is never released by TTL or heartbeat guesses.
G4 — end-to-end usability and documentation: for each enabled index type, one focused fixed-snapshot end-to-end test creates the index through the isolated worker and proves the current query planning path consumes it; documentation covers SQL, synchronous semantics, the four outcome classes, inspection, worker settings, and Section 10's limitations.

9.3 Safety configuration invariants

The 4.2 contract does not fix setting names. Mutation defaults disabled and cannot be enabled until the implementation has a verified hard worker memory/PID/runtime boundary, finite concurrency and queue bounds, a configured total time budget covering reads, queueing, execution, and refresh, and bounded metadata/input processing. Exact settings and reviewed defaults follow Doris conventions.

10. Accepted operational limitations

an indeterminate outcome affects resubmission decisions: there is no attribution, no automatic reconciliation, and no operator release path, and a blind resubmission may duplicate or race a mutation that committed without being reported;
same-name serialization is master-local: admission preflight plus the in-memory active-name entry. A busy entry temporarily blocks new same-name requests until the execution domain terminates — and, when the termination proof is lost, until the coordinator epoch is rebuilt. There is no serialization across FE failover or against external writers — an accepted window;
FE restart loses the in-memory control plane — name entries, target tokens, budgets, and FE-side accounting. In-flight clients observe a disconnect and follow the indeterminate rule; worker capacity on BEs is unaffected and drains by its own evidence;
classification loses information to aggregate provider codes: some outcomes that did not commit report indeterminate (distinct conflict and not-found variants share codes in the pinned ABI). Message text is never consulted to improve attribution;
cache freshness: committed indexes and authoritative no-ops trigger bounded refresh inside the statement, but a refresh failure leaves caches stale until an explicit refresh; authoritative SHOW/TVF reads remain available throughout;
one worker and one selected BE per mutation initially; no native progress, no resumable builds, cancellation is best-effort termination and never rollback, and there is no continuously appended-table freshness SLA;
query planning: branch-4.1's existing planner already generates splits for vector index segments and uncovered fragments; this issue adds no query architecture, and each enabled index type must prove the current path consumes it;
initial type, metric, nullability, schema, and property limits for CREATE and REPLACE remain strict; vector element nullability is not among them (it is unobservable at the schema level, Section 2.4).

11. Review request

Verified baseline evidence. The four facts below were corrected against the pinned sources on 2026-10-09 and shape the contract above; please flag any disagreement:

the commit-resolved baseline: Doris branch-4.1 c8c0ddca4d; FE lance-core 12.0.0; BE lance-c commit 98468344 (version string 0.1.9; vars.sh-pinned; Rust core 68c12dfd) — package versions and release tags alone do not determine ABI behavior;
panic handling in the pinned C binding: panics are caught at the FFI boundary and reported as a typed panic error; unwinding-incompatible build settings are rejected at compile time; OOM, stack overflow, and double panics can still terminate a process — which is why isolation requirements are retained on resource grounds;
coordinated-commit availability: the FE binding exposes commitExistingIndexSegments, and the pinned C binding exposes both the uncommitted build half (lance_index_segment_builder_execute_uncommitted) and a C-side segment-commit entry (lance_dataset_commit_index_segments);
error mapping: only the exact commit-conflict variant reaches the dedicated conflict code; other conflict variants share the aggregate internal code, and dataset/generic/index not-found share one code — so post-invocation aggregate codes cannot prove non-commit.

The single open decision — client-disconnect semantics. A plain disconnect after dispatch lets the dispatched work finish within its budget: no commit is guaranteed, no result packet can be delivered, and the client treats the outcome as indeterminate (Sections 1.1 and 5.5). Explicit KILL or statement timeout, by contrast, requests best-effort termination; a complete success already received before cancellation is still reported as committed, refresh incomplete rather than indeterminate (Section 6.3). This is the one place the design departs from Doris convention, which cancels queries when the client disconnects. The choice is a product trade-off, stated honestly: terminating on disconnect as well is coherent, would free the worker's CPU, memory, and slot sooner, and a disconnected client observes indeterminate under either policy — it cannot receive a result packet either way. Letting dispatched work finish is recommended because the work was explicitly requested, its completion becomes authoritatively inspectable, and its continued resource use is already bounded by the statement budget. Confirmation or rejection of this single choice is requested.

Boundary confirmation. Please confirm whether this document accurately reflects the final Doris 4.2 implementation boundary. A simple Yes would be helpful; if not, please identify any specific conflict with Required in 4.2 (Section 1.2) or Explicitly Deferred (Section 1.3).

Please review this comment-thread document as the complete proposal; no requirement should be inferred from v5/v5.1 or from the review history they capture.

PR roadmap (new live tracking comment — I will keep this one edited up to date as PRs move)

Replaces the earlier roadmap comment (pre-2026-10-08, now marked deprecated). The synchronous execution contract is the published v6 proposal (design comment, 2026-10-14); the sync-series PRs implement it. The split was redrawn after the synchronous-direction change (direction comment, 2026-10-08): mutations execute synchronously inside the SQL statement; the durable job framework was reverted rather than kept as a second path.

Piece	PR	Status	Contents
Read-only foundation	#66637	merged	Directory-authoritative `SHOW INDEX`
Read-only foundation	#66671	merged	`lance_index_entries` TVF (physical UUID / dataset-version inspection)
DDL surface	#67201 (PR3A)	merged	Statically validated DDL surface; mutations still rejected
Durable job framework	#67235 (PR3B)	merged → reverted by #68781	Job model/state machine, fences, quotas, replay
Admission + job SQL	#67630 (PR3C)	merged → job parts reverted by #68781	Privilege/schema/parameter/snapshot/IF admission (retained), job creation/JobId/`SHOW LANCE INDEX JOB(S)`/guards (reverted)
Async dispatcher	#67978 (PR3D)	closed	Superseded by the synchronous direction
FORCE_RELEASE + retention GC	#67754 (PR3E)	closed	Superseded (no durable job state remains)
Isolated worker (async form)	#68668 (PR4A)	closed	Isolation/supervisor assets carried into the synchronous worker PR
Tracer/fault suites (async form)	#68669 (PR4B)	closed	Retained as assets for the synchronous test PR
Revert	#68781	merged (2026-10-10)	Removes the durable job framework; keeps admission validation, `SHOW INDEX`, TVF, DDL guard, gate (default off)
Sync series 1	#68957	open (ready for review)	Synchronous execution path: four-class outcome model, thrift dispatch contract + BE stub, master-side executor (active-name entry, bounded admission, budget), refresh obligations, statement wiring
Sync series 2	#69145	draft	Isolated worker backend for the synchronous lifecycle: cgroup v2/pidfd supervisor, worker library with pinned-version re-verification, BE handler/service (bounded queue, dedup, two-state capacity, double-reject expiry), worker phase + identity echo on the result contract, golden contract fixtures locked from both sides (reworks the #68668/#68669 assets)
Sync series 3+	—	planned	Fault/isolation/query-consumption tests against the synchronous contract; BTREE/BITMAP lifecycle; user documentation

#66637 + #66671 (read-only, merged)
        |
#67201 (DDL surface, merged) --> #67630 (admission, merged)
        |                             |
        |                     #68781 (revert job framework, merged)
        |                             |
        v                             v
   #68957 (open): synchronous execution path on post-revert branch-4.1
        |
        v
   #69145 (draft, stacked on #68957): isolated worker execution --> tests / BTREE+BITMAP / docs (planned)

All PRs target branch-4.1. Mutation remains disabled (enable_lance_index_mutation default off) until the execution evidence gates pass.

[doc](lance) add doc about lance_index_entries TVF (#4311)

	\| CREATE INDEX (IF NOT EXISTS)? name=identifier
	ON tableName=multipartIdentifier identifierList
	(USING (NGRAM_BF \| INVERTED \| ANN))?
	properties=propertyClause? (COMMENT STRING_LITERAL)? #createIndex

	@Override
	public void validate(ConnectContext ctx) throws UserException {
	if (indexDef == null) {
	throw new AnalysisException("index definition expected.");
	}
	if (tableName != null) {
	tableName.analyze(ctx);
	}

	indexDef.validate();
	index = indexDef.translateToCatalogStyle();
	}

	@Override
	public AlterTableClause translateToLegacyAlterClause() {
	return new CreateIndexClause(tableName != null ? tableName.transferToTableName() : null,
	indexDef.translateToLegacyIndexDef(), index, alter);

[Feature](lance) Add vector and scalar index lifecycle management #66497

Description

Parent

Scope

Completion criteria

Delivery status — 2026-09-30

Activity

Gabriel39 commented on Aug 5, 2026

u70b3 commented on Aug 5, 2026

u70b3 commented on Aug 6, 2026

1. Goals and Phase-1 boundary

2. SQL surface

3. SHOW INDEX and the Lance index model

4. Execution model

5. Authorization

6. Lifecycle semantics

Versioning and visibility

Concurrency

Failure recovery and retries

Drop and metadata refresh

7. Initial supported types and catalogs

8. Implementation and test orientation

Questions for reviewers

References

Gabriel39 commented on Aug 6, 2026

Blocking concerns

1. A synchronous native build in FE is not a safe production execution model

2. Reusing the current Doris index command chain requires a larger refactor than the proposal describes

Other required revisions

3. Incremental maintenance is necessary for a usable external index lifecycle

4. Do not put unbounded physical segment metadata in SHOW INDEX.Properties

5. REST behavior should be capability-based, not rejected by catalog type

6. Narrow and rename the initial type set

7. Define SQL identifier semantics for index names

8. Refresh/recovery must handle post-commit failures

Answers to the review questions

u70b3 commented on Aug 6, 2026

Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle

Major changes from v1

1. Goals and Phase-1 boundary

2. SQL surface

Vector creation

Scalar creation

Full replacement and incremental maintenance

Inspection, status, cancellation, and drop

Statement semantics

3. Neutral command model and catalog dispatch

4. Asynchronous execution model

Durable job

Responsibility split

Resource and cancellation behavior

5. Provider and catalog behavior

Directory Catalog

REST Namespace Catalog

6. Initial type and property matrix

IVF_PQ

BTREE

BITMAP

Deferred types

7. Logical and physical metadata surfaces

SHOW INDEX

Physical metadata

8. Authorization and identifier semantics

9. Versioning, concurrency, and visibility

10. Failure recovery and metadata refresh

Confirmed pre-commit failure

Confirmed commit

Unknown or post-commit failure

11. Test plan

12. Confirmed design decisions and implementation decomposition

Remaining implementation-planning question

Gabriel39 commented on Aug 6, 2026

1. The REST design currently promises more than the pinned Namespace API can express

2. Reusing SHOW/CANCEL BUILD INDEX requires a real external-job integration design

3. REST reconciliation needs job-specific provenance, not only a matching postcondition

4. DROP also needs a durable operation record

5. Please make the distributed segment contract explicit

6. Metadata details

Integration-model recommendation

u70b3 commented on Aug 6, 2026