Repository navigation
branch-4.1: [feature](lance) Add FE orchestration and RPC contract for synchronous index mutations - #68957
Open
u70b3 wants to merge 9 commits into
Open
branch-4.1: [feature](lance) Add FE orchestration and RPC contract for synchronous index mutations#68957u70b3 wants to merge 9 commits into
u70b3 wants to merge 9 commits into
Conversation
Contributor
Author
|
run buildall |
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
u70b3
marked this pull request as ready for review
October 14, 2026 01:59
u70b3
force-pushed
the
sync1-lance-index-execution
branch
from
October 14, 2026 01:59
c9dfed7 to
60d0c2b
Compare
u70b3
marked this pull request as draft
October 14, 2026 01:59
1 of 13 tasks
u70b3
changed the title
u70b3
marked this pull request as ready for review
October 15, 2026 02:33
…ronous index execution Introduce LanceIndexMutationOutcome, the four-way client-facing outcome of one synchronous Lance index mutation statement (SUCCESS, CONFIRMED_FAILURE, INDETERMINATE, COMMITTED_REFRESH_INCOMPLETE). The model is immutable, carries the typed lance result code and a bounded sanitized message (control characters stripped, UTF-8 truncated to 1KB, sharing the LanceErrorMessages implementation), and maps each non-success kind onto its own stable error code via toUserException(). INDETERMINATE text states the mutation may or may not have committed and points at SHOW INDEX FROM / lance_index_entries() instead of inviting a blind retry; COMMITTED_REFRESH_INCOMPLETE keeps the commit fact and never reads as a build failure. Register the mutation error-code family 5104-5108 after the retired 5103 (kept unused per the existing comment): indeterminate, committed-refresh-incomplete, busy, refresh-failed (no-op local invalidation), and rejected (confirmed non-commit). No callers are wired yet; the statement/executor paths land in follow-up commits.
…end stub Add the FE->BE thrift dispatch contract for the synchronous Lance index mutation path (design v6 slice 2; executor wiring lands separately): - TLanceIndexMutationRequest / TLanceIndexMutationResult and TLanceIndexMutationType in BackendService.thrift, plus the lance_index_mutate service method: blocking single request-response semantics, single-send (at most one send per invocation_id, no retry on any path). Field surface follows the closed dispatcher series' TLanceIndexJobDispatch, renumbered from 1 because job_id and dispatch_revision died with the durable job framework; ids are additive-only from here on. deadline_ms is absolute epoch millis; storage_options keeps the TLanceScanParams provider-opaque contract; schema_contract_json carries the survived schema contract v1 representation (unset for DROP). - TLanceIndexMutationResult: status is consumed first - a complete non-OK status (NOT_IMPLEMENTED, UNKNOWN_METHOD from an old BE) proves not-executed and classifies as confirmed failure without retry; lance_error_code (raw native LanceErrorCode, 0 = LANCE_OK) is consumed before the bounded message and the message text never participates in classification. Slice 3 appends worker-phase markers at ids 4+. - BE stub: BaseBackendService answers NOT_IMPLEMENTED_ERROR with a fixed short message, mirroring the closed dispatcher series stub, so the new BackendServiceIf virtual does not break the BE build and a dispatched mutation classifies as a clean pre-invocation rejection. - FE test-side Iface implementors (MockedBackendFactory, GenericPoolTest) get the same clean rejection override. - LanceThriftContractTest pins both structs' field ids and IDL snake_case names via metaDataMap, the mutation type values, the service method existence, and a compact-protocol round trip. Generated thrift sources are not committed.
Execute one admitted Lance index mutation synchronously inside the statement on the master (design v6 slice 2; statement wiring lands in a follow-up commit). No statement class is touched yet. - Dispatch seam LanceIndexMutationDispatcher (request -> result): the production ThriftLanceIndexMutationDispatcher walks the blocking BackendService.Client from the shared backend pool with the remaining statement budget as the socket timeout (borrowObject(address, timeout)), returns the connection with the pool default timeout restored or invalidates it after a failed call, and never wraps a failure - any TException reaches the classifier raw. Single send is structural: one call per invocation id, no retry path exists. Cancellation is best effort: the blocking read cannot be interrupted, the budget-bounded socket timeout closes it, and cancelling never rolls back a possible commit. - LanceIndexMutationExecutor holds the synchronous semantics end to end: a per (catalog, normalized dataset uri, normalized index name) AtomicReference CAS entry as the busy guard (Dictionary LOADING style - acquired by map insertion, released exactly once by its owner in a finally, never persisted, self-healing on restart or master switch); same-name statements fail fast with the typed busy error 5106 and are never queued behind the running one; entry lifetime runs to classification and cleanup, outliving an early statement return. - Bounded admission: a dedicated pool shaped like the Lance read pool (ThreadPoolManager fixed + AbortPolicy), sized from three new config items (lance_index_mutation_max_concurrency=2, max_queued=16, budget_cap_seconds=300, each mutable/masterOnly with the positive-int callback; a live pool is not resizable so changes need a restart, and fe.conf-loaded values get the positive invariant re-asserted). A full pool or queue is the same typed busy rejection. - Entry phases in strict order: final preflight (the caller's throwing seam - a no-op verdict completes without any dispatch and even without budget; a typed rejection keeps its own error code), budget check (a pre-dispatch exhaustion is a confirmed failure 5108 whose message says nothing was sent), one alive backend via the scheduler policy (none is a confirmed failure), request build (invocation id minted once, BE process epoch from the heartbeat, absolute deadline_ms, storage options resolved from the catalog at send time, schema contract JSON for CREATE/REPLACE only), single dispatch, then classification. - Classification (v6 6.5, message text never participates): complete OK status with lance_error_code 0 -> success; complete non-OK status and the UNKNOWN_METHOD TApplicationException of a rolling-upgrade backend -> confirmed failure; the exact typed commit-conflict code 8 -> confirmed failure; any other typed code, missing code, transport error, timeout, or partial answer after the send -> indeterminate 5104; pre-dispatch rejections -> confirmed failure 5108. - budgetDeadlineMs: the statement budget is min(query_timeout, config cap) from statement entry, shared by admission, dispatch, and wait; a statement-side expiry during the wait surfaces indeterminate while the abandoned entry task converges alone (its own budget re-check stops an un-dispatched task) and releases the entry. - Tests: LanceIndexMutationExecutorTest covers the answer matrix (success, clean non-OK status, commit conflict, generic typed code, missing code, unknown method, timeout, connection error), pre-dispatch rejections (budget, no backend, cancelled), the no-op zero-dispatch assertion, preflight exception propagation with entry release, same-name fast-fail busy, guard-key normalization, pool-and-queue saturation busy, budget expiry during the wait with no resend, an 8-way same-name concurrency pass (exactly one dispatch, seven busy), a mixed-fault liveness pass asserting every entry is released, non-master rejection, wire request shape for CREATE and DROP, and the budget cap helper; the config validator test extends to the three new items (defaults, wiring, rejection, assignment).
…mutations Close the v6 section 6.4 (PR1 subset) obligation table of the synchronous Lance index mutation path: what each classified outcome owes the FE's local caches, and how a failed obligation is reported without ever misreporting the commit. Investigated first, then implemented (LanceIndexMutationRefresher javadoc records the facts): - The authoritative index metadata reads (SHOW INDEX, lance_index_entries(), the admission snapshot) are uncached end to end: LanceCatalogClient#inspectTableIndexes resolves table access uncached and opens a fresh independent Dataset per read at storage-latest, so no FE cache can serve a stale index listing and none is invalidated. - The Doris-side table-scoped caches that hold data derived from one dataset version (external schema cache, external row-count cache, the Lance table-access cache) are fenced by one purely local call, ExternalMetaCacheMgr.invalidateTableCache(table) - the same fence REFRESH TABLE applies for a typed table; no network read is performed or pretended. - The native Lance Session metadata/index caches are version-keyed and cannot be invalidated per table from the FE; a committed mutation produces new versions, which are new keys, so no stale index metadata is reachable through them. The REFRESH CATALOG Session rotation is deliberately not part of this obligation. LanceIndexMutationRefresher is the seam (invalidateTableMetadata() -> null on completion, bounded reason on failure); the production LocalTableInvalidation performs the invalidateTableCache call with a bounded retry (local and non-blocking, so no backoff) and never throws. The executor owns the obligation table: a request carrying the mutated table derives the production refresher (an explicit one still wins; the no-op default remains for tests and obligation-less callers) so the statement wiring cannot forget it. - Trusted commit -> local invalidation; failure downgrades the outcome to COMMITTED_REFRESH_INCOMPLETE (5105): the commit stands, is never rewritten as a build failure, never denied, never retried here. - No-op -> local invalidation (defensive fence for table-scoped caches populated against an older dataset version during the statement's lifetime); residual failure is the typed refresh error 5107 of the confirmed family - no mutation happened and the text says so. - Typed commit-conflict confirmed failure -> best-effort invalidation of the externally advanced metadata; failure only attaches a diagnostic to the message and never changes the classification (still 5108). - Every other class (clean pre-execution rejection, indeterminate) owes no refresh. Tests: LanceIndexMutationRefreshTest covers every row of the table - commit+refresh success, commit+refresh failure reported 5105 with the word committed in the client-facing text, no-op+refresh success with a zero-dispatch assertion, no-op+refresh failure reported 5107, conflict refresh attempted, conflict refresh failure keeping 5108 with the best-effort diagnostic, clean rejection and indeterminate owing nothing - plus the production refresher itself (single invalidateTableCache call, bounded retry with an honest attempt-counting reason, success on retry, missing manager reported), and the table-derives-production-refresher default with explicit override.
…ements Complete PR1 (design v6 slice 2): with the gate on, a top-level CREATE [OR REPLACE]/DROP INDEX that passes admission now executes synchronously inside the statement on the master instead of terminating with the shared not-supported rejection. The unsupported surfaces keep their 5101 exactly as before (REST catalogs fail fast, ALTER TABLE ADD/DROP INDEX keeps the generic external-table rejection), and the gate-off 5102 fork is untouched. - Admission ends the capture phase by returning an execution plan (LanceIndexMutationPlan) instead of rejecting: the pinned dataset version and uri, the normalized index name, the stored column name, the requested algorithm, the request properties, the IF flags, the schema contract (CREATE/REPLACE only), and the entry-internal final preflight supplier. The plan is produced inside the withLanceIndexAdmission critical section by local work only. The IF/existence determination is factored into decideCreate/DropAgainstSnapshot and shared between admission and the final preflight, so the entry's re-decision against current metadata keeps the exact admission semantics: a no-op verdict completes with zero dispatches, a mismatch keeps its own 5100-family rejection, only a fresh PROCEED may dispatch. The admission-level IF no-op early return stays: it is still the plain OK packet. - AlterTableCommand.run builds the MutationRequest outside the critical section: the budget deadline runs from statement entry (budgetDeadlineMs(statementStartMs)), the cancel signal is the command's cancel() flag (StmtExecutor.cancel calls it, the InsertOverwriteTableCommand precedent), setTable derives the refresh obligation, and a non-success outcome is thrown as its own client-facing class via toUserException(); the executor's typed busy 5106 and preflight rejections propagate unchanged. The ForwardWithSync master-forwarding semantics are untouched. - The enable_lance_index_mutation description no longer claims enabling only yields the not-supported error. - Tests: the mutation suites' gate-on admitted-mutation assertions flip from 5101 to "dispatch attempted, terminal confirmed failure" - in the UT chain (no alive backend) that is 5108 with the nothing-was-sent text, and both authoritative reads ran (admission's pinned one plus the entry preflight's fresh one). Plan-production assertions cover the pinned version, IF flags, normalized name, stored column, algorithm and contract presence; new admission tests drive the final preflight supplier through no-op/mismatch/proceed on both CREATE and DROP; new command tests assert the execution runs outside the admission critical section (the executor's fresh read finds the catalog exclusive lock acquirable), that cancel() converges the execution as a confirmed failure with exactly one metadata read, and that an entry-preflight mismatch propagates its 5100 through the executor. Gate-off, IF no-op, IF mismatch, REST and alter=true coverage is unchanged.
…tations
Flip the regression coverage to the post-PR1 terminal state: with the
mutation gate open, an admitted CREATE [OR REPLACE]/DROP INDEX now
dispatches synchronously to the backend, whose stub lance index worker
answers a complete NOT_IMPLEMENTED rejection, so the statement ends as
the typed confirmed failure 5108 with nothing committed (the UT chain
asserts the same class via its no-alive-backend text). The gate-off
5102 early rejection, the IF no-op OK packet, the static matrix, and
the REST fail-fast coverage are unchanged.
test_lance_index_admission: the two admitted CREATE cases and the
admitted DROP IF EXISTS flip from the old 5101 "not supported for Lance
catalog tables" text to the confirmed-failure class marker; one CREATE
copy pins the stub worker's own message ("lance index worker is not
available in this build") carried inside the typed error. A new closing
block cross-checks both read faces through the same per-run catalog:
lance_index_entries() still lists exactly the preloaded index with no
leftover from the rejected CREATEs, and SHOW INDEX still resolves the
preloaded name.
test_lance_index_ddl: keep the gate-off 5102 suite as the early-reject
coverage; add the missing negative neighbors - ALTER TABLE DROP INDEX
keeps the generic external-table rejection (ADD INDEX already had it),
BUILD INDEX on a Lance catalog table is rejected by its internal-catalog
lookup, and Lance-only syntax (USING ANN, CREATE OR REPLACE INDEX) keeps
failing on an ordinary internal table before internal index validation.
The suites cannot run locally (docker MinIO+REST fixture); both changed
files were syntax-verified by full groovy compilation.
…tion
Review-fix round for the synchronous Lance index mutation PR1 (three P2
precision/closure fixes plus the P3 cleanups):
- Admission IF no-op closure (finding 5-1): a CREATE IF NOT EXISTS or
DROP IF EXISTS whose no-op verdict was decided at admission now
completes the same bounded local table-cache invalidation the
entry-internal no-op completes, so both no-op paths behave
identically; a residual failure after the bounded retry surfaces as
the typed 5107 no-op refresh error instead of a plain OK.
- Provable pre-send transport failures (finding 1-1): the thrift
dispatcher splits the borrow from the single send. Everything the
borrow can throw belongs to connection establishment or pool
admission - the invocation is written only by lanceIndexMutate - so
it propagates wrapped in the new typed PreDispatchTransportException
and classifies as confirmed failure 5108 ("the backend connection
could not be established before the send ... nothing was sent").
Failures after the borrow succeeded (read timeout, reset, broken
pipe, EOF) stay indeterminate. The seam is the statement boundary,
never exception-type guessing.
- Typed entry preflight errors (finding 3-1): a non-AnalysisException
failure of the final preflight snapshot read (storage hiccup, JNI
failure - reachable, not a wiring bug) classifies as confirmed
failure 5108 with a bounded description instead of a bare
IllegalStateException; AnalysisException keeps its own error code.
- Dispatcher finally guard (finding 1-2): a pool failure while
returning a completed connection only degrades that connection to
invalidated - the trusted answer in hand is never replaced.
- 5108 fixed wording (finding 10-1): the parallel form "either nothing
was sent to the backend, or the complete trusted result proves this
invocation made no change".
- Budget fast convergence (finding 4-1): a queued task whose budget
already elapsed converges before spending the preflight JNI read;
the post-preflight recheck stays.
- Comment corrections (B/C/D): the master assertion is purposely
one-shot at execute entry, the admission no-op's empty fenced action
is deliberate target revalidation, and the outcome javadoc no longer
claims reportAnalysisException stamps ConnectContext state.
New LanceIndexMutationDispatcherTest drives the real dispatcher over
real loopback sockets: connect refusal types as pre-dispatch, a live
stub backend's answer is returned with the connection reused, a
call-phase failure stays raw and invalidates, and a pool return
failure does not discard the result. The admission no-op suites now
verify the invalidation; the executor's no-op-with-no-budget test is
replaced by the expired-budget convergence test, with the no-op success
staying pinned in the refresh suite.
Publish the mutation fact before refreshing caches so a later timeout keeps a known commit, conflict, or authoritative no-op. Synchronize the send decision with statement expiry and cancellation, and recheck it after acquiring a backend connection. Keep cleanup ownership until the task actually ends. Carry the master statement deadline through authoritative admission reads and no-op invalidation. Run namespace resolution on the bounded read pool with a generation lease held until the native owner returns. Include legacy physical names when deciding occupancy and IF behavior; revalidate the captured catalog identity and dataset URI during final preflight. Add optional request field 15 for CREATE display spelling, keeping normalized identity and existing thrift field ids unchanged. Add regression tests for the timeout, ownership, pre-send, legacy-name, display-name, and target-change boundaries.
u70b3
force-pushed
the
sync1-lance-index-execution
branch
from
October 15, 2026 02:36
f52a675 to
d7822bd
Compare
Draft
3 tasks done
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static review of PR #68957 at d7822bd covered all 30 changed paths. Two P2 defects are reported inline: synchronous validation/schema reads can escape the total mutation budget, and production error wrappers can erase a metadata timeout before its confirmed-failure classification. Both occur before backend dispatch. No existing inline findings needed to be carried forward, and no additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and behavior: the change introduces synchronous master-side admission, one dispatch, outcome classification, and local invalidation. The BE worker is intentionally a NOT_IMPLEMENTED placeholder. The two budget/error defects prevent the FE contract from being fully satisfied.
- Scope and clarity: the change is focused on the Lance index statement path, its wire contract, configuration, and tests.
- Concurrency and locks: statement, inspection, and mutation workers were traced. Catalog fences contain local actions; RPC/JNI runs outside catalog locks. The active-name entry monitor serializes send permission against stopWaiting, and finally owns release. No additional lock-order defect was substantiated.
- Lifecycle and initialization: a worker retains its catalog generation through namespace/JNI work and cleanup. Retirement rejects acquisition after the last lease ends; early caller return preserves worker ownership. No new cross-translation-unit static initialization dependency applies to the added C++ stub.
- Configuration: the budget cap is read per statement. Pool and queue sizes are positive and explicitly restart-only despite mutable configuration fields; validation callback tests cover their accepted/rejected values.
- Compatibility: the new RPC has an explicit old-BE UNKNOWN_METHOD rejection path. Existing wire/storage definitions are unchanged, and no persisted format changes.
- Shared definitions: the touched Thrift definitions were compared on origin/master and branch-4.2, branch-4.1, branch-4.0, and branch-3.1. All lack the new types/method; enum values and field IDs have no cross-branch collision. Required baseline fields belong to wholly new structs, so no shipped decoder acquires a new required field.
- Parallel paths: CREATE/REPLACE/DROP, both IF no-op paths, legacy physical-only names, case collisions, REST rejection, internal-table guards, ALTER TABLE syntax, BUILD INDEX, forwarding, and cancellation were inspected. Non-query forwarding disables replay retries.
- Conditional checks: pre-send expiration/cancellation checks cover separate phase boundaries; post-send ambiguity remains distinct from trusted refusal and known commit/refresh outcomes. No additional current-head conditional defect was verified.
- Test coverage: unit additions cover classification, bounded queues, same-name contention, borrow/answer/refresh expiry, ownership, preflight identity/existence, no-op refresh, socket rejection, wire shape, and text bounds. Regression changes exercise the rejecting stub and preserved index listings. Cold validation IO and an inner read timeout shorter than the outer budget need coverage with the two fixes.
- Expected results: no expected-result files changed; the inspected new expectations match the deliberate rejecting-worker behavior.
- Observability: bounded client-facing outcome codes and diagnostics cover the current dispatch/refusal/refresh paths; no further current-path observability defect was established.
- Persistence and failover: the busy map is intentionally local state; this slice adds no transaction persistence or EditLog records. Real native commit/recovery remains outside the placeholder implementation.
- Data writes and atomicity: the current BE always refuses before storage access. Native atomicity, optimistic conflict semantics, and provider behavior could only be assessed as future-worker protocol obligations, not executed behavior.
- FE/BE propagation: request construction carries identity, deadline, operation, normalized/display names, dataset URI/version, optional properties/schema contract, and send-time storage options. DROP omits create-only schema inputs; the result uses the same synchronous connection.
- Performance and resource use: mutation/inspection pools and queues are finite, same-name contention fails before queueing, and no resend loop was added. The synchronous resolution budget gap is the reported resource/liveness defect.
- Additional risk sweep: queued generation retirement, target/property changes around preflight, provider diagnostic redaction, cache invalidation, and known-outcome/refresh races were checked. Send-time routing snapshots, compute-group placement, and arbitrary provider-result sanitization must be reassessed when a real worker is added; no presently reachable wrong-target mutation or new credential disclosure was substantiated with the fixed rejecting stub.
Validation was static only: no builds, tests, source edits, or submodule initialization. No unresolved present-head candidate or changed-path coverage gap remains. Review rounds and convergence are recorded separately by this invocation's delivery metadata.
Review converged after 2 round(s).
| // The Lance mutation budget starts at statement entry and is shared by admission | ||
| // (the snapshot read below), the executor's admission queue, the dispatch and the wait. | ||
| long statementStartMs = System.currentTimeMillis(); | ||
| validate(ctx); |
Contributor
There was a problem hiding this comment.
[P2] Bound the metadata reads performed during validation
With a cold schema cache (for example after REFRESH TABLE), validate() calls LanceIndexMutationValidator.validateCreateIndex -> table.getColumn -> LanceExternalTable.initSchema -> LanceCatalogClient.readTableSnapshot, which performs namespace/Dataset IO synchronously on the statement thread. These reads run before any bounded wait starts, so even a 1-second lance_index_mutation_budget_cap_seconds can leave CREATE INDEX blocked for the storage timeout; the cancel flag is only observed later. Admission can similarly reload the schema via storedColumnName(). Enforce the shared deadline for these resolution/schema reads as well, with native resources remaining owned by their worker; recording the start time or checking the deadline after a synchronous read does not bound the wait.
| private static <T> T withAdmissionBudget(long deadlineMs, Callable<T> action) throws Exception { | ||
| try { | ||
| return LanceIndexInspectionExecutor.withDeadline(deadlineMs, action); | ||
| } catch (LanceIndexInspectionExecutor.MetadataReadTimeoutException e) { |
Contributor
There was a problem hiding this comment.
[P2] Preserve the timeout type through production error wrappers
This catch does not see the metadata reader timeout on the production path: LanceCatalogClient.inspectTableIndexes() catches it and LanceErrorMessages.failure() replaces it with a RuntimeException, then LanceExternalCatalog.loadTableIndexAdmissionSnapshot() wraps it again. With query_timeout=300 and the default 300-second mutation cap, the inspection executor times out after its own 60-second limit while the statement deadline still has time left, so withDeadline() rethrows the wrapper and this mapping is bypassed. The initial admission then reaches StmtExecutor as ERR_UNKNOWN_ERROR (1105), instead of the confirmed non-commit error (5108). Preserve the typed timeout across the sanitized wrappers and cover a reader timeout shorter than the statement budget; the new one-second test only covers exhaustion of the outer deadline.
Run cold database, table, and schema validation reads on the bounded metadata pool under the original statement deadline. Keep native resource cleanup with the worker after the statement stops waiting. Resolve stored column names directly from the pinned admission schema. Preserve the typed metadata timeout across sanitized error wrappers so an inner read timeout before dispatch remains a confirmed rejection. Add cold-resolution, snapshot-column, and production-wrapper regression coverage, including read ownership after the caller returns. Correct the internal ANN regression expectation, retain Lance-only BTREE/BITMAP guards, and cover the rejecting BE stub for all operations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem?
After #68781 removed the durable Lance index job framework, admitted index mutations still had no execution handoff. This PR adds FE orchestration and the FE-to-BE RPC contract for the synchronous SQL direction requested in #66497.
The v6 design proposal was published on 2026-10-14 and is under review. This draft PR is Sync series 1 in the current PR roadmap: the FE orchestration and RPC-contract foundation for the synchronous-execution slice in v6 §9.1, following the reviewer's October 8 direction.
V6 §11 still requests a decision on post-dispatch client-disconnect semantics and confirmation of the final Doris 4.2 boundary. This PR keeps the default-disabled gate and a rejecting BE stub, so its FE/RPC foundation can be reviewed independently. The real-worker follow-up will implement the disconnect policy agreed in design review; this PR does not establish full v6 acceptance or complete the enabled execution path.
Issue Number: #66497
Problem Summary
Directory-catalog
CREATE INDEX,CREATE OR REPLACE INDEX, andDROP INDEXretain privilege, schema, property, snapshot, and IF validation. Non-no-op requests execute on the master through an in-memory active-name guard, bounded admission, final authoritative preflight, and at most one execute send. No job id or mutation control state is persisted.The shipped BE implementation is a rejecting stub:
enable_lance_index_mutation=false(default)NOT_IMPLEMENTED_ERROR, classified as confirmed non-executionThe FE result model distinguishes success, confirmed failure, indeterminate outcome, and committed-with-refresh-incomplete (error codes 5104–5108, including busy and no-op refresh errors). Potentially committed mutations are never automatically retried. Only the exact pinned native commit-conflict code proves a post-invocation non-commit; aggregate codes remain indeterminate. Messages do not participate in classification.
Mutation facts are published independently of refresh completion. A success received before a refresh timeout remains committed-with-refresh-incomplete; a known conflict remains confirmed failure; a no-op refresh timeout stays in the no-mutation refresh-error family. Name entries and read resources remain owned until their actual cleanup finishes.
The master-side statement budget is
min(query_timeout, lance_index_mutation_budget_cap_seconds)(cap default: 300 seconds). Authoritative admission reads, FE queueing, result waiting, and local refresh share its deadline. Namespace resolution and JNI inspection run on the existing bounded read pool with a generation lease held until the read returns. Connection acquisition retains the shared backend pool connect timeout and may outlive the statement; a synchronized send permit prevents any later send once the statement has expired or been cancelled before dispatch. The FE mutation pool defaults to 2 running / 16 queued.Name occupancy uses the union of physical names and available logical descriptions, including legacy indexes without readable details. The final preflight revalidates the captured catalog identity and dataset URI. The RPC keeps normalized name identity and adds optional field 15,
index_display_name, to preserve CREATE spelling; DROP/REPLACE resolve existing names from the admitted physical snapshot.ALTER TABLE ... ADD/DROP INDEXandBUILD INDEXremain unsupported for Lance tables. Existing master-forwarding semantics are retained.Remaining integration and release work
The following v6 obligations remain for follow-up PRs before real mutation can be enabled:
These are remaining integration and release obligations of the proposal. Distributed construction and REST surfaces are separately deferred from the initial 4.2 scope by v6 §1.3. Mutation remains disabled by default; the rejecting BE stub performs no native mutation.
Validation
FE Maven reactor build, including Checkstyle and compilation, passed with:
fe-core: 44 test classes / 543 tests; fe-common: 5 tests. 0 failures, 0 errors, 7 existing environment/native skips. The broader selector includes command and catalog-guard suites.
Added 16 boundary tests covering known outcomes before refresh timeout, zero sends after expiry/cancellation during connection acquisition, initial admission and no-op refresh deadlines, retained read/cleanup ownership, physical-only legacy names, mixed-case CREATE spelling, and catalog/URI changes during final preflight.
Thrift regenerated with compiler 0.24.0; Java field-id/compact-protocol tests passed. The generated
BackendService_types.cpppassed a C++20 syntax compile against the installed Thrift headers (one Thrift deprecation warning).Full FE tests, full BE build/UT, and external integration regression were not rerun for this revision; their CI results must be checked separately. No historical full-suite or CI result is presented as validation of this commit.
Release note
None. The mutation gate remains disabled by default, and this build supplies a rejecting BE stub.