Repository navigation
Conversation
9532545 to
79d77e0
Compare
2a7270a to
4b54dff
Compare
d4976d2 to
24d0455
Compare
5e6aef2 to
1cb60c4
Compare
maximthomas
left a comment
There was a problem hiding this comment.
praise: The PR adds concrete ownership and delivery assertions around the concurrency paths it changes.
RemotePendingChanges.addDependencychecks both pending-object identity and replay ownership before parking; each matching dependency branch stops immediately.ReplicationServerShutdownSyncTest.shutdownForwardsRemoteDirectoryUpdatesToEveryPeercompares the exact delivered CSN sequence on both peers, including a peer held behind its send window.
issue (blocking): Old RS teardown can cancel the replacement handler's offline-message obligation.
opendj-server-legacy/src/main/java/org/opends/server/replication/server/ReplicationServerDomain.java:1393, :1361, :1375 — F-fe34bc52eb532c7384c0.
The new unlocked awaitShutdown() interval lets a same-ID replacement register and receive a freshly announced ReplicaOfflineMsg while the old handler is still closing. When the old wait finishes, its unconditional cleanup cancels current obligations using only the numeric ID:
localReplicationServer.getDSRSShutdownSync()
.replicaOfflineMsgNotForwarded(baseDN, rsHandler.getServerId());That can release collocated-DS shutdown before the replacement forwards its message. The new direct-update barrier cannot compensate because ReplicaOfflineMsg.contributesToDomainState() is false. The ID-only race existed in a narrow post-unlock interval at BASE; this PR expands it across potentially slow close/join operations.
Preserve handler/session ownership when cancelling forwarding obligations, or retire the old obligation atomically before replacement registration becomes possible. Pin: hold old teardown, connect a same-ID replacement, dispatch a fresh offline announcement with its send window closed, then release only the old teardown; DSRSShutdownSync.canShutdown(baseDN) must remain false until the replacement forwards or the real grace period expires. Independently verified by source tracing; not dynamically reproduced.
suggestion (non-blocking): Scale certificate allowances do not establish recovery of the affected DS-to-RS path.
.github/scripts/replication/assert-converged.sh:358, .github/scripts/replication/selftest-replication-errors.sh:207 — F-9571107b938a5f38cb07.
DS119 and RS105 warnings are accepted from their in-window node/endpoint match without a later successful connection. The existing fixture explicitly accepts a warning alone:
case_ new-cert-exact 0 1 '[ cert:n0:cv-r1' "$WBAD_RS" ']'Scale membership and convergence can succeed through another RS; unlike disturb-empty-rs.sh, the scale rows do not force every DS onto the returned RS afterward. Pin: keep the warning as an error without a later same-DS/domain connection to that RS, and reject recovery records for another RS/domain. No persistent TLS defect or failing CI run is claimed.
suggestion (non-blocking): The new direct-forwarding deadline has no expiration regression across domains.
opendj-server-legacy/src/test/java/org/opends/server/replication/server/ReplicationServerShutdownSyncTest.java:552, :2520 — F-876871837fb39d9f4960.
The direct-update test releases the slow peer before asserting completion:
slow.openSendWindow(10);
stopped.get(SOCKET_TIMEOUT_MS, TimeUnit.MILLISECONDS);The existing two-domain elapsed-time test arms DSRSShutdownSync without direct updates, so it exercises the separate offline-message grace period. Pin: populate direct-update targets in two domains, hold delivery, pass one short absolute deadline to both drains, and assert combined elapsed time plus timeout warnings; also protect the production caller's single deadline creation. The current production code passes the same deadline correctly by inspection.
suggestion (non-blocking): Worker tests do not cover LDAP entries followed by a failing final result.
opendj-ldap-toolkit/src/test/java/com/forgerock/opendj/ldap/tools/BootstrapSearchTestCase.java:38 — F-825a39aabf856b56f997.
The Java test only sends argument-parser/help requests, while the shell self-test substitutes a synthetic worker:
String input = request("--invalid-option") + request("--help");Pin: use a controlled LDAP endpoint returning an entry followed by sizeLimitExceeded (4); compare output/status with ordinary ldapsearch, assert no fallback, and check a subsequent successful request in the same worker contains only its own entries. Source inspection supports the current output/status implementation; no current loss or false-success defect is claimed.
suggestion (non-blocking): Launcher tests do not cover a failed waiter while the server child remains alive.
.github/scripts/selftest-start-ds.sh:38 — F-01fcfa7cdf40cb627857.
The waiter stub always waits for child termination and succeeds:
while kill -0 "$(cat "$INSTANCE_ROOT/logs/server.pid")" 2>/dev/null; do sleep 0.01; done
exit 0Pin: return waiter status 2 while a controlled child remains alive; assert bounded launcher completion, exit 2, WaitForFileDelete diagnostics with child status running, and no final checkStartability probe. Clean up the child afterward. The current implementation preserves this behavior by inspection; no runtime hang or status defect is established.
…e a DS is offline
…recovery Track peer history across RS registration and generation changes, and replay a missing prefix before a later direct update can advance the outgoing cursor. Ignore socket-read failures caused by a local session close while retaining unexpected peer-loss errors. Add ordering, registration, generation-reset and socket-close regressions. Allow 900 seconds for retained combined-node convergence based on the observed CI backlog drain. Preserve the 2700-second workload ceiling and strict data, history and throughput checks.
7451e3f to
6ca5195
Compare
|
@maximthomas I addressed all five points in
The targeted JDK 21 reactor passed 162 tests. All 27 replication self-tests and the launcher self-test passed. The full Docker convergence matrix has yet to run on this head. |
During replication recovery, later updates can overtake missing history, a dependency scan can hand one update to two replay threads, and a closing session can delay failover handshakes on a live replication server. This PR preserves ordered history and replay ownership, drains accepted writes during shutdown without blocking unrelated runtime handshakes, and exercises scale-in/scale-out under concurrent writes. Docker fixtures require explicit bootstrap success before resuming rolled writers, retain failure diagnostics and run OpenDJ without the former 512 MiB Docker cap.
Part of #1208; also exercises the empty-RS certificate and failover path tracked in #1188.
Scale scenarios and checks
rs534 / scale-rs-keptrs534 / scale-rs-emptycombined / scale-keptcombined / scale-emptyEach scenario removes one server from
REPLICATION_PEERS, rolls the survivors with the shorter list, verifies its removal from their registries and local RS lists, returns it, and rolls the survivors with the full list. Separate-server transitions roll two DS and one RS together, leaving another RS and a writer available. Combined nodes roll individually. A restarted separate-server group must finish the RS and both DS bootstraps before either writer resumes or the next group rolls; a combined node must finish its own bootstrap before resuming. Surviving writers remain active throughout these waits. Missing or failed bootstrap, failed load workers and workload deadlines still abort the scenario. Checkpoints precede stops, and writes on a surviving DS are verified while the group is stopped.While the removed server is absent, a dedicated client applies 50,001 distinct modifications. Every surviving RS must contain the complete CSN interval before the server returns. After scale-out and membership verification, every DS must complete a newly started load segment before the workload ends. Checks cover indexed LDAP results, matching offline LDIF exports, index verification, changelog continuity, RS mesh connectivity and replication errors. The existing
rs534 / empty-rsrow also tests failover in both directions after rebuilding an RS while a DS is offline.Scale load has a 2700-second ceiling and starts the disturbance at 120 seconds. Targets remain 100 modifications and 20 adds/deletes per second per DS, with error-free clients and a duration-weighted minimum of 10% of each target. Fifty-second segments rotate through three indexed surnames per DS. Other rows retain their 900-second load. The scale job and scenario step retain 75- and 70-minute limits respectively. Convergence budgets are 900 seconds for
combined / scale-kept, 600 seconds forcombined / scale-empty, and 300 seconds for both separate-server scale rows. Convergence still requires three complete zero-lag samples spanning more than the 60-second monitor period plus five seconds.Host-side checkpoint and membership queries use a lightweight LDAP reader through loopback-only admin ports, with explicit deadlines and complete-result checks. Artifacts retain checkpoints, full backlog coverage, pre-restart changelogs, offline-state snapshots, bootstrap timings, prior container logs, CSN traces and post-load convergence samples. Failed workers abort pending waits; failed snapshots or Docker commands cannot count as successful progress.
Replication correctness and bootstrap
getNextUpdate()already waits for all earlier pending changes, so repeated parking is unnecessary and can race with another replay thread taking ownership. Parking requires the same pending-change object and current replay owner; a stale scan after a clear/re-delivery or ownership transfer cannot hand out the CSN again. This prevents the reproduced double-handoff path ending in error9when the original replay commits.offline.statesnapshots alone remain diagnostic; incomplete traces and sequences reused by local operations cannot excuse a gap.Session.close()and reader/writer joins: closing drains queued socket writes and can block on a slow peer, while a reader applying its final topology message can need the same domain lock. Neither wait may delay another peer's failover handshake. Reacquire the lock before generation and topology decisions. Topology messages from removed or replaced handlers are ignored; replacement monitors cannot be deleted by late cleanup of the old handler. Receipt cleanup remains after session closure and joins, and the full-server accepted-write delivery barrier remains in place. Offline-message dispatch, socket-write confirmation, and teardown now identify the recipient by a per-connection token. The token does not retain the handler or session. The shutdown continues to wait for a replacement with the same server ID after the old connection finishes closing.DeleteMsgwhen its assured sender expects an acknowledgement.SocketExceptionafter a localSession.close()but before the handler is marked shutting down. The reader now checks the session's close state, consistently with its EOF path and the writer. A genuine remote TCP reset still reports error181; unexpected EOF from modern peers also remains an error. The harness does not broadly suppress shutdown disconnects.rs534 / scale-rs-emptysetsREPLICATION_RECOVER_FROM_RS=1on the returning RS. After scale-in has verified endpoint removal from every DS, bootstrap selects only ready, trusted RS peers and enablescn=admin data, leaving the new endpoint out of existing DS data-domain lists. Missing or unreadable sources are retried without falling back to a DS; source role and state are checked again under the join lock. A retained-history checkpoint must pass before scale-out DS restarts expose the endpoint. Final offline checks still compare every CSN. The mode is off by default, preserving ordinary bootstrap peer order and endpoint advertisement.readyremains a membership signal; the caller must enforce the history barrier.ldapsearch. Genuine LDAP errors retain their status. Existing batching ofdsconfigchanges and immediate one-use membership snapshots remain in place; topology data is not cached across mutations.--doNotStart, let its JVM exit, then start the server explicitly. This removes the setup/server/waiter memory overlap reproduced as an OOM kill at 512 MiB without swap. Preserve the final launcher probe output/status, an already-exited child status, container state and cgroup memory/OOM counters when startup fails..bootstrap-finishedon every start, including retained-container restarts, and creates it only after successful setup/upgrade and the requested background join. Both test harnesses require this positive result and give.bootstrap-failedpriority. An absent join process during setup no longer counts as completion. Health remains separate so retained volumes preserve OrderedReady behavior while membership repair runs.docker cp, including from stopped containers. It avoids command-line/password-file dumps. Replication fixtures enable bootstrap timing and upload diagnostics in artifacts named by image flavor and workflow attempt.docker runsites in the replication fixture and Docker workflow, including failed-bootstrap and upgrade cases, and remove the two unused memory variables. OpenDJ test containers now use the memory available on the runner without a replacement Docker cap. The persistent search worker and all correctness checks remain enabled; removing the Docker cap does not change the JVM arguments or workload ceiling. The small non-Java shared-memory holder retains its separate 64 MiB cap. Scale scenarios already ran without the 512 MiB Docker cap.cn=admin datatimeouts. During a separate-server roll, a surviving DS disconnect is accepted only for the stopped RS address and after a later successful connection to a verified eligible RS in the same domain and rolling window. The returned RS becomes eligible only during scale-out; accepted intervening timeouts must name the successful destination. Scale certificate warnings119and105also require recovery evidence for the returned RS. For119, the same DS ID and domain must connect to that RS after the warning and by the end of scale-out. For105, the pre-roll DS IP and a DS-side warning in the same logged second identify candidate paths, each of which must recover in a later second. The allowance applies to the exact warning line inside the original return window. RS certificate warning312requires a later313for the same RS ID, peer and domain inside the explicit replacement window. Unrecovered warnings and shutdown error181remain failures. The longer convergence budgets apply only to the two combined scale variants; the stable-zero criteria are unchanged.CSN tracing and bootstrap timing are opt-in. The four scale rows and
rs534 / empty-rsenable allocation traces; scale rows also enable bootstrap timing. Missing or unclosed traces fail validation. Failure collection first preserves the live state, then normally stops DS processes in parallel to close their traces;KEEP=1leaves the containers running.Validation
Review follow-up at
6ca519583asizeLimitExceeded(4) and verified output, status and the next worker request. The shell fixture also compares partial LDIF byte for byte and verifies that the worker result does not trigger a fallback.2, diagnostics with a still-running child, and that the final startability probe is skipped.git diff --checkpassed.Previous validation at
7451e3f79fBuild 43994616243, attempt 1, on head
7451e3f79f, passed all 33 jobs. All nine Maven jobs, both Docker replication/upgrade fixture jobs, and all 13 convergence scenarios passed. CodeQL also passed on its first attempt. The scale scenarios completed the workload and online/offline checks within their configured limits:rs534 / scale-rs-keptrs534 / scale-rs-emptycombined / scale-keptcombined / scale-emptyThe previously failing
rs534 / empty-rsscenario also passed. The Trivy check is neutral: GitHub could not compare alerts fortrivy-image-alpineandtrivy-image-latestbecause those configurations frommasterwere not found on the PR.The completed Build 43914906507, attempt 1, at preceding head
1cb60c42f9, finished with 29 successful jobs and four failures. Both complete Docker replication/upgrade fixture jobs and all nine Maven jobs passed. Head7451e3f79faddressed the four failing rows:1cb60c42f9combined / scale-empty181did not recur in this run.combined / scale-kept9failed the rowrs534 / scale-rs-keptrs534 / scale-rs-emptyrs534 / empty-rsLocal verification at
7451e3f79f:cn=admin data, with the original error text and line retained as evidence. Shell syntax, copyright checks andgit diff --checkpass.The exact CI thread interleavings were not captured; the regressions reproduce concrete ownership and handshake-blocking mechanisms consistent with the failures. Bootstrap overlap is established by the saved timing logs; the new scheduling completed all four scale workloads within 2700 seconds in the CI run above. No heavy local OpenDJ/Docker load run was repeated on the busy host.
The 2700-second scale workload ceiling, 900/600/300-second convergence budgets, throughput floors and strict data/index/history/error checks remain unchanged. Longer bootstrap waits fail within those bounds rather than bypassing them.