PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby#2566
Open
tkhurana wants to merge 4 commits into
Open
PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby#2566tkhurana wants to merge 4 commits into
tkhurana wants to merge 4 commits into
Conversation
…ndby Index tables no longer generate their own replication-log records. Instead the active ships only the data-table record plus a per-(row, ts) PRE_IMAGE (the prior row state it observed at lock time), and the standby regenerates every index entry from that record on the replay path. This removes the out-of-order replay hazard: because each replicated batch carries the exact prior-row snapshot it was built against, the standby derives each (row, ts) group's next state from its own shipped pre-image instead of a region scan, so shard files can be replayed in any order (or in parallel) and each group is an independent reproduction of the active preBatchMutate. It also fixes CDC/global/local index rowkeys that embed the encoded data-table region name (PARTITION_ID), which differs between clusters -- the standby rebuilds the rowkey with its own region name. Main changes: - Cell-oriented replication log format (codec + record + MutationCellGrouper) carrying data cells plus a METAFAMILY pre-image cell per row. - IndexRegionObserver: active-side pre-image capture; a forked standby replay path (prepareReplicatedIndexMutations) that groups by (row, ts) and builds index updates from the shipped pre-image. - PreImageLocalTable: a LocalHBaseState that serves the local-index builder's prior-row-state from the shipped pre-image instead of a region scan; new IndexBuildManager.getIndexUpdates overload accepts it. - ReplicationLogProcessor: replay a record as one indivisible batch; do not retry DoNotRetryIOException. Coverage: global, local, uncovered, atomic, conditional-TTL, CDC (plain and behind an EVENTUAL secondary index), concurrent/out-of-order replay, and unit tests for (row, ts) grouping isolation and Put/Delete decomposition.
Thread dataTableName into the standby pre-image decode/lookup so a contract violation names the offending table. decodePreImage and buildReplicatedRowGroups become instance methods reading the dataTableName field instead of taking it as a parameter; add a @VisibleForTesting IndexRegionObserver(String) constructor alongside the public no-arg one HBase loads reflectively. PreImageLocalTable takes the table name in its constructor. Doc notes: decodePreImage javadoc explains the schema-skew case as the one legitimate missing-pre-image path and why failing loud is correct; PreImageLocalTable and MutationCellGrouper get class-level thread-safety notes.
preBatchMutateWithExceptions interleaved the active write path and the standby replay path in one method behind five scattered isReplication guards, so the standby path was implicit (defined by what active-only blocks skip). That fragility surfaced a real bug: getCurrentRowStates was gated only on index flags, not isReplication, so the standby scanned the data-table region under lock for prior row state -- the out-of-order-unsafe read the per-row PRE_IMAGE exists to replace -- and then discarded the result. - Skip getCurrentRowStates on replication: prepareReplicatedIndexMutations reads only the group's PRE_IMAGE/nextState, never dataRowStates, so the scan and its pendingRows/lastConcurrentBatchContext bookkeeping are inert on the standby. - Dispatch to a new preBatchMutateReplication right after currentPhase = PRE and return, so the active body runs only on the active. Drop the five now-dead isReplication guards there. - Extract the shared two-phase commit (preparePreIndexMutations -> unlock -> doPre -> lock -> [wait] -> post) into prepareAndCommitGlobalIndexUpdates so the lockstep protocol stays single-sourced; the active/standby fork lives inside preparePreIndexMutations. The concurrent-batch wait is now reached only when lastConcurrentBatchContext is set, which never happens on the standby. Verified: IndexRegionObserverReplayTest 16/16, ReplicationLogGroupIT 15/15 (incl. testConcurrentUpserts, concurrent same-row standby replay).
… cell filter Both replication paths now drop local-index (L#) cells the same way. The WAL-restore path (replicateEditOnWALRestore) previously shipped the persisted WAL edit verbatim, leaking the L# cells HBase merged into the data mutation's family map -- a recovery-path-only leak no IT exercised. It now filters the cell stream through the shared isReplicableCell predicate. The synchronous path (replicateMutations) reads the batch's now-final cells from miniBatchOp in POST and drops L# cells with a single family-key check per family list, then appends the pre-image cells from WAL slot 0 (filtered through isReplicableCell). capturePreImageCells is now the sole producer of pre-image cells, written only to the WAL edit and read back by both paths, so they ship byte-identical output with no re-derivation. Removes the timing-based snapshot machinery (captureReplicationCells and the replicationCellsByRow field) that the sync path used to exclude L# cells, since the WAL-restore path could not use it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Index tables no longer generate their own replication-log records. The active ships only the data-table record plus a per-
(row, ts)PRE_IMAGE (the prior row state observed at lock time), and the standby regenerates every index entry from that record on the replay path.This targets the
PHOENIX-7562-feature-newfeature branch (builds on the per-batch coalescing / cell-oriented log format, #2540).Why
(row, ts)group's next state from its own shipped pre-image instead of a region scan. Shard files can be replayed in any order or in parallel; each group is an independent reproduction of the activepreBatchMutate.PARTITION_ID), which differs between clusters. Replicating index cells verbatim would carry the active's region name; the standby instead rebuilds the rowkey with its own.Main changes
MutationCellGrouper) carrying data cells plus one METAFAMILY pre-image cell per row.IndexRegionObserver: active-side pre-image capture; a forked standby replay path (prepareReplicatedIndexMutations) that groups by(row, ts)and builds index updates from the shipped pre-image.PreImageLocalTable: aLocalHBaseStateserving the local-index builder's prior-row-state from the shipped pre-image instead of a region scan; newIndexBuildManager.getIndexUpdatesoverload accepts it.ReplicationLogProcessor: replay a record as one indivisible batch; do not retryDoNotRetryIOException.Test coverage
Global, local, uncovered, atomic, conditional-TTL, and CDC (plain and behind an EVENTUAL secondary index) index regeneration; concurrent/out-of-order replay for both global and local indexes; unit tests for
(row, ts)grouping isolation and Put/Delete decomposition.Notes