Skip to content

Avoid full repository scans for hotfile replication jobs#8144

Open
greenc-FNAL with Copilot wants to merge 12 commits into
masterfrom
copilot/hotfile-replication-fix-initializing-tasks
Open

Avoid full repository scans for hotfile replication jobs#8144
greenc-FNAL with Copilot wants to merge 12 commits into
masterfrom
copilot/hotfile-replication-fix-initializing-tasks

Conversation

Copilot AI commented Jul 8, 2026

Copy link
Copy Markdown

Hotfile replication jobs were being created through the generic migration startup path, which performs a repository-wide initialization scan before the job can leave INITIALIZING. On large pools, that made single-file hotfile jobs unnecessarily expensive and could leave some of them stuck in that startup state.

  • Hotfile startup path

    • Change hotfile-triggered jobs to start from the already-resolved CacheEntry instead of invoking the generic full-repository initialization path.
    • This keeps hotfile replication scoped to the single file that crossed the threshold.
  • Job initialization API

    • Add a Job.start(Iterable<CacheEntry>) overload for callers that already know the exact entries to enqueue.
    • Preserve the existing asynchronous start() behavior for normal migration jobs that still need a repository scan.
  • Initialization handling

    • Factor the shared initialization/cleanup bookkeeping so both startup paths use the same state-transition rules.
    • Keep listener registration and initial enqueueing aligned under the job lock for the hotfile-specific path.
  • Regression coverage

    • Add a focused test to verify hotfile startup does not iterate the repository and still creates the expected job.

Example of the behavioral change:

// before
job.start();

// after
job.start(Collections.singletonList(cacheEntry));

Copilot AI linked an issue Jul 8, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Fix hotfile replication migration tasks in INITIALIZING state Avoid full repository scans for hotfile replication jobs Jul 8, 2026
Copilot AI requested a review from greenc-FNAL July 8, 2026 15:01
@greenc-FNAL

Copy link
Copy Markdown
Contributor

This PR has been built successfully with tests on a local node; CI in-progress.

@greenc-FNAL greenc-FNAL marked this pull request as ready for review July 8, 2026 16:54
@greenc-FNAL greenc-FNAL requested a review from Copilot July 8, 2026 19:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes hotfile-triggered migration jobs to avoid the generic startup path that scans the entire pool repository, keeping hotfile replication scoped to the specific file that crossed the threshold.

Changes:

  • Switch hotfile-triggered jobs to start from an already-resolved CacheEntry (no full repository scan).
  • Add Job.start(Iterable<CacheEntry>) to support synchronous startup with preselected entries, while keeping the existing async start() behavior for scan-based jobs.
  • Add a regression test asserting hotfile startup does not invoke Repository.iterator().

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
modules/dcache/src/test/java/org/dcache/pool/migration/MigrationModuleTest.java Adds a regression test to ensure hotfile job startup does not iterate the repository.
modules/dcache/src/main/java/org/dcache/pool/migration/MigrationModule.java Uses the new Job.start(Iterable<CacheEntry>) overload for hotfile-triggered jobs.
modules/dcache/src/main/java/org/dcache/pool/migration/Job.java Factors initialization bookkeeping and introduces a synchronous start path that enqueues known entries without scanning.

Comment thread modules/dcache/src/main/java/org/dcache/pool/migration/Job.java
Comment thread modules/dcache/src/main/java/org/dcache/pool/migration/Job.java
Copilot AI and others added 12 commits July 8, 2026 16:32
Motivation:

The javadoc for the synchronous Job.start(Iterable<CacheEntry>) variant
was brief and did not clearly explain the method's purpose, typical
callers, or the synchronous blocking behavior.

Modification:

Expanded the javadoc to clarify that this method bypasses the generic
repository scan, is typically used by hot-file replication, and blocks
the calling thread until all entries are examined and queued. Added
@param and @see tags for better cross-referencing with the asynchronous
start() method.

Result:

Developers reading the code now have a clearer understanding of when and
why to use this synchronous startup variant versus the asynchronous one.

Target: master
Requires-notes: no
Requires-book: no
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hotfile replication leaves some migration tasks in INITIALIZING state

3 participants