Make agent index generation stable irrespective of filesystem iteration order by bric3 · Pull Request #11889 · DataDog/dd-trace-java

bric3 · 2026-07-08T16:05:20Z

What Does This Do

Refactors AgentJarIndex.IndexGenerator to ensure the index file generation is immune to the unspecified order of filesystem traversal. This resolves inconsistencies in the index file created due to the dependency on traversal order of Files.walkFileTree.

Motivation

The previous implementation relied on the order in which files were traversed by Files.walkFileTree, which is inherently unstable (order-wsie) and filesystem-dependent. This led to varying index files, especially in CI environments with parallel Gradle tasks. Although the generated index was correct, the varying prefix IDs impacted reproducibility and the downstream shadowJar.

Additional Notes

This happens because Files.walkFileTree rely under the hood on UnixFileSystemProvider.newDirectoryStream which uses readdir(3) std lib C api which asks the file system via syscall, in particular we can read (bold are mine):

The order in which filenames are read by successive calls to
readdir() depends on the filesystem implementation; it is unlikely
that the names will be sorted in any fashion.

https://www.man7.org/linux/man-pages/man3/readdir.3.html
https://man7.org/linux/man-pages/man2/getdents.2.html

Note

generateAgentJarIndex is still re-executed because it relies on included jars from it.inputs.files(includedJarFileTree), the actual folder being:
workspace/dd-java-agent/build/generated/included

dd-trace-java/dd-java-agent/build.gradle

Lines 342 to 365 in fc7d48c

    
           def generateAgentJarIndex = tasks.register('generateAgentJarIndex', JavaExec) { 
        
             def destinationDir = project.layout.buildDirectory.dir("generated/${it.name}") 
        
             it.group = LifecycleBasePlugin.BUILD_GROUP 
        
             it.description = "Generate dd-java-agent.index" 
        
             it.mainClass = 'datadog.trace.bootstrap.AgentJarIndex$IndexGenerator' 
        
             it.inputs.files(includedJarFileTree) 
        
             it.inputs.files(it.classpath) 
        
             it.outputs.dir(destinationDir) 
        
             it.classpath = objects.fileCollection().tap { 
        
               it.from(project.configurations.named("shadowInclude")) 
        
               it.from(project.configurations.named('slf4j-simple')) 
        
             } 
        
             // debuggable within gradle using: 
        
             // it.jvmArgs("-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005") 
        
             it.argumentProviders.add(new CommandLineArgumentProvider() { 
        
                 @Override 
        
                 Iterable<String> asArguments() { 
        
                   return [includedAgentDir.get().asFile.path, destinationDir.get().asFile.path,] 
        
                 } 
        
               }) 
        
           } 
        
           sourceSets.main.resources.srcDir(generateAgentJarIndex)

However, the rehydrated pipeline cache from gitlab only capture folders in the .gradle folder (unerstandably)

- key: $CI_PIPELINE_ID-$CACHE_TYPE
    paths:
    - .gradle/caches/$GRADLE_VERSION
    - .gradle/$GRADLE_VERSION/executionHistory
    - workspace

The expandAgentShadowJar* tasks filling up includedAgentDir are Copy like tasks and as such are not cached as well, which made the task not up-to-date.
So tHere is another update coming to that task on the build file, PR not yet opened.

See Gradle build cache points (1, 2)

:processResources and :jar being Copy-like tasks which are not cacheable since it is generally faster to execute them.

Some tasks, like Copy or Jar, usually do not make sense to make cacheable because Gradle is only copying files from one location to another. It also doesn’t make sense to make tasks cacheable that do not produce outputs or have no task actions.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Add your completed PR to the merge queue by commenting /merge. You can also:
- Customize the commit message associated with the merge with /merge --commit-message "..."
- Remove your PR from the merge queue with /merge -c
- Skip all merge queue checks with /merge -f --reason "reason"; please use this judiciously, as some checks do not run at the PR-level (note: the PR still needs to be mergeable, this will only skip the pre-merge build)
- Get more information in this doc

Jira ticket: [PROJ-IDENT]

… iteration `AgentJarIndex.IndexGenerator` used `Files.walkFileTree` traversal order as stable input. But directory entry order is explicitly unspecified, and in CI, the folder creation in `included/` depends on when the parallel Gradle task are finished. This causes a different index file to be generated, and invalidates the input of the `shadowJar`. Note the index remains correct, however this behavior changes the prefix IDs in the index file. This happens because `Files.walkFileTree` rely under the hood on `UnixFileSystemProvider.newDirectoryStream` which uses `readdir(3)` C api which asks the file system via syscall, in particular we can read: > The order in which filenames are read by successive calls to > readdir() depends on the filesystem implementation; it is unlikely > that the names will be sorted in any fashion. https://www.man7.org/linux/man-pages/man3/readdir.3.html https://man7.org/linux/man-pages/man2/getdents.2.html The tool `find` has the same behavior. https://sources.debian.org/src/findutils/4.10.0-3/gl/lib/fts.c/#L1444

PerfectSlayer

Looking good and left minor comments

PerfectSlayer · 2026-07-09T08:09:52Z

+  private static Path writeIndex(Path resourcesDir, Path indexFile) throws IOException {
+    AgentJarIndex.IndexGenerator generator = new AgentJarIndex.IndexGenerator(resourcesDir);
+    generator.buildIndex();
+    generator.writeIndex(indexFile);
+    return indexFile;
+  }
+
+  private static List<String> readPrefixes(Path indexFile) throws IOException {
+    try (DataInputStream in =
+        new DataInputStream(new BufferedInputStream(Files.newInputStream(indexFile)))) {
+      int prefixCount = in.readInt();
+      String[] prefixes = new String[prefixCount];
+      for (int i = 0; i < prefixCount; i++) {
+        prefixes[i] = in.readUTF();
+      }
+      return Arrays.asList(prefixes);
+    }
+  }


🎯 suggestion: ‏What about reusing buildAndReadIndex instead?

I introduced writeIndex (returns a Path) / readPrefixes (returns a List) because these tests assert the serialized index, not only the lookup behavior after deserialization which is the goal of buildAndReadIndex (returns a AgentJarIndex) to me.
In particular for prefix ordering readPrefixes is what the test needs.

…variant

Originally, normalization was introduced so that the index would be same across OS, but it's not really improtant for windows.

dd-octo-sts · 2026-07-09T09:32:30Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.02 s	13.98 s	[-0.6%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	12.93 s	13.04 s	[-1.6%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	16.83 s	16.75 s	[-0.7%; +1.7%] (no difference)
startup:petclinic:iast:Agent	16.44 s	16.96 s	[-7.5%; +1.4%] (no difference)
startup:petclinic:profiling:Agent	16.46 s	16.84 s	[-3.2%; -1.3%] (significantly better)
startup:petclinic:sca:Agent	16.89 s	16.81 s	[-0.5%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	15.81 s	15.68 s	[-5.1%; +6.7%] (unstable)

Commit: 794f5f29 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

bric3 · 2026-07-09T09:54:41Z

By the way the reproducer, it leverages the Files.walkFileTree and we can see in docker (linux) that the iteration order is indeed different when the folders are created in a different order, we can see that find as the same behavior, which is consistent with the one from Java (since they use same libc API) :

WalkFileTreeOrder

import java.io.IOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Stream;

public class WalkFileTreeOrder {
  public static void main(String[] args) throws IOException {
    if (args.length != 2 || (!"--raw".equals(args[0]) && !"--sorted".equals(args[0]))) {
      System.err.println("Usage: java WalkFileTreeOrder.java (--raw|--sorted) <directory>");
      System.exit(2);
    }

    Path root = Path.of(args[1]).toAbsolutePath().normalize();
    List<String> folders =
        "--raw".equals(args[0]) ? rawWalkFileTreeOrder(root) : sortedFilesWalkOrder(root);

    folders.forEach(System.out::println);
  }

  private static List<String> rawWalkFileTreeOrder(Path root) throws IOException {
    List<String> folders = new ArrayList<>();
    Files.walkFileTree(
        root,
        new SimpleFileVisitor<>() {
          @Override
          public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
            Path normalized = dir.toAbsolutePath().normalize();
            if (!normalized.equals(root) && root.equals(normalized.getParent())) {
              folders.add(normalized.getFileName().toString());
              return FileVisitResult.SKIP_SUBTREE;
            }
            return FileVisitResult.CONTINUE;
          }
        });
    return List.copyOf(folders);
  }

  private static List<String> sortedFilesWalkOrder(Path root) throws IOException {
    try (Stream<Path> paths = Files.walk(root)) {
      return paths
          .filter(path -> Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS))
          .map(root::relativize)
          .filter(path -> path.getNameCount() >= 2)
          .sorted(Comparator.comparing(WalkFileTreeOrder::normalizedPath))
          .map(path -> path.getName(0).toString())
          .distinct()
          .toList();
    }
  }

  private static String normalizedPath(Path path) {
    return path.toString().replace(path.getFileSystem().getSeparator(), "/");
  }
}

the script that runs things

#!/usr/bin/env bash
set -euo pipefail

script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
image="${IMAGE:-gradle:9-jdk25}"
artifacts_dir="${ARTIFACTS_DIR:-$(mktemp -d "${TMPDIR:-/tmp}/walk-file-tree-order.XXXXXX")}"
container_user="$(id -u):$(id -g)"
workdir="/tmp/walk-file-tree-order"
java_file="/src/WalkFileTreeOrder.java"

docker_run() {
  docker run --rm \
    --user "$container_user" \
    --mount "type=bind,source=$script_dir,target=/src,readonly" \
    --mount "type=bind,source=$artifacts_dir,target=/artifacts" \
    "$image" "$@"
}

create_tree_and_walk() {
  local label="$1"
  local names="$2"

  printf '\n==> %s container\n' "$label"
  docker_run bash -lc "
    set -euo pipefail
    root='$workdir'
    target=\"\$root/$label\"
    rm -rf \"\$target\"
    mkdir -p \"\$target\"
    : > \"/artifacts/$label-created.txt\"

    for name in $names; do
      mkdir -p \"\$target/\$name/datadog\"
      printf 'created by shell\n' > \"\$target/\$name/datadog/Marker.classdata\"
      printf '%s\n' \"\$name\" >> \"/artifacts/$label-created.txt\"
    done

    echo 'created folders, in shell order:'
    cat \"/artifacts/$label-created.txt\"

    echo
    echo 'Files.walkFileTree raw order:'
    java '$java_file' --raw \"\$target\" | tee \"/artifacts/$label-walk-file-tree-raw.txt\"

    echo
    echo 'Files.walk sorted regular-file order (suggested IndexGenerator fix):'
    java '$java_file' --sorted \"\$target\" | tee \"/artifacts/$label-walk-file-tree-sorted.txt\"

    echo
    echo 'find order:'
    find \"\$target\" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
      | tee \"/artifacts/$label-find.txt\"
  "
}


command -v docker >/dev/null 2>&1 || {
  echo "error: docker is required" >&2
  exit 1
}

printf 'Docker image:  %s\n' "$image"
printf 'Artifacts:     %s\n' "$artifacts_dir"

build_creation_order="aiguard appsec ci-visibility cws-tls debugger feature-flagging iast llm-obs jmxfetch logs-intake metrics profiling shared trace inst"
create_tree_and_walk build-like "$build_creation_order"

deploy_creation_order="aiguard cws-tls ci-visibility appsec iast feature-flagging debugger jmxfetch llm-obs logs-intake metrics profiling shared trace inst"
create_tree_and_walk deploy-like "$deploy_creation_order"

printf '\n==> diffs\n'
(
  set -euo pipefail
  cd "$artifacts_dir"

  echo "Files.walkFileTree raw diff:"
  raw_walk_status=0
  diff -u build-like-walk-file-tree-raw.txt deploy-like-walk-file-tree-raw.txt || raw_walk_status=$?

  echo
  echo "find diff:"
  find_status=0
  diff -u build-like-find.txt deploy-like-find.txt || find_status=$?

  echo
  echo "Files.walk sorted regular-file diff:"
  sorted_walk_status=0
  diff -u build-like-walk-file-tree-sorted.txt deploy-like-walk-file-tree-sorted.txt || sorted_walk_status=$?

  echo
  sort build-like-walk-file-tree-raw.txt > build-like.sorted
  sort deploy-like-walk-file-tree-raw.txt > deploy-like.sorted
  if cmp -s build-like.sorted deploy-like.sorted; then
    echo "same directory names: true"
  else
    echo "same directory names: false"
  fi

  if [[ "$raw_walk_status" == 0 ]]; then
    echo "Files.walkFileTree raw order: true"
  else
    echo "Files.walkFileTree raw order: false"
  fi

  if [[ "$find_status" == 0 ]]; then
    echo "find order: true"
  else
    echo "find order: false"
  fi

  if [[ "$sorted_walk_status" == 0 ]]; then
    echo "Files.walk sorted regular-file order: true"
  else
    echo "Files.walk sorted regular-file order: false"
  fi
)

output

Docker image:  gradle:9-jdk25
Artifacts:     /var/folders/5_/gltgq4390wx07yvypgl1mc400000gq/T//walk-file-tree-order.GIdBOQ

==> build-like container
created folders, in shell order:
aiguard
appsec
ci-visibility
cws-tls
debugger
feature-flagging
iast
llm-obs
jmxfetch
logs-intake
metrics
profiling
shared
trace
inst

Files.walkFileTree raw order:
aiguard
appsec
ci-visibility
cws-tls
debugger
feature-flagging
iast
llm-obs
jmxfetch
logs-intake
metrics
profiling
shared
trace
inst

Files.walk sorted regular-file order (suggested IndexGenerator fix):
aiguard
appsec
ci-visibility
cws-tls
debugger
feature-flagging
iast
inst
jmxfetch
llm-obs
logs-intake
metrics
profiling
shared
trace

find order:
aiguard
appsec
ci-visibility
cws-tls
debugger
feature-flagging
iast
llm-obs
jmxfetch
logs-intake
metrics
profiling
shared
trace
inst

==> deploy-like container
created folders, in shell order:
aiguard
cws-tls
ci-visibility
appsec
iast
feature-flagging
debugger
jmxfetch
llm-obs
logs-intake
metrics
profiling
shared
trace
inst

Files.walkFileTree raw order:
aiguard
cws-tls
ci-visibility
appsec
iast
feature-flagging
debugger
jmxfetch
llm-obs
logs-intake
metrics
profiling
shared
trace
inst

Files.walk sorted regular-file order (suggested IndexGenerator fix):
aiguard
appsec
ci-visibility
cws-tls
debugger
feature-flagging
iast
inst
jmxfetch
llm-obs
logs-intake
metrics
profiling
shared
trace

find order:
aiguard
cws-tls
ci-visibility
appsec
iast
feature-flagging
debugger
jmxfetch
llm-obs
logs-intake
metrics
profiling
shared
trace
inst

==> diffs
Files.walkFileTree raw diff:
--- build-like-walk-file-tree-raw.txt   2026-07-08 15:07:39.025320248 +0200
+++ deploy-like-walk-file-tree-raw.txt  2026-07-08 15:07:40.277103884 +0200
@@ -1,12 +1,12 @@
 aiguard
-appsec
-ci-visibility
 cws-tls
-debugger
-feature-flagging
+ci-visibility
+appsec
 iast
-llm-obs
+feature-flagging
+debugger
 jmxfetch
+llm-obs
 logs-intake
 metrics
 profiling

Files.walk sorted regular-file diff:

find diff:
--- build-like-find.txt 2026-07-08 15:07:39.446845122 +0200
+++ deploy-like-find.txt        2026-07-08 15:07:40.659516870 +0200
@@ -1,12 +1,12 @@
 aiguard
-appsec
-ci-visibility
 cws-tls
-debugger
-feature-flagging
+ci-visibility
+appsec
 iast
-llm-obs
+feature-flagging
+debugger
 jmxfetch
+llm-obs
 logs-intake
 metrics
 profiling

same directory names: true
same Files.walkFileTree raw order: false
same Files.walk sorted regular-file order: true
same find order: false

bric3 requested a review from a team as a code owner July 8, 2026 16:05

bric3 requested review from PerfectSlayer, dougqh and mcculls July 8, 2026 16:05

bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jul 8, 2026

dougqh approved these changes Jul 8, 2026

View reviewed changes

PerfectSlayer reviewed Jul 9, 2026

View reviewed changes

bric3 added 2 commits July 9, 2026 10:40

test(agent): Replace awkward stream and for loop, with a pure stream …

5d8ef78

…variant

refactor(agent): Remove normalization for windows

51b20c7

Originally, normalization was introduced so that the index would be same across OS, but it's not really improtant for windows.

bric3 force-pushed the bdu/non-determistic-index-file-walking branch from 72bab0b to 51b20c7 Compare July 9, 2026 08:54

This comment has been minimized.

Sign in to view

refactor(agent): Remove unused Comparator import

794f5f2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make agent index generation stable irrespective of filesystem iteration order#11889

Make agent index generation stable irrespective of filesystem iteration order#11889
bric3 wants to merge 4 commits into
masterfrom
bdu/non-determistic-index-file-walking

bric3 commented Jul 8, 2026 •

edited

Loading

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer Jul 9, 2026

Uh oh!

bric3 Jul 9, 2026

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jul 9, 2026

Uh oh!

bric3 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	def generateAgentJarIndex = tasks.register('generateAgentJarIndex', JavaExec) {
	def destinationDir = project.layout.buildDirectory.dir("generated/${it.name}")

	it.group = LifecycleBasePlugin.BUILD_GROUP
	it.description = "Generate dd-java-agent.index"
	it.mainClass = 'datadog.trace.bootstrap.AgentJarIndex$IndexGenerator'

	it.inputs.files(includedJarFileTree)
	it.inputs.files(it.classpath)
	it.outputs.dir(destinationDir)
	it.classpath = objects.fileCollection().tap {
	it.from(project.configurations.named("shadowInclude"))
	it.from(project.configurations.named('slf4j-simple'))
	}
	// debuggable within gradle using:
	// it.jvmArgs("-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005")
	it.argumentProviders.add(new CommandLineArgumentProvider() {
	@Override
	Iterable<String> asArguments() {
	return [includedAgentDir.get().asFile.path, destinationDir.get().asFile.path,]
	}
	})
	}
	sourceSets.main.resources.srcDir(generateAgentJarIndex)

Uh oh!

Conversation

bric3 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

bric3 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jul 9, 2026

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

bric3 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bric3 commented Jul 8, 2026 •

edited

Loading