CLDSRV-955: Parallelize docker builds into a matrix

[tcarmet]

Speeds up CI by building the container images in parallel instead of one after another, so the test jobs that depend on them start sooner. The release-only images that depend on the freshly built server image and are not needed by any test are moved off the critical path to run alongside the tests. Image names and tags are unchanged. This is the second of two changes tracked under CLDSRV-955 to reduce the tests workflow wall time; the first caches the installed dependency tree.

[bert-e]

Hello tcarmet,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue CLDSRV-955 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.3.14

• 9.4.0

Please check the Fix Version/s of CLDSRV-955, or the target
branch of this pull request.

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
8490	2	8488	0

View the top 1 failed test(s) by shortest run time

should log correct bucketPutLogging operation with all required fields::Server Access Logs - File Output With v4 signature should log correct bucketPutLogging operation with all required fields

Stack Traces | 0.028s run time

Expected 2 log entries, got 3

3 !== 2

View the full list of 2 ❄️ flaky test(s)

"after each" hook for "should fail if trying to overwrite a delete marker"::MPU with x-scal-s3-version-id header With default signature "after each" hook for "should fail if trying to overwrite a delete marker"

Flake rate in main: 100.00% (Passed 0 times, Failed 217 times)

Stack Traces | 0.038s run time

We encountered an internal error. Please try again.

"after each" hook for "should fail if trying to overwrite a delete marker"::MPU with x-scal-s3-version-id header With v4 signature "after each" hook for "should fail if trying to overwrite a delete marker"

Flake rate in main: 100.00% (Passed 0 times, Failed 231 times)

Stack Traces | 0.01s run time

We encountered an internal error. Please try again.

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[tcarmet]

Docker build matrix - CI timing impact

Measured on a warm run of this branch (run 29056334730, warm attempt) vs a recent successful development/9.3 run (28062248343).

Build phase

	development/9.3	this PR	delta
Build phase wall time	623s (one sequential build job)	462s (matrix, max of 4 parallel entries)	−161s (~26%)
Time from run start → first test job starts	630s	470s	−160s

Matrix entry breakdown (warm):

entry	duration
cloudserver (production)	208s
cloudserver-testcoverage	462s ← matrix bottleneck
pykmip	14s
ci-mongodb	13s
build-federation (separate job)	113s - now runs in parallel with the tests, no longer blocking them

Honest caveats

• The win is real (~160s off the pre-test critical path) but smaller than it could be. cloudserver-testcoverage is FROM production; splitting it into its own parallel entry with a separate cache scope means it re-builds the production layers instead of reusing them (in the old sequential job it was only ~14s, right after production). A follow-up that couples production+testcoverage in a single matrix entry would drop the matrix wall to roughly max(~220s, pykmip, mongodb) ≈ ~220s, i.e. a ~400s build-phase reduction rather than ~160s.
• The development/9.3 GHA docker layer cache warmth varies (those runs are ~weekly and the cache evicts after 7 days), so the 623s baseline is not guaranteed fully-warm; treat these as directional.

Note: the functional suite is independently flaky on the shared MPU abort-race teardown (CLDSRV-938), unrelated to this change; this run needed reruns of a couple of single flaky tests to go green. All build/matrix jobs passed on every attempt.

[tcarmet]

Docker build - updated timing (warm, after restructure)

Structure is now a dedicated build-cloudserver job (builds production then testcoverage, sharing the cloudserver cache scope) + a build-images matrix (pykmip, ci-mongodb), with build-federation running off the critical path. Measured on a warm run (29100971369, attempt 2) vs development/9.3 (28062248343, one sequential build job = 623s).

job	duration (warm)	detail
build-cloudserver	510s	production 459s + testcoverage 18s
build-images (pykmip)	16s	parallel
build-images (ci-mongodb)	20s	parallel
build-federation	~113s	now parallel with the tests, off the pre-test path
build-phase wall (gates tests)	~510s	max(build-cloudserver, build-images)

vs development/9.3: 623s (sequential) → ~510s to first test job (~113s / ~18% faster). The gain comes almost entirely from moving build-federation (~113s) and the small pykmip/ci-mongodb builds off the pre-test critical path.

Honest findings

• The production image build dominates (~460s) and does not cache well across runs - it was 208s on one earlier run but 459–509s on every other, warm or cold. The yarn-install + TypeScript-compile builder stage is the cost; its GHA docker-layer cache reuse is inconsistent. This is the real floor on the build phase and is unchanged by this PR.
• Coupling testcoverage into build-cloudserver is a compute win, not a wall-time win. testcoverage now reuses the production layers (18s) instead of rebuilding them (~460s as a standalone parallel entry) - so we no longer pay for a duplicate production build. In wall-clock it adds only ~18s after production (vs overlapping in the old parallel design), which is why the wall is ~510s rather than ~462s; the trade is ~50s of wall time for roughly one full production build's worth of saved CI compute.
• Follow-up worth considering (separate ticket): make the cloudserver production image build cache reliably (e.g. dedicated buildx cache for the yarn/tsc builder layer) - that's where the remaining ~250s would come from.

Flaky note: the functional suite still trips the known MPU abort-race teardown (CLDSRV-938), unrelated to this change.