Handle BucketAlreadyExists races and bump arsenal for MPU afterEach fix by benzekrimaha · Pull Request #6222 · scality/cloudserver

benzekrimaha · 2026-07-08T15:25:52Z

Summary

Fix MPU shadow-bucket creation races in cloudserver callers and bump arsenal to 8.4.20 (ARSN-613).

ARSN-613 makes MongoClientInterface.createBucket() return BucketAlreadyExists on concurrent duplicates instead of silent idempotence. Cloudserver must handle that contract:

freshStartCreateBucket: continue addToUsersBucket / transient cleanup when the metastore entry already exists.
getMPUBucket: on BucketAlreadyExists, fetch and return the existing MPU shadow bucket.

This fixes the intermittent afterEach failure in mpuVersion.js ("We encountered an internal error") when ListMultipartUploads races on MPU shadow bucket creation during test cleanup.

Paired with Arsenal PR #2672 (ARSN-613).

Issue: CLDSRV-919

bert-e · 2026-07-08T15:25:57Z

Hello benzekrimaha,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-07-08T15:26:02Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-919 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.4.0

Please check the Fix Version/s of CLDSRV-919, or the target
branch of this pull request.

bert-e · 2026-07-08T15:34:23Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

bert-e · 2026-07-08T15:36:41Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

bert-e · 2026-07-08T15:38:05Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_pull_requests, create_integration_branches

codecov · 2026-07-08T15:38:59Z

Codecov Report

❌ Patch coverage is 83.15217% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.91%. Comparing base (055ae1b) to head (19a2bd2).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
lib/services.js	77.41%	28 Missing ⚠️
lib/api/apiUtils/bucket/bucketCreation.js	95.00%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/api/apiUtils/bucket/bucketCreation.js	`93.49% <95.00%> (+0.10%)`	⬆️
lib/services.js	`87.10% <77.41%> (-0.33%)`	⬇️

... and 1 file with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.3    #6222      +/-   ##
===================================================
- Coverage            84.93%   84.91%   -0.03%     
===================================================
  Files                  206      206              
  Lines                13366    13374       +8     
===================================================
+ Hits                 11353    11357       +4     
- Misses                2013     2017       +4

Flag	Coverage Δ
file-ft-tests	`68.36% <69.56%> (-0.03%)`	⬇️
file-ft-tests-null-compat	`68.87% <70.10%> (-0.03%)`	⬇️
kmip-ft-tests	`28.28% <36.41%> (-0.01%)`	⬇️
mongo-v0-ft-tests	`69.59% <72.28%> (-0.04%)`	⬇️
mongo-v1-ft-tests	`69.56% <70.65%> (-0.06%)`	⬇️
multiple-backend	`36.75% <33.15%> (-0.02%)`	⬇️
s3c-ft-tests-v0	`64.09% <73.91%> (-0.03%)`	⬇️
s3c-ft-tests-v0-null-compat	`64.15% <73.91%> (-0.03%)`	⬇️
s3c-ft-tests-v1	`64.07% <73.91%> (-0.03%)`	⬇️
sur-tests	`35.81% <51.08%> (-0.83%)`	⬇️
sur-tests-inflights	`37.63% <52.71%> (-0.04%)`	⬇️
unit	`70.52% <78.26%> (+0.01%)`	⬆️
utapi-v2-tests	`34.54% <55.97%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bert-e · 2026-07-09T15:33:27Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

+                    if (err) {
+                        return callback(err);
+                    }
+                    return callback(null, data);
+                });
+            },
        },
-    },
-    // Function to run upon finishing both parallel requests
-    (err, results) => {
-        if (err) {
-            return cb(err);


ARSN-613 makes Mongo createBucket return BucketAlreadyExists instead of silent idempotence. Continue bucket creation follow-up when a concurrent call already created the metastore entry (freshStartCreateBucket), and fetch the existing MPU shadow bucket when getMPUBucket races. Issue: CLDSRV-919

Cover freshStartCreateBucket continuing after a concurrent create and getMPUBucket refetching the existing MPU shadow bucket. Issue: CLDSRV-919

Pull in Arsenal 8.4.20 at 0bd1b04c (ARSN-613), including the usersBucketHack getBucket-first fix paired with this cloudserver change. Issue: CLDSRV-919

The changed-files prettier CI check validates whole files, not just the modified lines. Issue: CLDSRV-919

francoisferrand

nit: this is exactly the kind of PR where prettier is really adding noise... 5-10 lines of "real" change, but prettier fully rewrote 2 large files...

francoisferrand · 2026-07-10T12:38:27Z

+        } else {
+            log.trace('created bucket in metadata');
+        }
        return addToUsersBucket(canonicalID, bucketName, bucket, log, err => {


should not be be done if bucket already exists, to avoid race and/or incorrect content?

I also don't understand why bucket already exists error does not cause the function to return early with an error ?

More generally i don't understand why this wasn't catched before in the err ?

I would've expect current behavior to be :

CreateBucket(bucketName) => return err because collision in the db on bucket name, then return callback(err)

I think I'm missing some context information about : what trigger this bug precisely, cuz the pr talks about MPU but here we are fixing a function called freshStartCreateBucket, is this not called all the time to create any bucket ?

francoisferrand · 2026-07-10T12:47:00Z

                return metadata.createBucket(MPUBucketName, mpuBucket, log,
                    err => {
+                        if (err?.is?.BucketAlreadyExists) {
+                            return metadata.getBucket(MPUBucketName, log,


is this function getMPUBucket() called a lot (it is not an s3 api, not sure if it is called everywhere in mpu code, or just in a few places) : if that is the case, maybe we should plan another improvement to let createBucket return the bucket data as well it case it already exists.

...or maybe we could just return the (existing) mpuBucket variable? for actual buckets, there may be more in the BucketInfo -tags, lifecycle rules...-, but likely not the case for mpu's BucketInfo ?

SylvainSenechal · 2026-07-10T17:41:55Z

     *                        upload bucket or error if any
     * @return {undefined}
     */
    getMPUBucket(destinationBucket, bucketName, log, cb) {


Mhh. I find this very weird :

It's called
GetMpuBucket
It currently does :

GetBucket, if not found, create it.
So ok this is like a "get or create if not found".

But then with the new code changes, this function becomes

GetBucket, if not found, create it. When creating it, if it already exists, getBucket
But if the bucket was not found in the first place, why would we get an "alreadyExists" error when creating it 🤔 ? Race condition ? But then, this is infinite, because in the last getBucket you can also get a notExist error 🤔

benzekrimaha force-pushed the improvement/CLDSRV-919-afterEach-flakiness-internal-error-fix branch 2 times, most recently from ab63d85 to 2cae9bc Compare July 8, 2026 15:31

benzekrimaha changed the base branch from development/9.4 to development/9.3 July 8, 2026 15:31

benzekrimaha changed the title ~~CLDSRV-919: bump arsenal to 8.5.7 for MPU afterEach flake fix~~ Bump arsenal with MPU afterEach flakiness fix Jul 8, 2026

scality deleted a comment from bert-e Jul 8, 2026

benzekrimaha requested review from a team, SylvainSenechal and delthas July 8, 2026 15:37

bert-e mentioned this pull request Jul 8, 2026

INTEGRATION [PR#6222 > development/9.4] Bump arsenal with MPU afterEach flakiness fix #6223

Closed

benzekrimaha requested a review from tcarmet July 8, 2026 15:39

SylvainSenechal approved these changes Jul 8, 2026

View reviewed changes

tcarmet approved these changes Jul 8, 2026

View reviewed changes

delthas approved these changes Jul 9, 2026

View reviewed changes

benzekrimaha force-pushed the improvement/CLDSRV-919-afterEach-flakiness-internal-error-fix branch from 2cae9bc to 6178ad8 Compare July 9, 2026 15:30

benzekrimaha changed the title ~~Bump arsenal with MPU afterEach flakiness fix~~ Handle BucketAlreadyExists races and bump arsenal for MPU afterEach fix Jul 9, 2026

scality deleted a comment from bert-e Jul 9, 2026

benzekrimaha force-pushed the improvement/CLDSRV-919-afterEach-flakiness-internal-error-fix branch from 71b7d71 to 0a09689 Compare July 9, 2026 16:10

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

benzekrimaha force-pushed the improvement/CLDSRV-919-afterEach-flakiness-internal-error-fix branch from a216db5 to b8c7b85 Compare July 10, 2026 06:17

benzekrimaha added 4 commits July 10, 2026 08:51

Add unit tests for BucketAlreadyExists race handling

db0967a

Cover freshStartCreateBucket continuing after a concurrent create and getMPUBucket refetching the existing MPU shadow bucket. Issue: CLDSRV-919

Bump arsenal to ARSN-613 for BucketAlreadyExists contract

4e69ab0

Pull in Arsenal 8.4.20 at 0bd1b04c (ARSN-613), including the usersBucketHack getBucket-first fix paired with this cloudserver change. Issue: CLDSRV-919

Prettify files touched by PR

19a2bd2

The changed-files prettier CI check validates whole files, not just the modified lines. Issue: CLDSRV-919

benzekrimaha force-pushed the improvement/CLDSRV-919-afterEach-flakiness-internal-error-fix branch from b8c7b85 to 19a2bd2 Compare July 10, 2026 06:51

benzekrimaha requested review from SylvainSenechal, delthas, francoisferrand and tcarmet July 10, 2026 07:18

francoisferrand reviewed Jul 10, 2026

View reviewed changes

SylvainSenechal reviewed Jul 10, 2026

View reviewed changes

SylvainSenechal self-requested a review July 10, 2026 17:44

Uh oh!

Conversation

benzekrimaha commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

bert-e commented Jul 8, 2026

Hello benzekrimaha,

Uh oh!

bert-e commented Jul 8, 2026

Incorrect fix version

Uh oh!

bert-e commented Jul 8, 2026

Request integration branches

Uh oh!

bert-e commented Jul 8, 2026

Waiting for approval

Uh oh!

bert-e commented Jul 8, 2026

Waiting for approval

Uh oh!

codecov Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bert-e commented Jul 9, 2026

Request integration branches

Uh oh!

Uh oh!

francoisferrand left a comment

Choose a reason for hiding this comment

Uh oh!

francoisferrand Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

benzekrimaha commented Jul 8, 2026 •

edited

Loading

codecov Bot commented Jul 8, 2026 •

edited

Loading

SylvainSenechal Jul 10, 2026 •

edited

Loading