S3UTILS-157: explicit closing of mongodb cursors #310

williamlardier · 2024-03-26T11:07:41Z

There is suspicion of cursor leaks with the CountItems job, especially when enumerating a large number of cursors. This PR adds explicit checks after the end of the processing. The cursors should already be closed as per the nature of the forEach, that closes the cursor when there is no resource to check anymore.

Issue: S3UTILS-157

bert-e · 2024-03-26T11:07:44Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

utils/S3UtilsMongoClient.js

williamlardier · 2024-03-27T08:46:22Z

Putting this work on hold as for now this does not look like the cause of the number of open connections to MongoDB, we need to reproduce first, otherwise these changes are not needed.

utils/S3UtilsMongoClient.js

bert-e · 2024-03-29T15:04:30Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/S3UTILS-157-close-cursors into
development/1.14
w/1.15/bugfix/S3UTILS-157-close-cursors will be merged into development/1.15

The following branches will NOT be impacted:

development/1.13
development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-03-29T15:04:31Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

francoisferrand · 2024-03-29T16:53:19Z

utils/S3UtilsMongoClient.js

+            return usersBucketCreationDatesArray
+                .reduce((map, obj) => ({ ...map, [obj._id]: obj.value.creationDate }), {});
+        } catch (err) {
+            log.error('Failed to read __usersbucket collection', {


nit: any reason to catch the exception instead of letting it propagate up, like the other exceptions which may be generated from within getObjectMDStats() and get caught at the end of that function?

(i am fine either way, just wondering)

To was mainly to ensure the right error messages are logged, so we can return an arsenal error

bert-e · 2024-03-30T09:13:30Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

williamlardier · 2024-04-02T07:53:24Z

/create_integration_branches

bert-e · 2024-04-02T07:53:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

bert-e · 2024-04-02T07:55:16Z

History mismatch

Merge commit #5017e66391393dbb5a59114897abf68d59efdb21 on the integration branch
w/1.15/bugfix/S3UTILS-157-close-cursors is merging a branch which is neither the current
branch bugfix/S3UTILS-157-close-cursors nor the development branch
development/1.15.

It is likely due to a rebase of the branch bugfix/S3UTILS-157-close-cursors and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

williamlardier · 2024-04-02T07:55:18Z

/reset

bert-e · 2024-04-02T07:55:23Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2024-04-02T07:55:29Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/S3UTILS-157-close-cursors into
development/1.14
w/1.15/bugfix/S3UTILS-157-close-cursors will be merged into development/1.15

The following branches will NOT be impacted:

development/1.13
development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-04-02T07:55:29Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier · 2024-04-02T07:55:30Z

(rebasing to remove a unused line, leftover from testing)

williamlardier · 2024-04-02T08:14:49Z

/approve

bert-e · 2024-04-02T08:15:01Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/1.14
✔️ development/1.15

The following branches will NOT be impacted:

development/1.13
development/1.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

bert-e · 2024-04-02T08:15:11Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.14
✔️ development/1.15

The following branches have NOT changed:

development/1.13
development/1.4

Please check the status of the associated issue S3UTILS-157.

Goodbye williamlardier.

williamlardier requested review from francoisferrand and benzekrimaha March 26, 2024 11:08

scality deleted a comment from bert-e Mar 26, 2024

francoisferrand reviewed Mar 26, 2024

View reviewed changes

utils/S3UtilsMongoClient.js Outdated Show resolved Hide resolved

francoisferrand reviewed Mar 26, 2024

View reviewed changes

utils/S3UtilsMongoClient.js Outdated Show resolved Hide resolved

williamlardier force-pushed the bugfix/S3UTILS-157-close-cursors branch 2 times, most recently from c70767f to a07dc8e Compare March 29, 2024 11:17

williamlardier requested a review from francoisferrand March 29, 2024 11:20

williamlardier force-pushed the bugfix/S3UTILS-157-close-cursors branch 2 times, most recently from a1769c7 to e29bfb6 Compare March 29, 2024 12:41

francoisferrand reviewed Mar 29, 2024

View reviewed changes

utils/S3UtilsMongoClient.js Outdated Show resolved Hide resolved

williamlardier force-pushed the bugfix/S3UTILS-157-close-cursors branch 2 times, most recently from 54984b6 to 9ac96af Compare March 29, 2024 15:02

francoisferrand reviewed Mar 29, 2024

View reviewed changes

francoisferrand approved these changes Mar 29, 2024

View reviewed changes

francoisferrand requested review from Kerkesni, KillianG, jonathan-gramain and tmacro March 29, 2024 16:54

scality deleted a comment from bert-e Mar 30, 2024

scality deleted a comment from williamlardier Mar 30, 2024

scality deleted a comment from bert-e Mar 30, 2024

scality deleted a comment from williamlardier Mar 30, 2024

scality deleted a comment from bert-e Mar 30, 2024

scality deleted a comment from williamlardier Mar 30, 2024

scality deleted a comment from bert-e Mar 30, 2024

scality deleted a comment from williamlardier Mar 30, 2024

scality deleted a comment from bert-e Mar 30, 2024

benzekrimaha approved these changes Apr 2, 2024

View reviewed changes

williamlardier added 2 commits April 2, 2024 09:55

S3UTILS-157: explicitly close the mongodb cursors

96effd8

S3UTILS-157: bump project version

fc0694e

williamlardier force-pushed the bugfix/S3UTILS-157-close-cursors branch from 9ac96af to fc0694e Compare April 2, 2024 07:55

bert-e merged commit fc0694e into development/1.14 Apr 2, 2024
18 checks passed

bert-e deleted the bugfix/S3UTILS-157-close-cursors branch April 2, 2024 08:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3UTILS-157: explicit closing of mongodb cursors #310

S3UTILS-157: explicit closing of mongodb cursors #310

williamlardier commented Mar 26, 2024 •

edited by francoisferrand

Loading

bert-e commented Mar 26, 2024

williamlardier commented Mar 27, 2024

bert-e commented Mar 29, 2024

bert-e commented Mar 29, 2024

francoisferrand Mar 29, 2024 •

edited

Loading

williamlardier Apr 2, 2024

bert-e commented Mar 30, 2024

williamlardier commented Apr 2, 2024

bert-e commented Apr 2, 2024

bert-e commented Apr 2, 2024

williamlardier commented Apr 2, 2024

bert-e commented Apr 2, 2024

bert-e commented Apr 2, 2024

bert-e commented Apr 2, 2024

williamlardier commented Apr 2, 2024

williamlardier commented Apr 2, 2024

bert-e commented Apr 2, 2024

bert-e commented Apr 2, 2024

S3UTILS-157: explicit closing of mongodb cursors #310

S3UTILS-157: explicit closing of mongodb cursors #310

Conversation

williamlardier commented Mar 26, 2024 • edited by francoisferrand Loading

bert-e commented Mar 26, 2024

Hello williamlardier,

williamlardier commented Mar 27, 2024

bert-e commented Mar 29, 2024

Integration data created

bert-e commented Mar 29, 2024

Waiting for approval

francoisferrand Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

williamlardier Apr 2, 2024

Choose a reason for hiding this comment

bert-e commented Mar 30, 2024

Request integration branches

williamlardier commented Apr 2, 2024

bert-e commented Apr 2, 2024

Waiting for approval

bert-e commented Apr 2, 2024

History mismatch

williamlardier commented Apr 2, 2024

bert-e commented Apr 2, 2024

Reset complete

bert-e commented Apr 2, 2024

Integration data created

bert-e commented Apr 2, 2024

Waiting for approval

williamlardier commented Apr 2, 2024

williamlardier commented Apr 2, 2024

bert-e commented Apr 2, 2024

In the queue

bert-e commented Apr 2, 2024

williamlardier commented Mar 26, 2024 •

edited by francoisferrand

Loading

francoisferrand Mar 29, 2024 •

edited

Loading