Support lifecycle transitions on OOB bucket

[francoisferrand]

Core idea is to consider that OOB does not sync the whole bucket, but only the objects which are indeed in the OOB location : there may be objects in Zenko which are not in Metadata, as long as they are not marked as present in that location.

This requires just a few changes:

• Delete operations ignore objects whose dataStoreLocation is not the OOB location itself. This fixes most of the issues, and allows transitioning objects from S3C to cold stoage (they get removed from S3C completely at the end of the transition).
• When object is restored, cloudserver will now add an extra user metadata (x-amz-meta-scal-version-id): this is used to ensure we do not duplicate the object when it is restored.
• Finally, need to tweak the way updates (not actual ingestions) from S3C are processed, to ensure we keep the extra Zenko fields (esp. transition or cold storage fields), while updating only what is expected: i.e. only tags.

Issue: BB-590

[bert-e]

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[codecov]

Codecov Report

Attention: Patch coverage is 91.22807% with 5 lines in your changes missing coverage. Please review.

Project coverage is 54.69%. Comparing base (070472c) to head (2a5e212).

Files with missing lines	Patch %	Lines
lib/models/DeleteOpQueueEntry.js	50.00%	2 Missing ⚠️
lib/models/QueueEntry.js	0.00%	2 Missing ⚠️
extensions/mongoProcessor/MongoQueueProcessor.js	98.03%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/queuePopulator/LogReader.js	29.67% <ø> (ø)
extensions/mongoProcessor/MongoQueueProcessor.js	69.41% <98.03%> (+6.69%)	⬆️
lib/models/DeleteOpQueueEntry.js	62.50% <50.00%> (+52.50%)	⬆️
lib/models/QueueEntry.js	70.58% <0.00%> (+0.89%)	⬆️

... and 5 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	15.71% <ø> (ø)
Core Library	62.10% <33.33%> (+0.42%)	⬆️
Ingestion	69.09% <98.03%> (+1.55%)	⬆️
Lifecycle	46.61% <ø> (ø)
Oplog Populator	84.20% <ø> (ø)
Replication	51.44% <ø> (+0.03%)	⬆️
Bucket Scanner	85.60% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.0    #2606      +/-   ##
===================================================
+ Coverage            54.35%   54.69%   +0.34%     
===================================================
  Files                  201      201              
  Lines                13336    13368      +32     
===================================================
+ Hits                  7249     7312      +63     
+ Misses                6077     6046      -31     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.51% <0.00%> (-0.03%)	⬇️
api:routes	9.33% <0.00%> (-0.03%)	⬇️
bucket-scanner	85.60% <ø> (ø)
ingestion	12.63% <91.22%> (+0.34%)	⬆️
lib	7.35% <0.00%> (-0.02%)	⬇️
lifecycle	18.78% <0.00%> (-0.05%)	⬇️
notification	1.08% <0.00%> (-0.01%)	⬇️
replication	18.64% <0.00%> (+0.05%)	⬆️
unit	4.97% <0.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.