Code

Implement custom storage for orgs#2093

Code

Open

tw4l wants to merge 64 commits intomainwebrecorder/browsertrix:mainfrom issue-578-custom-storagewebrecorder/browsertrix:issue-578-custom-storageCopy head branch name to clipboard

+947-91

Commits

Commits on Dec 30, 2024

• Add back custom storage endpoints

  tw4l

  committed
  8388ce2

• Flush out tests for setting custom storage

  tw4l

  committed
  5fbdca1

• Fix test issue with bucket not existing for now

  tw4l

  committed
  fb18134

• Add additional tests

  tw4l

  committed
  0302275

• Fix custom storage so it works as expected

  Show description for 19850ba
  tw4l

  committed
  19850ba

• Actually unset custom replica storage before deleting

  tw4l

  committed
  c78b5b8

• Add TODO where custom storage deletion is failing

  tw4l

  committed
  d5fd8e1

• Fix check for whether storage label is in use

  tw4l

  committed
  1dcd39b

• Remove todo on endpoint that's fine

  tw4l

  committed
  1a1cb30

• Add todos re: tasks necessary to change storage

  tw4l

  committed
  7558ac3

• Check that no crawls are running before updating storage

  tw4l

  committed
  46e0fc9

• Start adding post-storage update logic

  tw4l

  committed
  791ca6e

• WIP: Add background job to copy old s3 bucket to new

  tw4l

  committed
  b899ecf

• WIP: Start adding logic to handle replica location updates

  tw4l

  committed
  21070d6

• Add additional note

  tw4l

  committed
  7b6a917

• Fix argument

  tw4l

  committed
  3b3116f

• Fix another argument

  tw4l

  committed
  9076555

• Fixups

  tw4l

  committed
  853b840

• Fix linting

  tw4l

  committed
  3a4b43c

• More linting fixes

  tw4l

  committed
  048a563

• Refactor, seperate storage and replicas updates

  tw4l

  committed
  e3c41b9

• More refactoring

  tw4l

  committed
  34310de

• Make post-update task methods private

  tw4l

  committed
  80cc5d4

• Check if any bg jobs running before changing storage

  tw4l

  committed
  4112044

• Check bg job finished as well

  tw4l

  committed
  287fae9

• Fixups

  tw4l

  committed
  5096005

• Storage update improvements

  tw4l

  committed
  7d7fd36

• Fixup

  tw4l

  committed
  cb461f6

• Remove TODO

  tw4l

  committed
  6e3f3ef

• Remove another todo

  tw4l

  committed
  cf40245

• More fixups

  tw4l

  committed
  3de96a8

• Add provider to s3storage for rclone

  tw4l

  committed
  db65dd8

• Fix typo

  tw4l

  committed
  89585fa

• Make API endpoints that change storage superuser-only for now

  tw4l

  committed
  8abe1ef

• Add typing for init_storages_api, import Callable

  tw4l

  committed
  c146ab3

• Add missing User import

  tw4l

  committed
  abc238e

• Fix StorageOps in operator main

  tw4l

  committed
  29a54b7

• Always use oid prefix in s3 storage

  Show description for f348a16
  tw4l

  committed
  f348a16

• Post-rebase fixups and remove create bucket fallback

  Show description for e67868b
  tw4l

  committed
  e67868b

• Create extra test buckets in CI

  tw4l

  committed
  e2dfa93

• Add test for non-verified custom storage

  tw4l

  committed
  e054158

• Refactor to move updates to FastAPI background tasks

  tw4l

  committed
  f0dc334

• Include default replicas in /storage response if no org replicas

  tw4l

  committed
  0d3aa35

• Fix unsetting of presigned URLs

  tw4l

  committed
  7a8ec58

• Add --progress flag to rclone copy command

  tw4l

  committed
  6277331

• Increase ttl seconds after finished for testing on dev

  tw4l

  committed
  1eb533c

• Ensure there are no double slashes between bucket name and oid

  tw4l

  committed
  be10a9a

• Increase memory limit/request for copy job to 500Mi

  tw4l

  committed
  e028dc8

• Reduce copy job ttlSecondsAfterFinished to 60

  tw4l

  committed
  edcd5c6

• Add storage tag to API endpoints

  tw4l

  committed
  d676113

• Add flags to rclone to reduce memory usage, set limit to 350Mi

  tw4l

  committed
  ef59178

• Fix positional operator in storage ref update

  tw4l

  committed
  07490fb

• One more positional operator fix

  tw4l

  committed
  4f1aa1d

• Update docstrings and comments

  tw4l

  committed
  5d81ed2

• Make all-storages response valid JSON with response model

  tw4l

  committed
  b095023

• Add admin docs for storage

  tw4l

  committed
  085209b

• Fix API endpoint path in docs example

  tw4l

  committed
  76285ef

• Docs typo fix

  tw4l

  committed
  41d4f79

• Add provider field note

  tw4l

  committed
  db12ef9

• Docs language cleanup

  tw4l

  committed
  5ca80f0

• Check /all-storages in backend tests

  tw4l

  committed
  af0d966

• Add API endpoint for background job progress

  tw4l

  committed
  b461113

• Fix linting

  tw4l

  committed
  60582c1

• Format post-rebase with Black

  tw4l

  committed
  c889664