Skip to content

Commit

Permalink
feat: Change upload to S3Manager instead of classic PUT to increase m…
Browse files Browse the repository at this point in the history 
…aximum file size upload

Related to #391
  • Loading branch information
@oxyno-zeta
oxyno-zeta committed Oct 23, 2023
1 parent d426ca2 commit 25bb790
Show file tree
Hide file tree
Showing 9 changed files with 258 additions and 114 deletions.
4 changes: 4 additions & 0 deletions conf/config-example.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,10 @@ targets:
region: eu-west-1
s3Endpoint:
disableSSL: false
# s3MaxUploadParts: 10000
# s3UploadPartSize: 5
# s3UploadConcurrency: 5
# s3UploadLeavePartsOnError: false
# s3ListMaxKeys: 1000
# credentials:
# accessKey:
Expand Down
4 changes: 4 additions & 0 deletions docs/configuration/example.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,10 @@ targets:
region: eu-west-1
s3Endpoint:
disableSSL: false
# s3MaxUploadParts: 10000
# s3UploadPartSize: 5
# s3UploadConcurrency: 5
# s3UploadLeavePartsOnError: false
# s3ListMaxKeys: 1000
# credentials:
# accessKey:
Expand Down
24 changes: 14 additions & 10 deletions docs/configuration/structure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -304,16 +304,20 @@ You can found more information [here](../feature-guide/webhooks.md) about webhoo

## BucketConfiguration

| Key | Type | Required | Default | Description |
| ------------- | --------------------------------------------------------------------- | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | String | Yes | None | Bucket name in S3 provider |
| prefix | String | No | None | Bucket prefix |
| region | String | No | `us-east-1` | Bucket region |
| s3Endpoint | String | No | None | Custom S3 Endpoint for non AWS S3 bucket |
| credentials | [BucketCredentialConfiguration](#bucketcredentialconfiguration) | No | None | Credentials to access S3 bucket |
| disableSSL | Boolean | No | `false` | Disable SSL connection |
| s3ListMaxKeys | Integer | No | `1000` | This flag will be used for the max pagination list management of files and "folders" in S3. In S3 list requests, the limit is fixed to 1000 items maximum. S3-Proxy will allow to increase this by making multiple requests to S3. Warning: This will increase the memory and CPU usage. |
| requestConfig | [BucketRequestConfigConfiguration](#bucketrequestconfigconfiguration) | No | `nil` | This will allow to customize requests sent to your S3 backend. |
| Key | Type | Required | Default | Description |
| ------------------------- | --------------------------------------------------------------------- | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | String | Yes | None | Bucket name in S3 provider |
| prefix | String | No | None | Bucket prefix |
| region | String | No | `us-east-1` | Bucket region |
| s3Endpoint | String | No | None | Custom S3 Endpoint for non AWS S3 bucket |
| credentials | [BucketCredentialConfiguration](#bucketcredentialconfiguration) | No | None | Credentials to access S3 bucket |
| disableSSL | Boolean | No | `false` | Disable SSL connection |
| s3ListMaxKeys | Integer | No | `1000` | This flag will be used for the max pagination list management of files and "folders" in S3. In S3 list requests, the limit is fixed to 1000 items maximum. S3-Proxy will allow to increase this by making multiple requests to S3. Warning: This will increase the memory and CPU usage. |
| requestConfig | [BucketRequestConfigConfiguration](#bucketrequestconfigconfiguration) | No | `nil` | This will allow to customize requests sent to your S3 backend. |
| s3MaxUploadParts | Integer | No | `10000` | MaxUploadParts is the max number of parts which will be uploaded to S3. |
| s3UploadPartSize | Integer | No | `5` | The buffer size (in megabytes) to use when buffering data into chunks and sending them as parts to S3. The minimum allowed part size is 5MB, and if this value is set to zero, the DefaultUploadPartSize value will be used. |
| s3UploadConcurrency | Integer | No | `5` | The number of goroutines to spin up in parallel per call to Upload when sending parts. If this is set to zero, the DefaultUploadConcurrency value will be used. |
| s3UploadLeavePartsOnError | Boolean | No | `false` | Setting this value to true will cause the SDK to avoid calling AbortMultipartUpload on a failure, leaving all successfully uploaded parts on S3 for manual recovery. |

## BucketRequestConfigConfiguration

Expand Down
26 changes: 18 additions & 8 deletions pkg/s3-proxy/config/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"time"

"emperror.dev/errors"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
)

// DefaultPort Default port.
Expand Down Expand Up @@ -132,6 +133,11 @@ var ErrMainBucketPathSupportNotValid = errors.New("main bucket path support opti
// TemplateErrLoadingEnvCredentialEmpty Template Error when Loading Environment variable Credentials.
var TemplateErrLoadingEnvCredentialEmpty = "error loading credentials, environment variable %s is empty" //nolint: gosec // No credentials here, false positive

// Default Upload configurations.
const DefaultS3MaxUploadParts = s3manager.MaxUploadParts
const DefaultS3UploadPartSize int64 = 5
const DefaultS3UploadConcurrency = s3manager.DefaultUploadConcurrency

const oidcLoginPathTemplate = "/auth/%s"
const oidcCallbackPathTemplate = "/auth/%s/callback"

Expand Down Expand Up @@ -468,14 +474,18 @@ type OPAServerAuthorization struct {

// BucketConfig Bucket configuration.
type BucketConfig struct {
Credentials *BucketCredentialConfig `mapstructure:"credentials" validate:"omitempty"`
RequestConfig *BucketRequestConfig `mapstructure:"requestConfig" validate:"omitempty"`
Name string `mapstructure:"name" validate:"required"`
Prefix string `mapstructure:"prefix"`
Region string `mapstructure:"region"`
S3Endpoint string `mapstructure:"s3Endpoint"`
S3ListMaxKeys int64 `mapstructure:"s3ListMaxKeys" validate:"gt=0"`
DisableSSL bool `mapstructure:"disableSSL"`
Credentials *BucketCredentialConfig `mapstructure:"credentials" validate:"omitempty"`
RequestConfig *BucketRequestConfig `mapstructure:"requestConfig" validate:"omitempty"`
Name string `mapstructure:"name" validate:"required"`
Prefix string `mapstructure:"prefix"`
Region string `mapstructure:"region"`
S3Endpoint string `mapstructure:"s3Endpoint"`
S3ListMaxKeys int64 `mapstructure:"s3ListMaxKeys" validate:"gt=0"`
S3MaxUploadParts int `mapstructure:"s3MaxUploadParts" validate:"required,gte=1"`
S3UploadPartSize int64 `mapstructure:"s3UploadPartSize" validate:"required,gte=5"`
S3UploadConcurrency int `mapstructure:"s3UploadConcurrency" validate:"required,gte=1"`
S3UploadLeavePartsOnError bool `mapstructure:"s3UploadLeavePartsOnError"`
DisableSSL bool `mapstructure:"disableSSL"`
}

// BucketRequestConfig Bucket request configuration.
Expand Down
12 changes: 12 additions & 0 deletions pkg/s3-proxy/config/managercontext.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -590,6 +590,18 @@ func loadBusinessDefaultValues(out *Config) error {
if item.Bucket != nil && item.Bucket.S3ListMaxKeys == 0 {
item.Bucket.S3ListMaxKeys = DefaultBucketS3ListMaxKeys
}
// Manage default s3 max upload parts
if item.Bucket != nil && item.Bucket.S3MaxUploadParts == 0 {
item.Bucket.S3MaxUploadParts = DefaultS3MaxUploadParts
}
// Manage default s3 upload part size
if item.Bucket != nil && item.Bucket.S3UploadPartSize == 0 {
item.Bucket.S3UploadPartSize = DefaultS3UploadPartSize
}
// Manage default s3 upload concurrency
if item.Bucket != nil && item.Bucket.S3UploadConcurrency == 0 {
item.Bucket.S3UploadConcurrency = DefaultS3UploadConcurrency
}
// Manage default configuration for target actions
if item.Actions == nil {
item.Actions = &ActionsConfig{GET: &GetActionConfig{Enabled: true}}
Expand Down
Loading

0 comments on commit 25bb790

Please sign in to comment.