[FEATURE REQUEST] Add Storj support

[RoiArthurB]

Hello,

This project is really awesome, however the proxy doesn't seems to properly handle every S3-compatible storage. For instance, the Storj doesn't seem to be properly supported even if it's possible to generate some S3-like credentials.

The main difference I saw during my experiments with other providers is the lack of an userid which gives my a endpoint like : this https://access-key:[email protected]/gateway.storjshare.io/test-bucket

I'd be glad to help to implement it, so feel free to ask me for more details/tests :)

[RoiArthurB]

After deeper experimentation, I think an issue appears as Storj requires having data encrypted. However, your project doesn't allow passing Encryption Passphrase as URL parameter.

Can you tell me if (1) this seems to be the problem from your point of view too, and (2) if it's a feature you're planing to add :)

Here's the error when I use the default gateway endpoint :

LFS: Client error: https://gateway.storjshare.io/hoankiemair/fe5cd9efa40ff4a210001748b9a64d2abd20f56755f060b130cae730c67ba69f?X-Amz-Expires=3600&X-Amz-Date=20230830T040314Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f10a19450038c42d54e73f69eb0dd14f66653da4e63e151947fbb43f42f5077e
LFS: Client error: https://gateway.storjshare.io/hoankiemair/9df36143adbd164eb4f66d26f472aff2ba04d49f6b5caeb2a56d758750222593?X-Amz-Expires=3600&X-Amz-Date=20230830T040318Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=a796d915e7439331b38204e90ef05e93df233221536dce60c799788a8f9c5001
LFS: Client error: https://gateway.storjshare.io/hoankiemair/2834c8a1aadf4a091d44e27b705056678e8962aebbc90efd9771c417e830a546?X-Amz-Expires=3600&X-Amz-Date=20230830T040318Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=761fe2f9d4387e77a12aea87427bf36c059bedd96e4784e03a08ebd6d17e92ed
LFS: Client error: https://gateway.storjshare.io/hoankiemair/fc31421d3c9f2a1d01143b7ba28d18c3a9d32e177cbecf8a8f551f287adc3adf?X-Amz-Expires=3600&X-Amz-Date=20230830T040321Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=cbc20408fbfe8b5ffed23721ea33177915d18578301df2b315202e824f292b51
...

Here's the error log when I use the explicit bucket endpoint :

$ git push        
LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=ap1.storj.io) %!!(string=/hoankiemair/2834c8a1aadf4a091d44e27b705056678e8962aebbc90efd9771c417e830a546) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=X-Amz-Expires=3600&X-Amz-Date=20230830T032410Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=e1ec169fa162436420b625a809c64be50c620acd320725cb228ddcbf36fd6ace) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=ap1.storj.io) %!!(string=/hoankiemair/9df36143adbd164eb4f66d26f472aff2ba04d49f6b5caeb2a56d758750222593) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=X-Amz-Expires=3600&X-Amz-Date=20230830T032410Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=306841ffb06a9ecff59e67465c6cb444eb3be655a0eca079478090a17ee2512a) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=ap1.storj.io) %!!(string=/hoankiemair/fe5cd9efa40ff4a210001748b9a64d2abd20f56755f060b130cae730c67ba69f) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=X-Amz-Expires=3600&X-Amz-Date=20230830T032411Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=0387b3daec17738a99fd8e06fa42a21c415365a7cb44baeae8b1208a0effec3d) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=ap1.storj.io) %!!(string=/hoankiemair/ef6f4886bc2265caa655a4c73cb6c939b677acb0f537ab596d3579f1505cf1f9) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=X-Amz-Expires=3600&X-Amz-Date=20230830T032413Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESS-KEY%2F20230830%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=9a51747b915f81ac73a6ea4c566c6bf03a6fb6f406bca97203c83d7b2ecb87c3) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
...

[milkey-mouse]

Thanks for the issue report. I don't have much experience with Storj, but if I'm reading the docs correctly, it doesn't look like you have to pass the encryption passphrase as a URL parameter when using presigned URLs. (Here's an example in the docs where they don't.) I think they use the Auth Service to map S3 access tokens to the encryption passphrase and such. So I don't think the lack of encryption passphrase is the issue; the tokens from "Generate S3 Gateway Credentials" following these steps should be sufficient.

The other thing I noticed is your LFS client is receiving HTTP 413 Content Too Large responses, so perhaps the issue has something to do with multipart uploads. Your Git LFS client (and every other I know of) doesn't support multipart uploads/downloads, but Storj recommends them for small uploads and may require them for larger ones (S3 requires multipart for objects larger than 5 GB). How big are the files you're trying to upload?

I probably won't have time to play with Storj for a few days at least, but let me know if you find another issue or if something here solves your problem.