Large files not uploaded (size > highWatermark)

[msageryd]

I can't get large file uploads to work other than if I raise highWatermark to above the file size. I thought highWatermark was the limit for when a new chunk should be sent. Have I misunderstood something?

const AWS = require('aws-sdk');
AWS.config.update({
  accessKeyId: accessKey, 
  secretAccessKey: secretKey
});

const S3 = AWS.S3;
const S3S = require('s3-streams');
const read = fs.createReadStream('./big.pdf') //18MB file

const s3Upload = S3S.WriteStream(new S3(), {Bucket: 'myBucket', Key: 'test'});

read.pipe(s3Upload)
.on('finish', function() {
  console.log(`finished piping to S3.. `);
})

[trevtrich]

I'm having this issue as well. Any movement on this?

[msageryd]

I suspect that the author won't prioritize this since the official AWS SDK handles this quite easily nowadays. You can throw anything on it (stream or buffer) and it just works =) I'm using the SDK instead of S3-streams and I'm a happy camper.

[msageryd]

My apology to Izaak for possibly leading richardson-trevor away from S3-Streams. But since my issue has been open for several months I suspect that the module might have been abandoned.

Here is how I do it (returns a promise):

//Declare AWS "global" to the module for use in all AWS related functions
const AWS = require('aws-sdk');
AWS.config.update({
  region: 'yourRegion',
  accessKeyId: 'yourAccessKey',
  secretAccessKey: 'yourSecret'
});

function upload(options) {
  const s3 = new AWS.S3();
  const s3Params = {
    Bucket: options.bucket,
    Key: options.key,
    Body: options.file  //stream or buffer
  };

  const s3Options = {
    partSize: 5 * 1024 * 1024,
    queueSize: 1
  };

  return new Promise(function uploadPromise(resolve, reject) {
    s3.upload(s3Params, s3Options, function s3Upload(err) {
      if(err) return reject(err);
      resolve();
    });
  });
}

[trevtrich]

Thanks for the input, that's probably right on. We'll take a second look at that with what we're doing!

Sent from my iPhone

On May 17, 2016, at 11:35 AM, michaelswe [email protected] wrote:

I suspect that the author won't prioritize this since the official AWS SDK handles this quite easily nowadays. You can throw anything on it (stream or buffer) and it just works =) I'm using the SDK instead of S3-streams and I'm a happy camper.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub