Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLDSRV-527 #5564

Draft
wants to merge 15 commits into
base: development/7.70
Choose a base branch
from
Draft

CLDSRV-527 #5564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
65d46c6
[edit]CLDSRV-527: add necessary packages
Jun 3, 2024
fd6298d
CLDSRV-527: accomodate POST obj API call
Jun 3, 2024
4913ab0
CLDSRV-527: add POST Object API Handler
Jun 3, 2024
587f5bf
CLDSRV-527: accomodate POST Object in extraneous code
Jun 3, 2024
77c69cc
[WIP]CLDSRV-527: add test files
Jun 3, 2024
aadef76
fixup: packages
Jun 4, 2024
77a4fa0
first functional tests are fucntional
Jun 12, 2024
7bd8623
update package
Jun 14, 2024
c95aa64
update func tests
Jun 14, 2024
a83edc3
fix other posts
Jun 14, 2024
0b3e802
checkpoint: changing the parser
Jun 17, 2024
15f1472
checkpoint: parser changed to custom, very slow on large files
Jun 19, 2024
7db8da2
checkpoint: acl tests
Jun 24, 2024
ae17d9a
restablish busboy
Jun 24, 2024
e86e23a
quick fixes + unit tests
Jun 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
270 changes: 233 additions & 37 deletions lib/api/api.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
const objectGetTagging = require('./objectGetTagging');
const objectHead = require('./objectHead');
const objectPut = require('./objectPut');
const objectPost = require('./objectPost');
const objectPutACL = require('./objectPutACL');
const objectPutLegalHold = require('./objectPutLegalHold');
const objectPutTagging = require('./objectPutTagging');
Expand All @@ -68,11 +69,128 @@
const parseCopySource = require('./apiUtils/object/parseCopySource');
const { tagConditionKeyAuth } = require('./apiUtils/authorization/tagConditionKeys');
const checkHttpHeadersSize = require('./apiUtils/object/checkHttpHeadersSize');
const { decryptToken } = require('./apiUtils/object/continueToken');
const busboy = require('busboy');
const fs = require('fs');

Check failure on line 74 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

'fs' is defined but never used
const path = require('path');

Check failure on line 75 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

'path' is defined but never used
const os = require('os');

Check failure on line 76 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

'os' is defined but never used
const { PassThrough } = require('stream');


const monitoringMap = policies.actionMaps.actionMonitoringMapS3;

auth.setHandler(vault);

function parseMultipartFormData(request, callback) {

Check failure on line 84 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

'parseMultipartFormData' is defined but never used
let algoOK = false;
let credOK = false;
let dateOK = false;
let sigOK = false;
let policyOK = false;
request.formData = {};

Check warning on line 90 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Assignment to property of function parameter 'request'

const boundary = request.headers['content-type'].split('boundary=')[1];
const boundaryBuffer = Buffer.from(`--${boundary}`);
const newlineBuffer = Buffer.from('\r\n');

let buffer = Buffer.alloc(0);
let currentField = null;

Check failure on line 97 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

'currentField' is defined but never used
let file = null;
let count = 0;

request.on('data', (chunk) => {
buffer = Buffer.concat([buffer, chunk]);

let boundaryIndex;
console.log('part count:', count++);

Check failure on line 105 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Unexpected console statement

while ((boundaryIndex = buffer.indexOf(boundaryBuffer)) !== -1) {

Check failure on line 107 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Unexpected assignment within a 'while' statement
let part = buffer.slice(0, boundaryIndex);
buffer = buffer.slice(boundaryIndex + boundaryBuffer.length);

if (part.length === 0) continue; // skip empty parts

let partToProcess = part;

Check warning on line 113 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

'partToProcess' is never reassigned, use 'const' instead
if (part.indexOf(newlineBuffer) === 0) {
part = part.slice(newlineBuffer.length);
}

const headersEndIndex = partToProcess.indexOf(newlineBuffer + newlineBuffer);
const headers = partToProcess.slice(0, headersEndIndex).toString().split('\r\n');
let content = partToProcess.slice(headersEndIndex + newlineBuffer.length * 2);
if (content.slice(-2).equals(newlineBuffer)) {
content = content.slice(0, -2);
}

const contentDisposition = headers.find(header => header.startsWith('Content-Disposition'));
const contentTypeHeader = headers.find(header => header.startsWith('Content-Type'));
const mimetype = contentTypeHeader ? contentTypeHeader.split(': ')[1] : '';

if (contentDisposition) {
const nameMatch = contentDisposition.match(/name="([^"]+)"/);
const filenameMatch = contentDisposition.match(/filename="([^"]+)"/);

if (nameMatch) {
const fieldname = nameMatch[1];
if (filenameMatch) {
// File field
const filename = filenameMatch[1];

// Remove the trailing CRLF from the content

// 'Content-Disposition: form-data; name="file"; filename="test.txt"'

//const mimetype = headers.find(header => header.startsWith('Content-Type')).split(': ')[1];

Check failure on line 143 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Expected exception block, space or tab after '//' in comment

file = new PassThrough();
file.write(content);

// Pipe the remaining data
request.pipe(file);
//request.pipe(fileStream);

Check failure on line 150 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Expected exception block, space or tab after '//' in comment

if (algoOK && credOK && dateOK && sigOK && policyOK) {
callback(null, { file, fieldname, filename, boundaryBuffer, mimetype });
}

currentField = null;
} else {
// Regular field
currentField = fieldname;
request.formData[fieldname] = content.toString();

Check warning on line 160 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Assignment to property of function parameter 'request'

if (fieldname === 'X-Amz-Algorithm') {
algoOK = true;
}
if (fieldname === 'X-Amz-Credential') {
credOK = true;
}
if (fieldname === 'X-Amz-Date') {
dateOK = true;
}
if (fieldname === 'X-Amz-Signature') {
sigOK = true;
}
if (fieldname === 'Policy') {
const decrypted = decryptToken(request.formData.Policy);
request.formData.decryptedPolicy = JSON.parse(decrypted);

Check warning on line 176 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Assignment to property of function parameter 'request'
policyOK = true;
}

currentField = null;
}
}
}
}
});

request.on('end', () => {
if (!algoOK || !credOK || !dateOK || !sigOK || !policyOK) {
callback(new Error('InvalidRequest'));
}
});
}

/* eslint-disable no-param-reassign */
const api = {
callApiMethod(apiMethod, request, response, log, callback) {
Expand Down Expand Up @@ -112,7 +230,7 @@

// no need to check auth on website or cors preflight requests
if (apiMethod === 'websiteGet' || apiMethod === 'websiteHead' ||
apiMethod === 'corsPreflight') {
apiMethod === 'corsPreflight') {
request.actionImplicitDenies = false;
return this[apiMethod](request, log, callback);
}
Expand Down Expand Up @@ -158,7 +276,7 @@
// second item checks s3:GetObject(Version)Tagging action
if (!authResults[1].isAllowed) {
log.trace('get tagging authorization denial ' +
'from Vault');
'from Vault');
returnTagCount = false;
}
} else {
Expand All @@ -184,8 +302,108 @@
}
return { returnTagCount, isImplicitDeny };
}
let bb;
let fileEventData = null;

if (apiMethod === 'objectPost' && request.headers['content-type'].includes('multipart/form-data')) {
bb = busboy({ headers: request.headers });
}

return async.waterfall([
next => {
if (apiMethod === 'objectPut' || apiMethod === 'objectPutPart') {
return next(null);
}
if (apiMethod === 'objectPost' && request.headers['content-type'].includes('multipart/form-data')) {
writeContinue(request, response);

let algoOK = false;
let credOK = false;
let dateOK = false;
let sigOK = false;
let policyOK = false;
request.formData = {};
bb.on('field', (fieldname, val) => {
request.formData[fieldname] = val;
if (request.formData.Policy) {
const decrypted = decryptToken(request.formData.Policy);
request.formData.decryptedPolicy = JSON.parse(decrypted);
}

// TODO - put content type field for file in request
if (fieldname === 'X-Amz-Algorithm') {
algoOK = true;
}
if (fieldname === 'X-Amz-Credential') {
credOK = true;
}
if (fieldname === 'X-Amz-Date') {
dateOK = true;
}
if (fieldname === 'X-Amz-Signature') {
sigOK = true;
}
if (fieldname === 'Policy') {
policyOK = true;
}
});

bb.on('file', (fieldname, file, filename, encoding, mimetype) => {

Check warning on line 351 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Expected to return a value at the end of this function
fileEventData = { fieldname, file, filename, encoding, mimetype };
if (algoOK && credOK && dateOK && sigOK && policyOK) {
return next(null);
}
});

bb.on('finish', () => {

Check warning on line 358 in lib/api/api.js

View workflow job for this annotation

GitHub Actions / linting-coverage 

Expected to return a value at the end of this function
// if authorization field is not found, return error
if (!algoOK || !credOK || !dateOK || !sigOK || !policyOK) {
return next(errors.InvalidRequest);
}
});
request.pipe(bb);

// parseMultipartFormData(request, (err, data) => {
// if (err) {
// return next(err);
// }
// fileEventData = data;
// return next(null);
// });
} else {
// issue 100 Continue to the client
writeContinue(request, response);
const MAX_POST_LENGTH = request.method === 'POST' ?
1024 * 1024 : 1024 * 1024 / 2; // 1 MB or 512 KB
const post = [];
let postLength = 0;
request.on('data', chunk => {
postLength += chunk.length;
// Sanity check on post length
if (postLength <= MAX_POST_LENGTH) {
post.push(chunk);
}
});

request.on('error', err => {
log.trace('error receiving request', {
error: err,
});
return next(errors.InternalError);
});

request.on('end', () => {
if (postLength > MAX_POST_LENGTH) {
log.error('body length is too long for request type',
{ postLength });
return next(errors.InvalidRequest);
}
request.post = Buffer.concat(post, postLength).toString();
return next(null);
});
}
return undefined;
},
next => auth.server.doAuth(
request, log, (err, userInfo, authorizationResults, streamingV4Params) => {
if (err) {
Expand All @@ -200,41 +418,7 @@
authNames.userName = userInfo.getIAMdisplayName();
}
log.addDefaultFields(authNames);
if (apiMethod === 'objectPut' || apiMethod === 'objectPutPart') {
return next(null, userInfo, authorizationResults, streamingV4Params);
}
// issue 100 Continue to the client
writeContinue(request, response);
const MAX_POST_LENGTH = request.method === 'POST' ?
1024 * 1024 : 1024 * 1024 / 2; // 1 MB or 512 KB
const post = [];
let postLength = 0;
request.on('data', chunk => {
postLength += chunk.length;
// Sanity check on post length
if (postLength <= MAX_POST_LENGTH) {
post.push(chunk);
}
});

request.on('error', err => {
log.trace('error receiving request', {
error: err,
});
return next(errors.InternalError);
});

request.on('end', () => {
if (postLength > MAX_POST_LENGTH) {
log.error('body length is too long for request type',
{ postLength });
return next(errors.InvalidRequest);
}
// Convert array of post buffers into one string
request.post = Buffer.concat(post, postLength).toString();
return next(null, userInfo, authorizationResults, streamingV4Params);
});
return undefined;
return next(null, userInfo, authorizationResults, streamingV4Params);
},
// Tag condition keys require information from CloudServer for evaluation
(userInfo, authorizationResults, streamingV4Params, next) => tagConditionKeyAuth(
Expand All @@ -244,6 +428,10 @@
apiMethod,
log,
(err, authResultsWithTags) => {
// TODO CLDSRV-527 remove ignore for POST object here
if (apiMethod === 'objectPost') {
return next(null, userInfo, authorizationResults, streamingV4Params);
}
if (err) {
log.trace('tag authentication error', { error: err });
return next(err);
Expand Down Expand Up @@ -271,6 +459,13 @@
return acc;
}, {});
}
if (apiMethod === 'objectPost' && fileEventData) {
request._response = response;
request.file = fileEventData.file;
request.fileEventData = fileEventData;
return this[apiMethod](userInfo, request, streamingV4Params,
log, callback, authorizationResults);
}
if (apiMethod === 'objectPut' || apiMethod === 'objectPutPart') {
request._response = response;
return this[apiMethod](userInfo, request, streamingV4Params,
Expand Down Expand Up @@ -337,6 +532,7 @@
objectCopy,
objectHead,
objectPut,
objectPost,
objectPutACL,
objectPutLegalHold,
objectPutTagging,
Expand Down
4 changes: 4 additions & 0 deletions lib/api/apiUtils/object/createAndStoreObject.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,10 @@ function createAndStoreObject(bucketName, bucketMD, objectKey, objMD, authInfo,
metadataStoreParams.contentMD5 = constants.emptyFileMd5;
return next(null, null, null);
}
if (request.apiMethod === 'objectPost') {
return dataStore(objectKeyContext, cipherBundle, request.file, size,
streamingV4Params, backendInfo, log, next);
}
return dataStore(objectKeyContext, cipherBundle, request, size,
streamingV4Params, backendInfo, log, next);
},
Expand Down
2 changes: 1 addition & 1 deletion lib/api/apiUtils/object/prepareStream.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ const V4Transform = require('../../../auth/streamingV4/V4Transform');
* the type of request requires them
*/
function prepareStream(stream, streamingV4Params, log, errCb) {
if (stream.headers['x-amz-content-sha256'] ===
if (stream.headers && stream.headers['x-amz-content-sha256'] ===
'STREAMING-AWS4-HMAC-SHA256-PAYLOAD') {
if (typeof streamingV4Params !== 'object') {
// this might happen if the user provided a valid V2
Expand Down
Loading
Loading