Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement/cldsrv 553 bump deps #5711

Open
wants to merge 18 commits into
base: development/9.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/setup-ci/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ runs:
mkdir -p /tmp/artifacts/${JOB_NAME}/;
- uses: actions/setup-node@v4
with:
node-version: '16'
node-version: '22'
cache: 'yarn'
- name: install typescript
shell: bash
Expand Down
70 changes: 35 additions & 35 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ jobs:
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '16'
node-version: '22'
cache: yarn
- name: install typescript
shell: bash
Expand Down Expand Up @@ -106,7 +106,7 @@ jobs:
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '16'
node-version: '22'
cache: yarn
- name: install typescript
shell: bash
Expand Down Expand Up @@ -149,7 +149,7 @@ jobs:
if: always()

build:
runs-on: ubuntu-20.04
runs-on: ubuntu-latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

best to specify the version explicitely (24.04)

permissions:
contents: read
packages: write
Expand Down Expand Up @@ -361,38 +361,38 @@ jobs:
source: /tmp/artifacts
if: always()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be skipped instead of removed:

Suggested change
utapi-v2-tests:
if: false

utapi-v2-tests:
runs-on: ubuntu-latest
needs: build
env:
ENABLE_UTAPI_V2: t
S3BACKEND: mem
BUCKET_DENY_FILTER: utapi-event-filter-deny-bucket
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
JOB_NAME: ${{ github.job }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup CI environment
uses: ./.github/actions/setup-ci
- name: Setup CI services
run: docker compose up -d
working-directory: .github/docker
- name: Run file utapi v2 tests
run: |-
set -ex -o pipefail;
bash wait_for_local_port.bash 8000 40
yarn run test_utapi_v2 | tee /tmp/artifacts/${{ github.job }}/tests.log
- name: Upload logs to artifacts
uses: scality/action-artifacts@v4
with:
method: upload
url: https://artifacts.scality.net
user: ${{ secrets.ARTIFACTS_USER }}
password: ${{ secrets.ARTIFACTS_PASSWORD }}
source: /tmp/artifacts
if: always()
# utapi-v2-tests:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to skip the job than to comment it:

Suggested change
# utapi-v2-tests:
utapi-v2-tests:
if: false
[...]

# runs-on: ubuntu-latest
# needs: build
# env:
# ENABLE_UTAPI_V2: t
# S3BACKEND: mem
# BUCKET_DENY_FILTER: utapi-event-filter-deny-bucket
# CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
# MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
# JOB_NAME: ${{ github.job }}
# steps:
# - name: Checkout
# uses: actions/checkout@v4
# - name: Setup CI environment
# uses: ./.github/actions/setup-ci
# - name: Setup CI services
# run: docker compose up -d
# working-directory: .github/docker
# - name: Run file utapi v2 tests
# run: |-
# set -ex -o pipefail;
# bash wait_for_local_port.bash 8000 40
# yarn run test_utapi_v2 | tee /tmp/artifacts/${{ github.job }}/tests.log
# - name: Upload logs to artifacts
# uses: scality/action-artifacts@v4
# with:
# method: upload
# url: https://artifacts.scality.net
# user: ${{ secrets.ARTIFACTS_USER }}
# password: ${{ secrets.ARTIFACTS_PASSWORD }}
# source: /tmp/artifacts
# if: always()

quota-tests:
runs-on: ubuntu-latest
Expand Down
4 changes: 3 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ARG NODE_VERSION=16.20.2-bookworm-slim
ARG NODE_VERSION=22.4.0-bookworm-slim

FROM node:${NODE_VERSION} AS builder

Expand All @@ -22,8 +22,10 @@ RUN apt-get update \
&& ssh-keyscan -H github.com > /root/ssh/known_hosts

ENV PYTHON=python3
RUN npm install -g node-gyp
COPY package.json yarn.lock /usr/src/app/
RUN npm install [email protected] -g

Comment on lines +25 to 27
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

best to install all "global" packages in a single step,
before package.json copy (to avoid invalidatoin),
and specifying the node-gyp version (just like we do for typescript)

Suggested change
RUN npm install -g node-gyp
COPY package.json yarn.lock /usr/src/app/
RUN npm install [email protected] -g
RUN npm install -g \
[email protected] \
[email protected]
COPY package.json yarn.lock /usr/src/app/

RUN yarn install --production --ignore-optional --frozen-lockfile --ignore-engines --network-concurrency 1

################################################################################
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
[![Docker Pulls][badgedocker]](https://hub.docker.com/r/zenko/cloudserver)
[![Docker Pulls][badgetwitter]](https://twitter.com/zenko)

## Build Status

![Public Build Status][badgepub]
![Private Build Status][badgepriv]

## Overview

CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible
Expand Down
2 changes: 1 addition & 1 deletion bin/create_bucket_with_NFS_enabled.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/usr/bin/env node
'use strict'; // eslint-disable-line strict
'use strict';

require('../lib/nfs/utilities.js').createBucketWithNFSEnabled();
2 changes: 1 addition & 1 deletion bin/create_encrypted_bucket.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/sh
// 2>/dev/null ; exec "$(which nodejs || which node)" "$0" "$@"
'use strict'; // eslint-disable-line strict
'use strict';

require('../lib/kms/utilities.js').createEncryptedBucket();
2 changes: 1 addition & 1 deletion bin/list_bucket_metrics.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/usr/bin/env node
'use strict'; // eslint-disable-line strict
'use strict';

require('../lib/utapi/utilities.js').listMetrics('buckets');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should keep this file, and just add early return (if needed) in listMetrics()

2 changes: 1 addition & 1 deletion bin/list_metrics.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/usr/bin/env node
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should keep this file, and just add early return (if needed) in listMetrics()

'use strict'; // eslint-disable-line strict
'use strict';

require('../lib/utapi/utilities.js').listMetrics();
2 changes: 1 addition & 1 deletion bin/metrics_server.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env node
'use strict'; // eslint-disable-line strict
'use strict';

const {
startWSManagementClient,
Expand Down
2 changes: 1 addition & 1 deletion bin/search_bucket.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
// 2>/dev/null ; exec "$(which nodejs 2>/dev/null || which node)" "$0" "$@"
'use strict'; // eslint-disable-line strict
'use strict';

const { auth } = require('arsenal');
const commander = require('commander');
Expand Down
2 changes: 1 addition & 1 deletion bin/secure_channel_proxy.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env node
'use strict'; // eslint-disable-line strict
'use strict';

const {
startWSManagementClient,
Expand Down
12 changes: 6 additions & 6 deletions config.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,27 +67,27 @@
"allowFrom": ["127.0.0.1/8", "::1"]
},
"metadataClient": {
"host": "localhost",
"host": "127.0.0.1",
"port": 9990
},
"dataClient": {
"host": "localhost",
"host": "127.0.0.1",
"port": 9991
},
"pfsClient": {
"host": "localhost",
"host": "127.0.0.1",
"port": 9992
},
"metadataDaemon": {
"bindAddress": "localhost",
"bindAddress": "127.0.0.1",
"port": 9990
},
"dataDaemon": {
"bindAddress": "localhost",
"bindAddress": "127.0.0.1",
"port": 9991
},
"pfsDaemon": {
"bindAddress": "localhost",
"bindAddress": "127.0.0.1",
"port": 9992
},
"recordLog": {
Expand Down
4 changes: 2 additions & 2 deletions constants.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
const crypto = require('crypto');
const mycrypto = require('crypto');

const constants = {
/*
Expand Down Expand Up @@ -90,7 +90,7 @@ const constants = {
maxHttpHeadersSize: 14122,

// hex digest of sha256 hash of empty string:
emptyStringHash: crypto.createHash('sha256')
emptyStringHash: mycrypto.createHash('sha256')
.update('', 'binary').digest('hex'),

// Queries supported by AWS that we do not currently support.
Expand Down
2 changes: 1 addition & 1 deletion dataserver.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
'use strict'; // eslint-disable-line strict
'use strict';

const arsenal = require('arsenal');
const { config } = require('./lib/Config.js');
Expand Down
38 changes: 28 additions & 10 deletions .eslintrc → eslint.config.mjs
Original file line number Diff line number Diff line change
@@ -1,9 +1,28 @@
{
"extends": "scality",
"plugins": [
"mocha"
],
"rules": {
import mocha from "eslint-plugin-mocha";
import path from "node:path";
import { fileURLToPath } from "node:url";
import js from "@eslint/js";
import { FlatCompat } from "@eslint/eslintrc";

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const compat = new FlatCompat({
baseDirectory: __dirname,
recommendedConfig: js.configs.recommended,
allConfig: js.configs.all
});

export default [...compat.extends("scality"), {
plugins: {
mocha,
},

languageOptions: {
ecmaVersion: 2020,
sourceType: "script",
},

rules: {
"import/extensions": "off",
"lines-around-directive": "off",
"no-underscore-dangle": "off",
Expand Down Expand Up @@ -48,7 +67,6 @@
"quote-props": "off",
"mocha/no-exclusive-tests": "error",
},
"parserOptions": {
"ecmaVersion": 2020
}
}
}];


2 changes: 1 addition & 1 deletion index.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
'use strict'; // eslint-disable-line strict
'use strict';

require('werelogs').stderrUtils.catchAndTimestampStderr(
undefined,
Expand Down
24 changes: 11 additions & 13 deletions lib/Config.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
const fs = require('fs');
const path = require('path');
const url = require('url');
const crypto = require('crypto');
const mycrypto = require('crypto');
Copy link
Contributor

@francoisferrand francoisferrand Dec 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why rename?
can't we use node22's new built-in crypto module?


const { v4: uuidv4 } = require('uuid');
const cronParser = require('cron-parser');
Expand All @@ -13,14 +13,11 @@
const validateAuthConfig = arsenalAuth.inMemory.validateAuthConfig;
const { buildAuthDataAccount } = require('./auth/in_memory/builder');
const validExternalBackends = require('../constants').externalBackends;
const {
azureAccountNameRegex,
base64Regex,
allowedUtapiEventFilterFields,
allowedUtapiEventFilterStates,
supportedLifecycleRules,
const { azureAccountNameRegex, base64Regex,
allowedUtapiEventFilterFields, allowedUtapiEventFilterStates,
supportedLifecycleRules
} = require('../constants');
Comment on lines +16 to 19
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change not needed?

const { utapiVersion } = require('utapi');
// const { utapiVersion } = require('utapi');
const { scaleMsPerDay } = s3middleware.objectUtils;

const constants = require('../constants');
Expand Down Expand Up @@ -1236,7 +1233,7 @@
maxStaleness,
enableInflights,
};
if (config.utapi) {
benzekrimaha marked this conversation as resolved.
Show resolved Hide resolved
if (config.utapi && false) { // eslint-disable-line no-constant-condition
Copy link
Contributor

@francoisferrand francoisferrand Dec 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to change, will work in case we don't have utapi?

this.utapi = { component: 's3' };
if (config.utapi.host) {
assert(typeof config.utapi.host === 'string',
Expand All @@ -1249,7 +1246,7 @@
'bad config: utapi port must be a positive integer');
this.utapi.port = config.utapi.port;
}
if (utapiVersion === 1) {
if (utapiVersion === 1) { // eslint-disable-line no-undef

Check warning on line 1249 in lib/Config.js

View check run for this annotation

Codecov / codecov/patch

lib/Config.js#L1249

Added line #L1249 was not covered by tests
if (config.utapi.workers !== undefined) {
assert(Number.isInteger(config.utapi.workers)
&& config.utapi.workers > 0,
Expand Down Expand Up @@ -1349,7 +1346,7 @@
}
}

if (utapiVersion === 2 && config.utapi.filter) {
if (utapiVersion === 2 && config.utapi.filter) { // eslint-disable-line no-undef

Check warning on line 1349 in lib/Config.js

View check run for this annotation

Codecov / codecov/patch

lib/Config.js#L1349

Added line #L1349 was not covered by tests
const { filter: filterConfig } = config.utapi;
const utapiResourceFilters = {};
allowedUtapiEventFilterFields.forEach(
Expand All @@ -1368,8 +1365,9 @@
this.utapi.filter = utapiResourceFilters;
}
}
// eslint-disable-next-line no-constant-condition
if (Object.keys(this.locationConstraints).some(
loc => this.locationConstraints[loc].sizeLimitGB)) {
loc => this.locationConstraints[loc].sizeLimitGB) && false) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to 'fase' this, we should never run into this code anyway

assert(this.utapi && this.utapi.metrics &&
this.utapi.metrics.includes('location'),
'bad config: if storage size limit set on a location ' +
Expand Down Expand Up @@ -1971,7 +1969,7 @@
}

setPublicInstanceId(instanceId) {
this.publicInstanceId = crypto.createHash('sha256')
this.publicInstanceId = mycrypto.createHash('sha256')
.update(instanceId)
.digest('hex');
}
Expand Down
3 changes: 0 additions & 3 deletions lib/api/api.js
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,9 @@ auth.setHandler(vault);
const api = {
callApiMethod(apiMethod, request, response, log, callback) {
// Attach the apiMethod method to the request, so it can used by monitoring in the server
// eslint-disable-next-line no-param-reassign
request.apiMethod = apiMethod;
// Array of end of API callbacks, used to perform some logic
// at the end of an API.
// eslint-disable-next-line no-param-reassign
request.finalizerHooks = [];

const actionLog = monitoringMap[apiMethod];
Expand Down Expand Up @@ -149,7 +147,6 @@ const api = {
// Extract all the _apiMethods and store them in an array
const apiMethods = requestContexts ? requestContexts.map(context => context._apiMethod) : [];
// Attach the names to the current request
// eslint-disable-next-line no-param-reassign
request.apiMethods = apiMethods;

function checkAuthResults(authResults) {
Expand Down
Loading
Loading