Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate a single caching hash #60

Merged
merged 10 commits into from
Jul 30, 2024
Merged

generate a single caching hash #60

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c2fa6a3
generate a single caching hash
rarmatei Jul 30, 2024
8b2f446
add backwards compatibility
rarmatei Jul 30, 2024
5b22344
print out key info
rarmatei Jul 30, 2024
e24d373
add cache step test setup
rarmatei Jul 30, 2024
32adc70
add cache step test setup
rarmatei Jul 30, 2024
d668014
add tests for caching step
rarmatei Jul 30, 2024
f31f728
build source files
rarmatei Jul 30, 2024
fa694d4
improve messaging
rarmatei Jul 30, 2024
5768122
commit changed source files
rarmatei Jul 30, 2024
784fa9f
add comment
rarmatei Jul 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions workflow-steps/cache/hashing-utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import * as string_decoder from 'string_decoder';

const fs = require('fs');
const crypto = require('crypto');
import { glob } from 'glob';
Expand All @@ -11,14 +13,8 @@ function hashFileContents(pattern: string) {
let megaHash = '';
files.forEach((file) => {
const fileContent = fs.readFileSync(file);
const fileHash = crypto
.createHash('sha256')
.update(fileContent)
.digest('hex');
megaHash = crypto
.createHash('sha256')
.update(fileHash + megaHash)
.digest('hex');
const fileHash = hash(fileContent);
megaHash = hash(fileHash + megaHash);
});
return megaHash;
}
Expand All @@ -40,5 +36,14 @@ export function hashKey(key: string): string {
const globHashes = globsToHash.map((globPattern) => {
return hashFileContents(globPattern);
});
return [...hardcodedKeys, ...globHashes].join(' | ');
const hashCollections = [...hardcodedKeys, ...globHashes];

if (hashCollections.length > 1) {
return hash(hashCollections.join(' | '));
}
return hashCollections.join(' | ');
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could've just always returned

return hash(hashCollections.join(' | '));

but I kept the old way of hashing when it's a single-key for backward compatibility purposes - so we don't bust everyone's cache when this gets merged in

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave that in a comment in the code?

}

function hash(input: string) {
return crypto.createHash('sha256').update(input).digest('hex');
}
13 changes: 10 additions & 3 deletions workflow-steps/cache/output/main.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5959,8 +5959,8 @@ function hashFileContents(pattern) {
let megaHash = "";
files.forEach((file) => {
const fileContent = fs.readFileSync(file);
const fileHash = crypto.createHash("sha256").update(fileContent).digest("hex");
megaHash = crypto.createHash("sha256").update(fileHash + megaHash).digest("hex");
const fileHash = hash(fileContent);
megaHash = hash(fileHash + megaHash);
});
return megaHash;
}
Expand All @@ -5978,7 +5978,14 @@ function hashKey(key2) {
const globHashes = globsToHash.map((globPattern) => {
return hashFileContents(globPattern);
});
return [...hardcodedKeys, ...globHashes].join(" | ");
const hashCollections = [...hardcodedKeys, ...globHashes];
if (hashCollections.length > 1) {
return hash(hashCollections.join(" | "));
}
return hashCollections.join(" | ");
}
lourw marked this conversation as resolved.
Show resolved Hide resolved
function hash(input) {
return crypto.createHash("sha256").update(input).digest("hex");
}

// main.ts
Expand Down
13 changes: 10 additions & 3 deletions workflow-steps/cache/output/post.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5947,8 +5947,8 @@ function hashFileContents(pattern) {
let megaHash = "";
files.forEach((file) => {
const fileContent = fs.readFileSync(file);
const fileHash = crypto.createHash("sha256").update(fileContent).digest("hex");
megaHash = crypto.createHash("sha256").update(fileHash + megaHash).digest("hex");
const fileHash = hash(fileContent);
megaHash = hash(fileHash + megaHash);
});
return megaHash;
}
Expand All @@ -5966,7 +5966,14 @@ function hashKey(key) {
const globHashes = globsToHash.map((globPattern) => {
return hashFileContents(globPattern);
});
return [...hardcodedKeys, ...globHashes].join(" | ");
const hashCollections = [...hardcodedKeys, ...globHashes];
if (hashCollections.length > 1) {
return hash(hashCollections.join(" | "));
}
return hashCollections.join(" | ");
}
lourw marked this conversation as resolved.
Show resolved Hide resolved
function hash(input) {
return crypto.createHash("sha256").update(input).digest("hex");
}

// post.ts
Expand Down