Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add blob storage #2287

Merged
merged 221 commits into from
Oct 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
221 commits
Select commit Hold shift + click to select a range
68908b0
feat: adding inital blob setup
taty2010 Aug 11, 2023
3fe87c5
feat: added get and set for ssr files
taty2010 Aug 11, 2023
2fe2c83
chore: prettier
taty2010 Aug 11, 2023
407da47
chore: prettier
taty2010 Aug 11, 2023
01f44e3
updated tsconfig and added dynamic imports
taty2010 Aug 23, 2023
a75b428
chore: reverted some changes to bring project back to commonjs
nickytonline Aug 29, 2023
79acf0b
chore: some cleanup
nickytonline Aug 29, 2023
7db5872
chore: ran formatter
nickytonline Aug 29, 2023
8b34135
chore: more cleanup related to going back to commonjs
nickytonline Aug 29, 2023
a6941f2
chore: reverting some more stuff from async
nickytonline Aug 29, 2023
90333d4
chore: getting blob storage in a usable state in commonjs land
nickytonline Aug 29, 2023
bba2391
chore: testing blob storage
nickytonline Aug 29, 2023
a9faf28
chore: memoized getting blob storage
nickytonline Aug 29, 2023
a20f8c4
chore: small refactor
nickytonline Aug 30, 2023
8c769af
chore: trigger build
nickytonline Aug 30, 2023
271624d
chore: cleaned up blob storage instantiation
nickytonline Aug 30, 2023
7361ef8
chore: fixed types for getting blob storage
nickytonline Aug 31, 2023
4497b56
chore: removed todo comment
nickytonline Aug 31, 2023
ae7f9b6
chore: removed a TODO comment
nickytonline Aug 31, 2023
58a62b3
chore: refactored getBlobStorage
nickytonline Aug 31, 2023
4d45ddf
chore: putting up test for Eduardo
nickytonline Sep 5, 2023
110816e
chore: add debug logging
eduardoboucas Sep 5, 2023
171b5a7
refactor: rebuild
eduardoboucas Sep 5, 2023
9f47f42
refactor: remove debug logging
eduardoboucas Sep 5, 2023
1aa8ab1
chore: wip
nickytonline Sep 5, 2023
6e1b151
chore: updated package lock
nickytonline Sep 5, 2023
e3b5a74
chore: testing out blob storage for prerendered content
nickytonline Sep 6, 2023
a30ffba
chore: added a check for deploy ID for blob storage
nickytonline Sep 6, 2023
4a509fe
chore: small refactor
nickytonline Sep 6, 2023
048a0c2
chore: reverted some refactor not related to the PR
nickytonline Sep 6, 2023
d856aec
chore: undid encoding of blob storage keys
nickytonline Sep 7, 2023
0725388
chore: removed unnecessary comment
nickytonline Sep 7, 2023
3604f11
chore: added some logging for blob storage
nickytonline Sep 7, 2023
893b5ee
chore: testing/logging blob storage
nickytonline Sep 7, 2023
26346da
chore: stripping starting slash for blob storage keys
nickytonline Sep 7, 2023
6feaac3
chore: using netliblob.setJSON now instead of .set
nickytonline Sep 8, 2023
913ba1b
chore: cleaned up setting blob storage
nickytonline Sep 8, 2023
b464472
chore: fixed issue with ssrDependencies not being iterable
nickytonline Sep 8, 2023
6d4f1be
chore: some logging cleanup
nickytonline Sep 8, 2023
9e44536
chore: reverted some unnecessary changes from 01f44e391c0368e4488ef02…
nickytonline Sep 8, 2023
c2dc32d
chore: reverted some unnecessary changes from 01f44e391c0368e4488ef02…
nickytonline Sep 8, 2023
e6eb363
chore: reverted some unnecessary changes
nickytonline Sep 8, 2023
1639999
chore: reverted change not related to work in PR
nickytonline Sep 8, 2023
ed3cb71
chore: updated package lock
nickytonline Sep 8, 2023
183b7aa
chore: added a comment about starting slashes removed from blob stora…
nickytonline Sep 8, 2023
391feaf
chore: added some more comments
nickytonline Sep 8, 2023
e664884
chore: trigger build
nickytonline Sep 8, 2023
cc866a1
chore: removed logging feature flags
nickytonline Sep 8, 2023
dfc5f21
chore: testing blob storage flag
nickytonline Sep 11, 2023
946dac8
chore: cleaned up blob storage check
nickytonline Sep 12, 2023
bce39ce
chore: trigger build
nickytonline Sep 12, 2023
217bd9a
chore: added some temp logging
nickytonline Sep 12, 2023
5af0a8a
Revert "chore: added some temp logging"
nickytonline Sep 12, 2023
dc26736
chore: fixed blob storage check condition 🙃
nickytonline Sep 12, 2023
eb469b9
chore: bumped @netlify/build to 29.20.15
nickytonline Sep 12, 2023
f316e5f
chore: trigger build
nickytonline Sep 12, 2023
0a97c3b
feat: implemented incremental cache
nickytonline Aug 30, 2023
18fc31b
chore: some cleanup
nickytonline Aug 31, 2023
aae443c
chore: removed a TODO comment
nickytonline Aug 31, 2023
f0e5f6d
chore: wip
nickytonline Aug 31, 2023
a9eac3b
chore: refactored getBlobStorage
nickytonline Aug 31, 2023
d1d6d00
chore: added incremental cache with memory cache for testing key and …
nickytonline Sep 5, 2023
609945e
add new incremental cache file + store blob
taty2010 Sep 8, 2023
7edf24c
removing blob type for now
taty2010 Sep 8, 2023
55a5699
updated file location for incremental-cache.js
taty2010 Sep 8, 2023
3ef276b
updated filepath
taty2010 Sep 8, 2023
cd15334
adding publish dir intead
taty2010 Sep 8, 2023
4dd2961
use publish dir for config
taty2010 Sep 8, 2023
6fd1266
feat: moving incremental cache to functions
taty2010 Sep 10, 2023
9a2f9b3
chore: creating cache file right before config set
taty2010 Sep 10, 2023
f1cb1e3
chore: added require.resolve for incremental cache
taty2010 Sep 10, 2023
98a2bc2
chore: uncomment import that got commented out during merge
nickytonline Sep 12, 2023
4a5cfb5
chore: no longer infer the type for Blobs
nickytonline Sep 12, 2023
a5f66b7
chore: updated @netlify/build to 29.21.0
nickytonline Sep 13, 2023
ce8c746
Merge remote-tracking branch 'origin' into nickytonline/blob-storage
nickytonline Sep 13, 2023
eb69bcc
chore: formatting fixes
nickytonline Sep 13, 2023
5eb29b7
chore: cleaned up how netliblob is determined for SSR lambdas
nickytonline Sep 14, 2023
cc41ab3
chore: now incremental cache handler gets generated if blob storage i…
nickytonline Sep 14, 2023
4059344
chore: fixed resolving file path for blobStorage.js
nickytonline Sep 14, 2023
da0edf3
chore: trigger build
nickytonline Sep 14, 2023
296f50c
chore: cleaned up getting blob credentials from context
nickytonline Sep 14, 2023
9863fa8
chore: added some temporary logging
nickytonline Sep 14, 2023
fbf1952
chore: added another log statement in incremental cache handler
nickytonline Sep 14, 2023
75d17c3
chore: trigger build
nickytonline Sep 14, 2023
f0c92dc
chore: stripped starting slash for keys for now
nickytonline Sep 15, 2023
4b75d4e
chore: changed where incremental cache handler gets put
nickytonline Sep 15, 2023
6e88f67
chore: removed test incremental cache handler from demo site
nickytonline Sep 15, 2023
d54e062
Merge remote-tracking branch 'origin' into nickytonline/blob-storage
nickytonline Sep 15, 2023
30075f7
chore: update
lukasholzer Sep 20, 2023
f7f511f
chore: update
lukasholzer Sep 20, 2023
f7147d7
chore: updates
lukasholzer Sep 21, 2023
eb8674f
chore: make blob storage available during runtime
lukasholzer Sep 21, 2023
78a1a08
chore: update
lukasholzer Sep 21, 2023
a7f6696
chore: use the blob store during request time differently
lukasholzer Sep 21, 2023
913a6d2
chore: update
lukasholzer Sep 21, 2023
b1a4dad
chore: test
lukasholzer Sep 22, 2023
8dfab95
chore: update logging
lukasholzer Sep 22, 2023
2cc9633
chore: update
lukasholzer Sep 22, 2023
43860c6
chore: update
lukasholzer Sep 22, 2023
500e462
chore: update
lukasholzer Sep 22, 2023
6e56190
chore: fix
lukasholzer Sep 22, 2023
54fa413
chore: update
lukasholzer Sep 22, 2023
1538fec
chore: improve logging
lukasholzer Sep 22, 2023
3be1496
chore: update
lukasholzer Sep 22, 2023
da170da
chore: update
lukasholzer Sep 22, 2023
3a0a011
chore: update
lukasholzer Sep 22, 2023
b9ff1a6
chore: update
lukasholzer Sep 22, 2023
b7946c1
chore: change encoding of key
lukasholzer Sep 22, 2023
ef2ec91
refactor: remove incremental cache handler injection
pieh Sep 25, 2023
7c91a71
refactor: eslint ignore file via .eslintignore and not config
pieh Sep 25, 2023
1b40b4b
feat: bump @netlify/blobs to 2.0.0 (key hashing)
pieh Sep 25, 2023
2d13071
refactor: allow @netlify/blobs to do the key hashing
pieh Sep 25, 2023
18adedc
refactor: some debugging
pieh Sep 26, 2023
bad1b2b
test: will this be set?
pieh Sep 26, 2023
4849ec8
test: will this be set?2
pieh Sep 26, 2023
2375994
test: will this be set?3
pieh Sep 26, 2023
65875bd
test: will this be set?4
pieh Sep 26, 2023
2664912
test: will this be set?5
pieh Sep 26, 2023
a118b9e
test: will this be set?6
pieh Sep 26, 2023
299418c
test: will this be set?7
pieh Sep 26, 2023
0e4a3b7
test: will this be set?8
pieh Sep 26, 2023
41cade6
test: will this be set?9
pieh Sep 26, 2023
fa32780
fix: lambda response is base64 encoded
pieh Sep 26, 2023
7b270ad
test: check how things look in moveStaticPages
pieh Sep 26, 2023
de16614
test: check how things look in moveStaticPages2
pieh Sep 26, 2023
200738f
test: check how things look in moveStaticPages3
pieh Sep 26, 2023
20aff69
test: check if we can skip key normalization already
pieh Sep 27, 2023
fb5fd25
fix: don't compile blob helpers for function bundling, copy files ins…
pieh Sep 27, 2023
663b343
feat: cleanup
pieh Sep 27, 2023
b5f9a51
fix: clean up lambda a bit
pieh Sep 27, 2023
f1201dd
fix: clean up some logs when used locally
pieh Sep 27, 2023
dffcc01
fix: move esbuild back to deps, as it's used for middleware compilation
pieh Sep 27, 2023
659f693
feat: don't warm up blobs when configuring functions (we will do this…
pieh Sep 27, 2023
08008f2
fix: more cleanup
pieh Sep 27, 2023
4ccfd0e
feat: start uploading to blob store during static pages moves
pieh Sep 27, 2023
ae664e8
feat: log number of assets moved to blob storage and delete those files
pieh Sep 27, 2023
0506a30
test: check simple blob keys ... again
pieh Sep 27, 2023
b6ee3a4
test: restore key normalization ... again
pieh Sep 27, 2023
5ab0b49
feat: apply trailing slash to blob keys
pieh Sep 27, 2023
a17498a
feat: initial default locale handling
pieh Sep 27, 2023
ebb3abd
chore: some logs
pieh Sep 27, 2023
4226d39
fix: commit exported isLocalized
pieh Sep 27, 2023
49519fc
fix: fix isLocalized import
pieh Sep 27, 2023
bc5afd3
chore: expose router push
pieh Sep 27, 2023
e0449b5
feat: blob manifest for i18n aliases
pieh Sep 28, 2023
6cb125c
chore: log blob manifest
pieh Sep 28, 2023
d54f824
fix: fix blobs-manifest filenames
pieh Sep 28, 2023
297320e
chore: a bit more type safety
pieh Sep 28, 2023
468ecc3
chore: bump cypress to latest v12
pieh Sep 28, 2023
d6b8646
test: test fix1
pieh Sep 28, 2023
dcc1e77
chore: drop some test code
pieh Sep 28, 2023
dec7888
fix: remove blobs-manifest before publishing to cdn
pieh Sep 28, 2023
e62e879
Merge remote-tracking branch 'origin/main' into nickytonline/blob-sto…
pieh Sep 28, 2023
248dea6
feat: initial headers setting
pieh Sep 28, 2023
5f46053
feat: initial headers setting2
pieh Sep 28, 2023
9b150af
chore: some more visible rsc-data logs
pieh Sep 29, 2023
e41a370
chore: some more visible rsc-data logs2
pieh Sep 29, 2023
aa3c09d
chore: some more visible rsc-data logs3
pieh Sep 29, 2023
edefc1b
fix: normalize path after blob store checks
pieh Sep 29, 2023
310a547
fix: strip trailing slash for .rsc/ paths before checking blob store
pieh Sep 29, 2023
293ed48
test: log maybe more details
pieh Sep 29, 2023
393a92e
test: maybe i will get more details without --json?
pieh Sep 29, 2023
1d24c2b
test: inject global fetch polyfill
pieh Sep 29, 2023
3275332
test: inject global fetch polyfill2
pieh Sep 29, 2023
1d840ae
test: restore json output
pieh Sep 29, 2023
24956d1
test: remove logs from fetch polyfill that were breaking ntl deploy o…
pieh Sep 29, 2023
9c71569
chore: some cleanup
pieh Sep 29, 2023
c815d64
chore: cleanup lambda handler
pieh Sep 29, 2023
f3a5e0d
chore: drop mime-types dep as it's not used
pieh Sep 29, 2023
69071d5
Merge remote-tracking branch 'origin/main' into nickytonline/blob-sto…
pieh Sep 29, 2023
b5ff0ab
test: let's see basepath
pieh Sep 29, 2023
c5687c6
chore: revert basepath for now
pieh Oct 2, 2023
189bc12
chore: try out slightly different approach
pieh Oct 2, 2023
ef67939
chore: show time on index page
pieh Oct 2, 2023
ed18202
chore: fix var name
pieh Oct 2, 2023
25a45bd
chore: set to array fix
pieh Oct 2, 2023
bb643a2
test: try pulling from blob store in fs augment
pieh Oct 2, 2023
5320e7f
test: maybe fix blob type
pieh Oct 2, 2023
b4d343e
chore: stuff
pieh Oct 2, 2023
3662003
chore: stuff2
pieh Oct 2, 2023
e050866
chore: add deployID to logs
pieh Oct 2, 2023
ad426cb
chore: log missed blobs
pieh Oct 2, 2023
1fe2413
chore: log filepath
pieh Oct 2, 2023
4ee7225
chore: is cached file being read?
pieh Oct 2, 2023
6a25768
chore: is blobbing erroring out?
pieh Oct 2, 2023
219215b
chore: ensure directory stuff
pieh Oct 2, 2023
1ae3e29
chore: oneline log for easier grepping
pieh Oct 2, 2023
811daf7
chore: more oneliners
pieh Oct 2, 2023
dcd1c76
chore: don't use asynclocalstorage ...!
pieh Oct 2, 2023
45a5548
chore: log served message
pieh Oct 2, 2023
a40be22
chore: log served message2
pieh Oct 2, 2023
7c67874
chore: what if we use asynclocalstorage later
pieh Oct 3, 2023
4868f7e
chore: cleanups
pieh Oct 3, 2023
ed2843f
chore: cleanups2
pieh Oct 4, 2023
542642b
chore: cleanups3
pieh Oct 4, 2023
108dca6
chore: cleanups4
pieh Oct 4, 2023
bc75e8f
chore: cleanups5
pieh Oct 4, 2023
686050a
chore: cleanups6
pieh Oct 4, 2023
05fd540
Merge remote-tracking branch 'origin/main' into blob-test-variants
pieh Oct 4, 2023
2489637
chore: cleanups7
pieh Oct 4, 2023
548b092
chore: test basePath
pieh Oct 4, 2023
57e9bb3
chore: test basePath, no i18n
pieh Oct 4, 2023
5d3d456
chore: test no basePath no i18n
pieh Oct 4, 2023
4850a9d
chore: restore default demo config
pieh Oct 4, 2023
a81452b
chore: cleanups8
pieh Oct 4, 2023
84d020b
test: fix unit tests, maybe fix API functions
pieh Oct 4, 2023
4662234
test: maybe fix API functions2
pieh Oct 4, 2023
316ca06
chore: try to make fallback:true act as fallback:blocking
pieh Oct 4, 2023
30cdeb0
chore: fix some logic in fallback: blocking forcing and add comments
pieh Oct 4, 2023
01d3058
chore: update test comments
pieh Oct 4, 2023
3074b31
chore: move some imports so comment make sense
pieh Oct 5, 2023
b38586d
chore: strip node: from compiled @netlify/blobs
pieh Oct 5, 2023
d5b818a
chore: restore revalidation on non-first odb request
pieh Oct 5, 2023
4027de1
chore: add a feature flag for using blobs for isr pages
pieh Oct 10, 2023
c9e173c
chore: sync package-lock.json
pieh Oct 10, 2023
b704128
Merge remote-tracking branch 'origin/main' into nickytonline/blob-sto…
pieh Oct 10, 2023
239e044
fix: import type from package entry and not internals
pieh Oct 10, 2023
563a189
Merge remote-tracking branch 'origin/main' into nickytonline/blob-sto…
pieh Oct 10, 2023
3f3deb1
Merge remote-tracking branch 'origin/main' into nickytonline/blob-sto…
pieh Oct 12, 2023
12c4723
chore: bump @netlify/blobs, don't build CJS version anymore
pieh Oct 12, 2023
ff4d8c5
Merge remote-tracking branch 'origin/main' into nickytonline/blob-sto…
pieh Oct 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions cypress/e2e/default/dynamic-routes.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,13 @@ describe('Dynamic Routing', () => {
expect(res.body).to.contain('Under the Dome')
})
})
it('renders fallback page via ODB on a non-prerendered dynamic route with fallback: true', () => {
it('does not render fallback page via ODB on a non-prerendered dynamic route with fallback: true', () => {
// unfortunately there is a problem with `fallback: true` in ODB context - the fallback would be cached indefinitely
// so visits to those pages would always render the fallback first and browser client would later re-render with correct
// content. As this is not ideal the `fallback: true` is treated as `fallback: blocking`.
cy.request({ url: '/getStaticProps/withFallback/3/', headers: { 'x-nf-debug-logging': '1' } }).then((res) => {
expect(res.status).to.eq(200)
// expect 'odb' until https://github.com/netlify/pillar-runtime/issues/438 is fixed
expect(res.headers).to.have.property('x-nf-render-mode', 'odb')
// expect 'Bitten' until the above is fixed and we can test for fallback 'Loading...' message
expect(res.body).to.contain('Bitten')
})
Comment on lines +64 to 72
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually not changing any assertions, just update the comment why the assertions will stay as they are. Visiting https://651d4678cf52d90008b40118--netlify-plugin-nextjs-demo.netlify.app/getStaticProps/withRevalidate/withFallback/3/ (older preview) shows the problematic behavior (that fallback that flashes before browser client update) - you can also just check with JS disabled in browser and then you never get to see actual proper content

})
Expand Down Expand Up @@ -115,12 +116,14 @@ describe('Dynamic Routing', () => {
},
)
})
it('renders fallback page via ODB on a non-prerendered dynamic route with revalidate and fallback: true', () => {
it('does not render fallback page via ODB on a non-prerendered dynamic route with revalidate and fallback: true', () => {
// unfortunately there is a problem with `fallback: true` in ODB context - the fallback would be cached indefinitely
// so visits to those pages would always render the fallback first and browser client would later re-render with correct
// content. As this is not ideal the `fallback: true` is treated as `fallback: blocking`.
cy.request({ url: '/getStaticProps/withRevalidate/withFallback/3/', headers: { 'x-nf-debug-logging': '1' } }).then(
(res) => {
expect(res.status).to.eq(200)
expect(res.headers).to.have.property('x-nf-render-mode', 'odb ttl=60')
// expect 'Bitten' until https://github.com/netlify/pillar-runtime/issues/438 is fixed
expect(res.body).to.contain('Bitten')
},
)
Expand Down
82 changes: 42 additions & 40 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions packages/runtime/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"manifest.yml"
],
"dependencies": {
"@netlify/blobs": "^2.2.0",
"@netlify/esbuild": "0.14.39",
"@netlify/functions": "^1.6.0",
"@netlify/ipx": "^1.4.5",
Expand Down
1 change: 1 addition & 0 deletions packages/runtime/src/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ export const HIDDEN_PATHS = destr(process.env.NEXT_KEEP_METADATA_FILES)
'/prerender-manifest.js',
'/required-server-files.json',
'/static-manifest.json',
'/blobs-manifest.json',
]

export const ODB_FUNCTION_PATH = `/.netlify/builders/${ODB_FUNCTION_NAME}`
Expand Down
3 changes: 3 additions & 0 deletions packages/runtime/src/helpers/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ const defaultFailBuild = (message: string, { error }): never => {
export const getNextConfig = async function getNextConfig({
publish,
failBuild = defaultFailBuild,
}: {
publish: string
failBuild?: (message: string, { error }) => never
}): Promise<NextConfig> {
try {
const { config, appDir, ignore }: RequiredServerFiles = await readJSON(join(publish, 'required-server-files.json'))
Expand Down
Loading