Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: new way of creating temporary files backed up by fs #137

Merged
merged 2 commits into from
Jul 6, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,34 @@ console.log(blob.size) // ~4 GiB

`blobFrom|blobFromSync|fileFrom|fileFromSync(path, [mimetype])`

### Creating a temporary file on the disk
(requires [FinalizationRegistry] - node v14.6)

When using both `createTemporaryBlob` and `createTemporaryFile`
then you will write data to the temporary folder in their respective OS.
The arguments can be anything that [fsPromises.writeFile] supports. NodeJS
v14.17.0+ also supports writing (async)Iterable streams and passing in a
AbortSignal, so both NodeJS stream and whatwg streams are supported. When the
file have been written it will return a Blob/File handle with a references to
this temporary location on the disk. When you no longer have a references to
this Blob/File anymore and it have been GC then it will automatically be deleted.

This files are also unlinked upon exiting the process.
```js
import { createTemporaryBlob, createTemporaryFile } from 'fetch-blob/from.js'

const req = new Request('https://httpbin.org/image/png')
const res = await fetch(req)
const type = res.headers.get('content-type')
const signal = req.signal
let blob = await createTemporaryBlob(res.body, { type, signal })
// const file = createTemporaryBlob(res.body, 'img.png', { type, signal })
blob = undefined // loosing references will delete the file from disk
```

`createTemporaryBlob(data, { type, signal })`
`createTemporaryFile(data, FileName, { type, signal, lastModified })`

### Creating Blobs backed up by other async sources
Our Blob & File class are more generic then any other polyfills in the way that it can accept any blob look-a-like item
An example of this is that our blob implementation can be constructed with parts coming from [BlobDataItem](https://github.com/node-fetch/fetch-blob/blob/8ef89adad40d255a3bbd55cf38b88597c1cd5480/from.js#L32) (aka a filepath) or from [buffer.Blob](https://nodejs.org/api/buffer.html#buffer_new_buffer_blob_sources_options), It dose not have to implement all the methods - just enough that it can be read/understood by our Blob implementation. The minium requirements is that it has `Symbol.toStringTag`, `size`, `slice()` and either a `stream()` or a `arrayBuffer()` method. If you then wrap it in our Blob or File `new Blob([blobDataItem])` then you get all of the other methods that should be implemented in a blob or file
Expand All @@ -104,3 +132,5 @@ See the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/Blo
[install-size-image]: https://flat.badgen.net/packagephobia/install/fetch-blob
[install-size-url]: https://packagephobia.now.sh/result?p=fetch-blob
[fs-blobs]: https://github.com/nodejs/node/issues/37340
[fsPromises.writeFile]: https://nodejs.org/dist/latest-v18.x/docs/api/fs.html#fspromiseswritefilefile-data-options
[FinalizationRegistry]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry
66 changes: 62 additions & 4 deletions from.js
Original file line number Diff line number Diff line change
@@ -1,11 +1,20 @@
import { statSync, createReadStream, promises as fs } from 'node:fs'
import { basename } from 'node:path'
import {
realpathSync,
statSync,
rmdirSync,
createReadStream,
promises as fs
} from 'node:fs'
import { basename, sep, join } from 'node:path'
import { tmpdir } from 'node:os'
import process from 'node:process'
import DOMException from 'node-domexception'

import File from './file.js'
import Blob from './index.js'

const { stat } = fs
const { stat, mkdtemp } = fs
let i = 0, tempDir, registry

/**
* @param {string} path filepath on the disk
Expand Down Expand Up @@ -49,6 +58,42 @@ const fromFile = (stat, path, type = '') => new File([new BlobDataItem({
start: 0
})], basename(path), { type, lastModified: stat.mtimeMs })

/**
* Creates a temporary blob backed by the filesystem.
* NOTE: requires node.js v14 or higher to use FinalizationRegistry
*
* @param {*} data Same as fs.writeFile data
* @param {BlobPropertyBag & {signal?: AbortSignal}} options
* @param {AbortSignal} [signal] in case you wish to cancel the write operation
* @returns {Promise<Blob>}
*/
const createTemporaryBlob = async (data, {signal, type} = {}) => {
registry = registry || new FinalizationRegistry(fs.unlink)
tempDir = tempDir || await mkdtemp(realpathSync(tmpdir()) + sep)
const id = `${i++}`
const destination = join(tempDir, id)
if (data instanceof ArrayBuffer) data = new Uint8Array(data)
await fs.writeFile(destination, data, { signal })
const blob = await blobFrom(destination, type)
registry.register(blob, destination)
return blob
}

/**
* Creates a temporary File backed by the filesystem.
* Pretty much the same as constructing a new File(data, name, options)
*
* NOTE: requires node.js v14 or higher to use FinalizationRegistry
* @param {*} data
* @param {string} name
* @param {FilePropertyBag & {signal?: AbortSignal}} opts
* @returns {Promise<File>}
*/
const createTemporaryFile = async (data, name, opts) => {
const blob = await createTemporaryBlob(data)
return new File([blob], name, opts)
jimmywarting marked this conversation as resolved.
Show resolved Hide resolved
}

/**
* This is a blob backed up by a file on the disk
* with minium requirement. Its wrapped around a Blob as a blobPart
Expand Down Expand Up @@ -102,5 +147,18 @@ class BlobDataItem {
}
}

process.once('exit', () => {
tempDir && rmdirSync(tempDir, { recursive: true })
})

export default blobFromSync
export { File, Blob, blobFrom, blobFromSync, fileFrom, fileFromSync }
export {
Blob,
blobFrom,
blobFromSync,
createTemporaryBlob,
File,
fileFrom,
fileFromSync,
createTemporaryFile
}
41 changes: 40 additions & 1 deletion test/own-misc-test.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,14 @@

import fs from 'node:fs'
import buffer from 'node:buffer'
import syncBlob, { blobFromSync, blobFrom, fileFromSync, fileFrom } from '../from.js'
import syncBlob, {
blobFromSync,
blobFrom,
fileFromSync,
fileFrom,
createTemporaryBlob,
createTemporaryFile
} from '../from.js'

const license = fs.readFileSync('./LICENSE')

Expand Down Expand Up @@ -189,6 +196,38 @@ promise_test(async () => {
assert_equals(await (await fileFrom('./LICENSE')).text(), license.toString())
}, 'blob part backed up by filesystem slice correctly')

promise_test(async () => {
let blob
// Can construct a temporary blob from a string
blob = await createTemporaryBlob(license.toString())
assert_equals(await blob.text(), license.toString())

// Can construct a temporary blob from a async iterator
blob = await createTemporaryBlob(blob.stream())
assert_equals(await blob.text(), license.toString())

// Can construct a temporary file from a arrayBuffer
blob = await createTemporaryBlob(await blob.arrayBuffer())
assert_equals(await blob.text(), license.toString())

// Can construct a temporary file from a arrayBufferView
blob = await createTemporaryBlob(await blob.arrayBuffer().then(ab => new Uint8Array(ab)))
assert_equals(await blob.text(), license.toString())

// Can specify a mime type
blob = await createTemporaryBlob('abc', { type: 'text/plain' })
assert_equals(blob.type, 'text/plain')

// Can create files too
let file = await createTemporaryFile('abc', 'abc.txt', {
type: 'text/plain',
lastModified: 123
})
assert_equals(file.name, 'abc.txt')
assert_equals(file.size, 3)
assert_equals(file.lastModified, 123)
}, 'creating temporary blob/file backed up by filesystem')

promise_test(async () => {
fs.writeFileSync('temp', '')
await blobFromSync('./temp').text()
Expand Down