Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

node:fs createWriteStream - IO blocks event loop #14542

Open
nesterow opened this issue Oct 14, 2024 · 0 comments
Open

node:fs createWriteStream - IO blocks event loop #14542

nesterow opened this issue Oct 14, 2024 · 0 comments
Labels
bug Something isn't working confirmed bug We can reproduce this issue linux An issue that occurs on Linux macOS An issue that occurs on macOS node:fs

Comments

@nesterow
Copy link

What version of Bun is running?

1.1.30

What platform is your computer?

Darwin 23.6.0 arm64 arm

What steps can reproduce the bug?

When piping to a file (fs.createWriteStream) the IO blocks event loop. For example, the timers become unresponsive.
Following code uses two streams. Readable - generates some chunks, fs WriteStream is supposed to write them to a file.
For example I use a setInterval which prints some stats every second and a setTimeout for some delayed task.

import { Readable } from "node:stream";
import { createWriteStream } from "node:fs";

function* generate(sizeMb = 512) {
  let size = 0;
  while (size <= sizeMb * 1024 ** 2) {
    const data = new Array(64).fill("X").join("");
    size += data.length;
    yield data;
  }
}

// use stream to generate some bytes
const Iterator = generate();
let readBytes = 0;
const reader = new Readable({
  objectMode: false,
  read(size) {
    const data = Iterator.next();
    if (data.done) {
      return this.push(null);
    }
    readBytes += data.value.length;
    this.push(data.value);
  },
});

// write to file
const file = createWriteStream("fake.data.txt");
reader.pipe(file);
reader.once("close", () => {
  clearInterval(interval);
  stats(readBytes);
  console.log("🟩 finished writing file");
});

// print stats
const interval = setInterval(() => stats(readBytes), 1000);

setTimeout(() => {
  console.log("⚠️ Some dealyed task");
});

const stats = (written: number) => {
  console.log(
    "💾 saved ",
    Math.round(written / 1024 / 1024),
    "MB;",
    "📈 RSS:",
    Math.round(process.memoryUsage().rss / 1024 / 1024),
    "MB",
  );
};

console.log("🔁 started writing file");

What is the expected behavior?

Expected the timers to work while performing IO. Deno/Node output:

deno -A write-file.ts
🔁 started writing file
⚠️ Some dealyed task
💾 saved  65 MB; 📈 RSS: 85 MB
💾 saved  134 MB; 📈 RSS: 85 MB
💾 saved  202 MB; 📈 RSS: 85 MB
💾 saved  271 MB; 📈 RSS: 85 MB
💾 saved  338 MB; 📈 RSS: 85 MB
💾 saved  405 MB; 📈 RSS: 85 MB
💾 saved  473 MB; 📈 RSS: 85 MB
💾 saved  512 MB; 📈 RSS: 85 MB
🟩 finished writing file

What do you see instead?

Bun starts the timers after the write stream is closed. Which is practically fs.writeFileSync:

bun write-file.ts
🔁 started writing file
💾 saved  512 MB; 📈 RSS: 83 MB
🟩 finished writing file
⚠️ Some dealyed task

Additional information

No response

@nesterow nesterow added bug Something isn't working needs triage labels Oct 14, 2024
@RiskyMH RiskyMH added macOS An issue that occurs on macOS linux An issue that occurs on Linux confirmed bug We can reproduce this issue labels Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working confirmed bug We can reproduce this issue linux An issue that occurs on Linux macOS An issue that occurs on macOS node:fs
Projects
None yet
Development

No branches or pull requests

3 participants