wip use wrapper for managing process (#8456)

* WIP sync close (shows ref count bug in stream)

* fix closing on PipeWriter and PipeReader

* remove old todos

* join

* Some shell changes

at least it compiles

* fix some compile errors

* fix ref/unref server on windows

* actually use the ref count in this places

* make windows compile again

* more tests passing

* Make shell compile again

* Slowly remove some `@panic("TODO SHELL")`

* Eliminate `@panic("TODO SHELL")` for BufferedWriter

* Holy cleansing of `@panic("TODO SHELL")`

at least it compiles now

* Okay now the shell compiles, but segfaults

* Fix compiler errors

* more stable stream and now Content-Range pass

* make windows compile again

* revert stuff until the fix is actually ready

* revert onDone thing

* Fix buffered writer for shell

* Fix buffered writer + shell/subproc.zig and windows build

* Fix for #8982 got lost in the merge

* Actually buffer subproc output

* Fix some stuff shell

* oops

* fix context deinit

* fix renderMissing

* shell: Fix array buffer

* more stable streams (#9053)

fix stream ref counting

* wip

* Remove `@panic("TODO")` on shell event loop tasks and Redirect  open flags got lost in merge

* Support redirects

* fixes

cc @cirospaciari

* Update ReadableStreamInternals.ts

* Fix spurious error

* Update stream.js

* leak

* Fix UAF

cc @cirospaciari

* Fix memory leaks

* HOLY FUCK big refactor

* misc cleanup

* shell: Fix a bunch of tests

* clean up

* gitignore: fix ending newline

* get windows compiling again

* tidy

* hide linker warn with icu

* closeIfPossible

* Better leak test

* Fix forgetting to decrement reference count

* Update stdio.zig

* Fix shell windows build

* Stupid unreachable

* Woops

* basic echo hi works on windows

* Fix flaky test on Windows

* Fix windows regression in Bun.main (#9156)

* Fix windows regression in Bun.main

* Handle invalid handles

* Fix flaky test

* Better launch config

* Fixup

* Make this test less flaky on Windows

* Fixup

* Cygwin

* Support signal codes in subprocess.kill(), resolve file path

* Treat null as ignore

* Ignore carriage returns

* Fixup

* shell: Fix IOWriter bug

* shell: Use custom `open()`/`openat()`

* windows shell subproc works

* zack commit

* I think I understand WindowsStreamingWriter

* fix thing

* why were we doing this in tests

* shell: Fix rm

* shell: Add rm -rf node_modules/ test

* shell: use `.runAsTest()` in some places to make it easier to determine which test failed

* [autofix.ci] apply automated fixes

* woopsie

* Various changes

* Fix

* shell: abstract output task logic

* shell: mkdir builtin

* fixup

* stuff

* shell: Make writing length of 0 in IOWriter immediately resolve

* shell: Implement `touch`

* shell: basic `cat` working

* Make it compile on windows

* shell: Fix IOReader bug

* [autofix.ci] apply automated fixes

* fix windows kill on subprocess/process

* fix dns tests to match behavior on windows (same as nodejs)

* fix windows ci

* again

* move `close_handle` to flags in `PipeWriter` and fix shell hanging

* Fix `ls` not giving non-zero exit code on error

* Handle edgecase in is_atty

* Fix writer.flush() when there's no data

* Fix some tests

* Disable uv_unref on uv_process_t on Windows, for now.

* fix writer.end

* fix stdout.write

* fix child-process on win32

* Make this test less flaky on Windows

* Add assertion

* Make these the same

* Make it pass on windows

* Don't commit

* Log the test name

* Make this test less flaky on windows

* Make this test less flaky on windows

* Print which test is taking awhile in the runner

* fixups

* Fixups

* Add some assertions

* Bring back test concurrency

* shell: bring back redirect stdin

* make it compile again cc @zackradisic

* initialize env map with capacity

* some fixes

* cleanup

* oops

* fix leak, fix done

* fix unconsumedPromises on events

* always run expect

* Update child_process.test.ts

* fix reading special files

* Fix a test

* Deflake this test

* Make these comparisons easier

* Won't really fix it but slightly cleaner

* Update serve.test.ts

* Make the checks for if the body is already used more resilient

* Move this to the harness

* Make this test not hang in development

* Fix this test

* Make the logs better

* zero init some things

* Make this test better

* Fix readSocket

* Parallelize this test

* Handle EPipe and avoid big data

* This was a mistake

* Fix a bunch of things

* Fix memory leak

* Avoid sigpipe + optimize + delete dead code

* Make this take less time

* Make it bigger

* Remove some redundant code

* Update process.zig

* Merge and hopefully don't breka things along teh way

* Silence build warning

* Uncomment on posix

* Skip test on windows

* windows

* Cleanup test

* Update

* Deflake

* always

* less flaky test

* [autofix.ci] apply automated fixes

* logs

* fix uaf on shell IOReader

* stuff to make it work with mini event loop

* fix 2 double free scenarios, support redirections on windows

* shell: Make `1>&2` and `2>&1` work with libuv

* yoops

* Partial fix

* Partial fix

* fix build

* fix build

* ok

* Make a couple shell tests pass

* More logging

* fix

* fix

* Fix build issue

* more tests pass

* Deflake

* Deflake

* Use Output.panic instead of garbled text

* Formatting

* Introduce `bun.sys.File`, use it for `Output.Source.StreamType`, fix nested Output.scoped() calls, use Win32 `ReadFile` API for reading when it's not a libuv file descriptor.

This lets us avoid the subtle usages of `unreachable` in std.os when writing to stdout/stderr.

Previously, we were initializing the libuv loop immediately at launch due to checking for the existence of a bun build --compile'd executable. When the file descriptor is not from libuv, it's just overhead to use libuv

cc @paperdave, please tell me if Iany of that is incorrect or if you think this is a bad idea.

* Fix closing undefined memory file descriptors in spawn

cc @zackradisic

* pause instead of close

* Fix poorly-written test

* We don't need big numbers for this test

* sad workaround

* fixup

* Clearer error handling for this test

* Fix incorrect test

@Electroid when ReadableStream isn't closed, hanging is the correct behavior when consuming buffered data. We cannot know if the buffered data is finished if the stream never closes.

* Fix build

* Remove known failing on windows

* Deflake

* Mark no longer failing

* show all the failing tests

* Sort the list of tests

* fix argument handling

* dont show "posix_spawn" as an error code on windows

* make bun-upgrade.test.ts pass on windows

* fix bunx and bun create again sorry

* a

* fix invalidexe because we should not be running javascript files as if they were exes

* Concurrency in test runner + better logging

* Revert "fix invalidexe because we should not be running javascript files as if they were exes"

This reverts commit da47cf8.

* WIP: Unix fixes (#9322)

* wip

* [autofix.ci] apply automated fixes

* wip 2

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jarred Sumner <[email protected]>

* Update runner.node.mjs

* Update runner.node.mjs

* Document some environment variables

* shell: Make `Response` work with builtins

* Make it compile

* make pwd test pass

* [autofix.ci] apply automated fixes

* Fix printing garbage for source code previews

* Update javascript.zig

* Fix posix test failures

* Fix signal dispatch

cc @paperdave. Signals can be run from any thread. This causes an assertion failure when the receiving thread happens to not be the main thread. Easiest to reproduce on linux when you spawn 100 short-lived processes at once.

* windows

---------

Co-authored-by: cirospaciari <[email protected]>
Co-authored-by: Zack Radisic <[email protected]>
Co-authored-by: Zack Radisic <[email protected]>
Co-authored-by: Jarred Sumner <[email protected]>
Co-authored-by: Meghan Denny <[email protected]>
Co-authored-by: Zack Radisic <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: dave caruso <[email protected]>
Co-authored-by: Dylan Conway <[email protected]>

.github/workflows/bun-linux-build.yml

@@ -262,9 +262,6 @@ jobs:
           TLS_POSTGRES_DATABASE_URL: ${{ secrets.TLS_POSTGRES_DATABASE_URL }}
         # if: ${{github.event.inputs.use_bun == 'false'}}
         run: |
-          ulimit -c unlimited
-          ulimit -c
-
           node packages/bun-internal-test/src/runner.node.mjs || true
       # - uses: actions/upload-artifact@v4
       #   if: steps.test.outputs.failing_tests != ''

.github/workflows/bun-mac-aarch64.yml

@@ -223,6 +223,7 @@ jobs:
 
           cmake -S $SOURCE_DIR -B $OBJ_DIR \
             -G Ninja \
+            -DUSE_LTO=ON \
             -DCMAKE_BUILD_TYPE=Release \
             -DBUN_CPP_ONLY=1 \
             -DNO_CONFIGURE_DEPENDS=1
@@ -308,6 +309,7 @@ jobs:
           cmake $SRC_DIR \
             -G Ninja \
             -DCMAKE_BUILD_TYPE=Release \
+            -DUSE_LTO=ON \
             -DBUN_LINK_ONLY=1 \
             -DBUN_ZIG_OBJ="${{ runner.temp }}/release/bun-zig.o" \
             -DBUN_CPP_ARCHIVE="${{ runner.temp }}/bun-cpp-obj/bun-cpp-objects.a" \

.github/workflows/bun-mac-x64-baseline.yml

@@ -213,6 +213,7 @@ jobs:
 
           cmake -S $SOURCE_DIR -B $OBJ_DIR \
             -G Ninja \
+            -DUSE_LTO=ON \
             -DCMAKE_BUILD_TYPE=Release \
             -DBUN_CPP_ONLY=1 \
             -DNO_CONFIGURE_DEPENDS=1
@@ -293,6 +294,7 @@ jobs:
           cd ${{runner.temp}}/link-build
           cmake $SRC_DIR \
             -G Ninja \
+            -DUSE_LTO=ON \
             -DCMAKE_BUILD_TYPE=Release \
             -DBUN_LINK_ONLY=1 \
             -DBUN_ZIG_OBJ="${{ runner.temp }}/release/bun-zig.o" \

.github/workflows/bun-mac-x64.yml

@@ -212,6 +212,7 @@ jobs:
           cmake -S $SOURCE_DIR -B $OBJ_DIR \
             -G Ninja \
             -DCMAKE_BUILD_TYPE=Release \
+            -DUSE_LTO=ON \
             -DBUN_CPP_ONLY=1 \
             -DNO_CONFIGURE_DEPENDS=1
 
@@ -292,6 +293,7 @@ jobs:
           cmake $SRC_DIR \
             -G Ninja \
             -DCMAKE_BUILD_TYPE=Release \
+            -DUSE_LTO=ON \
             -DBUN_LINK_ONLY=1 \
             -DBUN_ZIG_OBJ="${{ runner.temp }}/release/bun-zig.o" \
             -DBUN_CPP_ARCHIVE="${{ runner.temp }}/bun-cpp-obj/bun-cpp-objects.a" \

.github/workflows/bun-windows.yml

@@ -417,6 +417,9 @@ jobs:
         uses: actions/setup-node@v4
         with:
           node-version: 20
+      - uses: secondlife/setup-cygwin@v1
+        with:
+          packages: bash
       - name: Install dependencies
         run: |
           # bun install --verbose
@@ -434,12 +437,12 @@ jobs:
           TMPDIR: ${{runner.temp}}
           TLS_MONGODB_DATABASE_URL: ${{ secrets.TLS_MONGODB_DATABASE_URL }}
           TLS_POSTGRES_DATABASE_URL: ${{ secrets.TLS_POSTGRES_DATABASE_URL }}
+          SHELLOPTS: igncr
+          BUN_PATH_BASE: ${{runner.temp}}
+          BUN_PATH: release/${{env.tag}}-${{ matrix.arch == 'x86_64' && 'x64' || 'aarch64' }}${{ matrix.cpu == 'nehalem' && '-baseline' || '' }}-profile/bun.exe
         run: |
-          try {
-            $ErrorActionPreference = "SilentlyContinue"
-            $null = node packages/bun-internal-test/src/runner.node.mjs ${{runner.temp}}/release/${{env.tag}}-${{ matrix.arch == 'x86_64' && 'x64' || 'aarch64' }}${{ matrix.cpu == 'nehalem' && '-baseline' || '' }}-profile/bun.exe || $true
-          } catch {}
-          $ErrorActionPreference = "Stop"
+          node packages/bun-internal-test/src/runner.node.mjs  || true
+        shell: bash
       - uses: sarisia/actions-status-discord@v1
         if: always() && steps.test.outputs.failing_tests != '' && github.event_name == 'pull_request'
         with:

.gitignore

@@ -160,9 +160,10 @@ x64
 /.cache
 /src/deps/libuv
 /build-*/
+/kcov-out
 
 .vs
 
 **/.verdaccio-db.json
 /test-report.md
-/test-report.json
+/test-report.json

CMakeLists.txt

@@ -41,7 +41,11 @@ elseif(CMAKE_BUILD_TYPE STREQUAL "Release")
         # it is enabled for the time being to make sure to catch more bugs in the experimental windows builds
         set(DEFAULT_ZIG_OPTIMIZE "ReleaseSafe")
     else()
-        set(bun "bun-profile")
+        if(ZIG_OPTIMIZE STREQUAL "Debug")
+            set(bun "bun-debug")
+        else()
+            set(bun "bun-profile")
+        endif()
     endif()
 endif()
 
@@ -227,6 +231,13 @@ set(DEFAULT_USE_DEBUG_JSC, OFF)
 
 if(CMAKE_BUILD_TYPE STREQUAL "Debug")
     set(DEFAULT_USE_DEBUG_JSC ON)
+    set(DEFAULT_LTO OFF)
+elseif(CMAKE_BUILD_TYPE STREQUAL "Release")
+    if(CI)
+        set(DEFAULT_LTO ON)
+    else()
+        set(DEFAULT_LTO OFF)
+    endif()
 endif()
 
 if(WIN32)
@@ -263,6 +274,8 @@ option(USE_DEBUG_JSC "Enable assertions and use a debug build of JavaScriptCore"
 option(USE_UNIFIED_SOURCES "Use unified sources to speed up the build" OFF)
 option(USE_STATIC_LIBATOMIC "Statically link libatomic, requires the presence of libatomic.a" ${DEFAULT_USE_STATIC_LIBATOMIC})
 
+option(USE_LTO "Enable Link-Time Optimization" ${DEFAULT_LTO})
+
 if(USE_VALGRIND)
     # Disable SIMD
     set(USE_BASELINE_BUILD ON)
@@ -430,7 +443,13 @@ if(NOT WEBKIT_DIR)
         set(BUN_WEBKIT_PACKAGE_NAME_SUFFIX "-debug")
         set(ASSERT_ENABLED "1")
     elseif(NOT DEBUG AND NOT WIN32)
-        set(BUN_WEBKIT_PACKAGE_NAME_SUFFIX "-lto")
+        # Avoid waiting for LTO in local release builds outside of CI
+        if(USE_LTO)
+            set(BUN_WEBKIT_PACKAGE_NAME_SUFFIX "-lto")
+        else()
+            set(BUN_WEBKIT_PACKAGE_NAME_SUFFIX "")
+        endif()
+
         set(ASSERT_ENABLED "0")
     endif()
 
@@ -958,15 +977,28 @@ if(CMAKE_BUILD_TYPE STREQUAL "Debug")
 
     add_compile_definitions("BUN_DEBUG=1")
 elseif(CMAKE_BUILD_TYPE STREQUAL "Release")
+    set(LTO_FLAG "")
+
     if(NOT WIN32)
-        target_compile_options(${bun} PUBLIC -O3 -flto=full -emit-llvm -g1
+        if(USE_LTO)
+            list(APPEND LTO_FLAG "-flto=full" "-emit-llvm")
+        endif()
+
+        target_compile_options(${bun} PUBLIC -O3 ${LTO_FLAG} -g1
             -Werror=return-type
             -Werror=return-stack-address
             -Werror=implicit-function-declaration
         )
     else()
-        target_compile_options(${bun} PUBLIC /O2 -flto=full /DEBUG /Z7)
-        target_link_options(${bun} PUBLIC /LTCG /DEBUG)
+        set(LTO_LINK_FLAG "")
+
+        if(USE_LTO)
+            list(APPEND LTO_FLAG "-flto=full" "-emit-llvm")
+            list(APPEND LTO_LINK_FLAG "/LTCG")
+        endif()
+
+        target_compile_options(${bun} PUBLIC /O2 ${LTO_FLAG} /DEBUG /Z7)
+        target_link_options(${bun} PUBLIC ${LTO_LINK_FLAG} /DEBUG)
     endif()
 endif()
 
@@ -1018,11 +1050,15 @@ else()
 endif()
 
 if(APPLE)
+    # this is gated to avoid the following warning when developing on modern versions of macOS.
+    # ld: warning: object file (/opt/homebrew/opt/icu4c/lib/libicudata.a[2](icudt73l_dat.o)) was built for newer 'macOS' version (14.0) than being linked (11.0)
+    if(DEFINED ENV{CI})
     if(ARCH STREQUAL "x86_64")
         set(CMAKE_OSX_DEPLOYMENT_TARGET "10.14")
     else()
         set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0")
     endif()
+    endif()
 
     target_link_options(${bun} PUBLIC "-dead_strip")
     target_link_options(${bun} PUBLIC "-dead_strip_dylibs")

Dockerfile

@@ -372,7 +372,7 @@ ENV CCACHE_DIR=/ccache
 RUN --mount=type=cache,target=/ccache  mkdir ${BUN_DIR}/build \
   && cd ${BUN_DIR}/build \
   && mkdir -p tmp_modules tmp_functions js codegen \
-  && cmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DUSE_DEBUG_JSC=${ASSERTIONS} -DBUN_CPP_ONLY=1 -DWEBKIT_DIR=/build/bun/bun-webkit -DCANARY=${CANARY} -DZIG_COMPILER=system \
+  && cmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DUSE_LTO=ON -DUSE_DEBUG_JSC=${ASSERTIONS} -DBUN_CPP_ONLY=1 -DWEBKIT_DIR=/build/bun/bun-webkit -DCANARY=${CANARY} -DZIG_COMPILER=system \
   && bash compile-cpp-only.sh -v
 
 FROM bun-base-with-zig as bun-codegen-for-zig
@@ -419,6 +419,7 @@ RUN mkdir -p build \
   && cmake .. \
   -G Ninja \
   -DCMAKE_BUILD_TYPE=Release \
+  -DUSE_LTO=ON \
   -DZIG_OPTIMIZE="${ZIG_OPTIMIZE}" \
   -DCPU_TARGET="${CPU_TARGET}" \
   -DZIG_TARGET="${TRIPLET}" \
@@ -476,6 +477,7 @@ RUN cmake .. \
   -DCMAKE_BUILD_TYPE=Release \
   -DBUN_LINK_ONLY=1 \
   -DBUN_ZIG_OBJ="${BUN_DIR}/build/bun-zig.o" \
+  -DUSE_LTO=ON \
   -DUSE_DEBUG_JSC=${ASSERTIONS} \
   -DBUN_CPP_ARCHIVE="${BUN_DIR}/build/bun-cpp-objects.a" \
   -DWEBKIT_DIR="${BUN_DIR}/bun-webkit" \
@@ -540,6 +542,7 @@ RUN cmake .. \
   -DNO_CONFIGURE_DEPENDS=1 \
   -DCANARY="${CANARY}" \
   -DZIG_COMPILER=system \
+  -DUSE_LTO=ON \
   && ninja -v \
   && ./bun --revision \
   && mkdir -p /build/out \

cp.js

@@ -0,0 +1,16 @@
+const { spawn } = require("child_process");
+console.clear();
+console.log("--start--");
+const proc = spawn("sleep", ["0.5"], { stdio: ["ignore", "ignore", "ignore"] });
+
+console.time("Elapsed");
+process.on("exit", () => {
+  console.timeEnd("Elapsed");
+});
+proc.on("exit", (code, signal) => {
+  console.log(`child process terminated with code ${code} and signal ${signal}`);
+  timer.unref();
+});
+proc.unref();
+
+var timer = setTimeout(() => {}, 1000);

docs/runtime/env.md

@@ -163,6 +163,16 @@ These environment variables are read by Bun and configure aspects of its behavio
 
 ---
 
+- `BUN_CONFIG_MAX_HTTP_REQUESTS`
+- Control the maximum number of concurrent HTTP requests sent by fetch and `bun install`. Defaults to `256`. If you are running into rate limits or connection issues, you can reduce this number.
+
+---
+
+- `BUN_CONFIG_NO_CLEAR_TERMINAL_ON_RELOAD`
+- If `BUN_CONFIG_NO_CLEAR_TERMINAL_ON_RELOAD=1`, then `bun --watch` will not clear the console on reload
+
+---
+
 - `DO_NOT_TRACK`
 - Telemetry is not sent yet as of November 28th, 2023, but we are planning to add telemetry in the coming months. If `DO_NOT_TRACK=1`, then analytics are [disabled](https://do-not-track.dev/). Bun records bundle timings (so we can answer with data, "is Bun getting faster?") and feature usage (e.g., "are people actually using macros?"). The request body size is about 60 bytes, so it's not a lot of data. Equivalent of `telemetry=false` in bunfig.
 

docs/runtime/shell.md

@@ -400,25 +400,26 @@ await $`echo ${{ raw: '$(foo) `bar` "baz"' }}`
 // => baz
 ```
 
-## .bun.sh file loader
+## .sh file loader
 
-For simple shell scripts, instead of `sh`, you can use Bun Shell to run shell scripts.
+For simple shell scripts, instead of `/bin/sh`, you can use Bun Shell to run shell scripts.
 
-To do that, run any file with bun that ends with `.bun.sh`:
+To do so, just run the script with `bun` on a file with the `.sh` extension.
 
-```sh
-$ echo "echo Hello World!" > script.bun.sh
-$ bun ./script.bun.sh
-> Hello World!
+```sh#script.sh
+echo "Hello World! pwd=$(pwd)"
 ```
 
-On Windows, Bun Shell is used automatically to run `.sh` files when using Bun:
-
 ```sh
-$ echo "echo Hello World!" > script.sh
-# On windows, .bun.sh is not needed, just .sh
 $ bun ./script.sh
-> Hello World!
+Hello World! pwd=/home/demo
+```
+
+Scripts with Bun Shell are cross platform, which means they work on Windows:
+
+```
+PS C:\Users\Demo> bun .\script.sh
+Hello World! pwd=C:\Users\Demo
 ```
 
 ## Credits

package.json

@@ -24,6 +24,7 @@
     "build": "if [ ! -e build ]; then bun setup; fi && ninja -C build",
     "build:valgrind": "cmake . -DZIG_OPTIMIZE=Debug -DUSE_DEBUG_JSC=ON -DCMAKE_BUILD_TYPE=Debug -GNinja -Bbuild-valgrind && ninja -Cbuild-valgrind",
     "build:release": "cmake . -DCMAKE_BUILD_TYPE=Release -GNinja -Bbuild-release && ninja -Cbuild-release",
+    "build:debug-zig-release": "cmake . -DCMAKE_BUILD_TYPE=Release -DZIG_OPTIMIZE=Debug -GNinja -Bbuild-debug-zig-release && ninja -Cbuild-debug-zig-release",
     "build:safe": "cmake . -DZIG_OPTIMIZE=ReleaseSafe -DUSE_DEBUG_JSC=ON -DCMAKE_BUILD_TYPE=Release -GNinja -Bbuild-safe && ninja -Cbuild-safe",
     "typecheck": "tsc --noEmit && cd test && bun run typecheck",
     "fmt": "prettier --write --cache './{.vscode,src,test,bench,packages/{bun-types,bun-inspector-*,bun-vscode,bun-debug-adapter-protocol}}/**/*.{mjs,ts,tsx,js,jsx}'",

packages/bun-internal-test/package.json

@@ -11,6 +11,7 @@
     "p-queue": "^8.0.1"
   },
   "devDependencies": {
+    "@types/p-queue": "^3.2.1",
     "bun-types": "canary",
     "prettier": "^2.8.2"
   },