Worker (#133)

* App storage move (#111)

* app storage move

* pear shift

* make fs ops async

* gc

* gc folder may not be there, but it does not matter, ignore the error

* fix

* Trust dialog poc (#110)

* Implement title bar buttons for generic plaforms like linux

* introduced trust dialog

* add platform trust report

* renamed report to permission required

* linux frame tweaks

* resize window to fit trust dialog

* pear-rpc

* inf wakeup loop fix

* sidecar starting

* wakeup align

* run align

* refs no longer needed, dep fix, arg fix

* fixes

* refactor gui layer to its own folder, pear-rpc now also between renderer and main

* fix

* desktop rpc almost there

* gui dual export (renderer and main), desktop rpc proven

* sucessfully opens on desktop, remaining sidecar ipc statebug

* pear-rpc -> pear-ipc

* unloading on ipc instance in electron-main fix

* explicit forwarding

* desktop working

* stage fix, naming, tidy-up

* graceful-goodbye needed for bootstrap script

* bump pear-ipc

* add verbose (#96)

* uncomment ipc call

* readded used imports

* port in 5d5083f (app upgrades fix)

* conflict fix

* Adapt tests to pear-ipc (#97)

* adapt helper

* adapt helper

* adapting smoke test

* add shutdown test

* cleanup

* client freelist lives in pear-ipc

* fix

* Adapt tests to pear-ipc (#100)

* adapt helper

* add shutdown test

* cleanup

* add tests

* desktop wakeup fix, desktop close fix

* ipc reports stream

* fix ipc close bug

* close client on session teardown

* test fixes, messages fix

* adapt to pear-rpc

* exposed trust method

* ipc close namespace clash fix

* added id in ipc.restart call from trust-dialog

* update test improve, helper improve, RPCStream not ending case in terminal example

* improve smoke eval code

* preferences

* use hypercore-id-encoding from version

* updates test working

* lint

* yeild null to close streams

* fixed decal.html for pear-rpc (#104)

Co-authored-by: rafapaezbas <[email protected]>

* use-after-close destroy fix

* client restart fix

* --detached fix

* platform restart fix

* devtools opening fix

* client close when not tearing down

* improve client close

* fix cancel and confirm buttons

* fix mac os traffic lights

* restore ERR_PERMISSION_REQUIRED throw

* title bar dragable

* move permission required report to ctx sidecar

* replaced trusted set with array

* revert set trust set change and fixed key check

* *

* added spawned flag

* dont bundle app if key is not trusted

* replace throw with return bail

* remove sidecar untrusted else

* mac titlebar drag fix

* format fix

* reverted uneeded changes

* Revert "mac titlebar drag fix"

This reverts commit f036b99.

* fix for mac titlebar drag

* fix mac bar

* mac traffic lights / drag bar fix

---------

Co-authored-by: jb <[email protected]>
Co-authored-by: rafapaezbas <[email protected]>
Co-authored-by: dmc <[email protected]>
Co-authored-by: Dan <[email protected]>

* copy/paste text only using right-click (#123)

* use right-click to copy/paste text only

* fix Inspect Element menu

* Pear info <channelName> (#108)

* pear info channel

* remove display nested props

* bring json to end of usage info

* use parse defaults

* handle in ipc info handler

* keys as metadata

* parsed as ternary

* join bundle, close bundle

* update usage, newline in full changelog

* ignore stdio for --detached run (#129)

* ignore stdio for --detached run

* Revert "ignore stdio for --detached run"

This reverts commit e52d862.

* ignore stdio for --detached run

---------

Co-authored-by: rafapaezbas <[email protected]>

* Clean up errors (#125)

* added errors class and replaced gui errors

* clean up errors in preload, parse and run

* refactored sidecar errors

* replaced iface InputError

* reverted errors clean up in cmd and prefixed with ERR

* removed static functions from PearError

* added ERR_ASSERTION

* added invalid storage error

* added ERR_INSTANCE_CLOSED

* reverted changes in cmd/seed

* renamed ERR_SHIFT_STORAGE to ERR_SHIFT_STORAGE_ERROR

* reverted error cleanup in gui/* files

---------

Co-authored-by: rafapaezbas <[email protected]>

* filter Linux Electron error logs (#134)

Co-authored-by: rafapaezbas <[email protected]>

* worker

* config.links & trust links when key is trusted

* retrust (cheap) to get new links on start & Pear.config.links

* should never be a json parse error by this point, but just in case there is avoid throwing a SyntaxError (+ other error case output refinement)

* use bundle.drive

* lint 😒

* trust fix + win fix for openSync(3) when parent)

* cannot spawn attached from renderer on windows, spawning from electron-main solves

* working on Windows

* use isBare instead of global.Bare

* move noop

* syntax tidy

* free pipe on close

* spacing

* ERR_INVALID_PACKAGE_JSON

* propagate close/destroy

* post conflict res syntax fixes

---------

Co-authored-by: rafapaezbas <[email protected]>
Co-authored-by: jb <[email protected]>
Co-authored-by: rafapaezbas <[email protected]>
Co-authored-by: Dan <[email protected]>
Co-authored-by: yasser <[email protected]>

ctx/shared.js

@@ -31,6 +31,7 @@ module.exports = class Context {
     ctx.options = pkg?.pear || pkg?.holepunch || {}
     ctx.name = pkg?.pear?.name || pkg?.holepunch?.name || pkg?.name || null
     ctx.type = pkg?.pear?.type || (/\.(c|m)?js$/.test(ctx.main) ? 'terminal' : 'desktop')
+    ctx.links = pkg?.pear?.links || null
     ctx.dependencies = [
       ...(pkg?.dependencies ? Object.keys(pkg.dependencies) : []),
       ...(pkg?.devDependencies ? Object.keys(pkg.devDependencies) : []),
@@ -56,9 +57,9 @@ module.exports = class Context {
   }
 
   static configFrom (ctx) {
-    const { id, key, alias, env, cwd, options, checkpoint, flags, dev, tier, stage, storage, trace, name, main, dependencies, args, channel, release, link, linkData, dir } = ctx
+    const { id, key, links, alias, env, cwd, options, checkpoint, flags, dev, tier, stage, storage, trace, name, main, dependencies, args, channel, release, link, linkData, dir } = ctx
     const pearDir = PLATFORM_DIR
-    return { id, key, alias, env, cwd, options, checkpoint, flags, dev, tier, stage, storage, trace, name, main, dependencies, args, channel, release, link, linkData, dir, pearDir }
+    return { id, key, links, alias, env, cwd, options, checkpoint, flags, dev, tier, stage, storage, trace, name, main, dependencies, args, channel, release, link, linkData, dir, pearDir }
   }
 
   update (state) {

examples/desktop/app.js

@@ -23,3 +23,8 @@ document.getElementById('platformLength').innerText = platform.length
 document.getElementById('appKey').innerText = app.key
 document.getElementById('appFork').innerText = app.fork
 document.getElementById('appLength').innerText = app.length
+
+if (config.args.includes('--worker-demo')) {
+  global.pipe = Pear.worker.run(config.links.worker)
+  console.info('Pipe Duplex Stream available as `pipe`')
+}

examples/desktop/package.json

@@ -9,6 +9,9 @@
   "type": "module",
   "pear": {
     "name": "deskex",
+    "links": {
+      "worker": "pear://456cup6hg5pwpekmygzhbxyxgtpknk66mbccohcuo795dj6biueo"
+    },
     "gui": {
       "width": 900,
       "height": 500,

examples/terminal/index.js

@@ -17,3 +17,17 @@ const out = `${grn}           ▅
        ▄▄▄▄▆▆▆▆
 `
 console.log('\n\x1b[s\x1b[J' + out)
+
+const pipe = Pear.worker.pipe()
+const isWorker = pipe !== null
+if (isWorker) {
+  pipe.on('data', (data) => {
+    const str = data.toString()
+    console.log('parent:', str)
+    if (str === 'hello') console.log('world')
+    if (str === 'exit') {
+      console.log('exiting')
+      Pear.exit()
+    }
+  })
+}

gui/gui.js

@@ -7,7 +7,9 @@ const path = require('path')
 const { isMac, isLinux } = require('which-runtime')
 const IPC = require('pear-ipc')
 const ReadyResource = require('ready-resource')
+const Worker = require('../lib/worker')
 const constants = require('../lib/constants')
+
 const kMap = Symbol('pear.gui.map')
 const kCtrl = Symbol('pear.gui.ctrl')
 
@@ -1352,6 +1354,8 @@ class PearGUI extends ReadyResource {
       },
       connect: tryboot
     })
+    this.worker = new Worker()
+    this.pipes = new Freelist()
     this.ipc.once('close', () => this.close())
 
     electron.ipcMain.on('id', async (event) => {
@@ -1418,18 +1422,50 @@ class PearGUI extends ReadyResource {
     electron.ipcMain.handle('versions', (evt, ...args) => this.versions(...args))
     electron.ipcMain.handle('restart', (evt, ...args) => this.restart(...args))
 
+    electron.ipcMain.on('workerRun', (evt, link) => {
+      const pipe = this.worker.run(link)
+      const id = this.pipes.alloc(pipe)
+      pipe.on('close', () => {
+        this.pipes.free(id)
+        evt.reply('workerPipeClose')
+      })
+      pipe.on('data', (data) => { evt.reply('workerPipeData', data) })
+      pipe.on('end', () => { evt.reply('workerPipeData', null) })
+      pipe.on('error', (err) => { evt.reply('pipeError', err.stack) })
+    })
+
+    electron.ipcMain.on('workerPipeId', (evt) => {
+      evt.returnValue = this.pipes.nextId()
+      return evt.returnValue
+    })
+
+    electron.ipcMain.on('workerPipeClose', (evt, id) => {
+      const pipe = this.pipes.from(id)
+      if (!pipe) return
+      pipe.destroy()
+    })
+
+    electron.ipcMain.on('workerPipeWrite', (evt, id, data) => {
+      const pipe = this.pipes.from(id)
+      if (!pipe) {
+        console.error('Unexpected workerPipe error (unknown id)')
+        return
+      }
+      pipe.write(data)
+    })
+
     // DEPRECATED - assess to remove from Sep 2024
-    electron.ipcMain.on('preferences', (event) => {
+    electron.ipcMain.on('preferences', (evt) => {
       const preferences = this.preferences()
-      preferences.on('data', (data) => event.reply('preferences', data))
-      preferences.on('end', () => event.reply('preferences', null))
+      preferences.on('data', (data) => evt.reply('preferences', data))
+      preferences.on('end', () => evt.reply('preferences', null))
     })
     electron.ipcMain.handle('setPreference', (evt, ...args) => this.setPreference(...args))
     electron.ipcMain.handle('getPreference', (evt, ...args) => this.getPreference(...args))
-    electron.ipcMain.on('iteratePreferences', (event) => {
+    electron.ipcMain.on('iteratePreferences', (evt) => {
       const iteratePreferences = this.iteratePreferences()
-      iteratePreferences.on('data', (data) => event.reply('iteratePreferences', data))
-      iteratePreferences.on('end', () => event.reply('iteratePreferences', null))
+      iteratePreferences.on('data', (data) => evt.reply('iteratePreferences', data))
+      iteratePreferences.on('end', () => evt.reply('iteratePreferences', null))
     })
   }
 
@@ -1643,4 +1679,39 @@ class PearGUI extends ReadyResource {
   iteratePreference () { return this.ipc.iteratePreference() }
 }
 
+class Freelist {
+  alloced = []
+  freed = []
+
+  nextId () {
+    return this.freed.length === 0 ? this.alloced.length : this.freed[this.freed.length - 1]
+  }
+
+  alloc (item) {
+    const id = this.freed.length === 0 ? this.alloced.push(null) - 1 : this.freed.pop()
+    this.alloced[id] = item
+    return id
+  }
+
+  free (id) {
+    this.freed.push(id)
+    this.alloced[id] = null
+  }
+
+  from (id) {
+    return id < this.alloced.length ? this.alloced[id] : null
+  }
+
+  emptied () {
+    return this.freed.length === this.alloced.length
+  }
+
+  * [Symbol.iterator] () {
+    for (const item of this.alloced) {
+      if (item === null) continue
+      yield item
+    }
+  }
+}
+
 module.exports = PearGUI

gui/preload.js

@@ -5,6 +5,7 @@ const { EventEmitter } = require('events')
 const Iambus = require('iambus')
 const ReadyResource = require('ready-resource')
 const electron = require('electron')
+const Worker = require('../lib/worker')
 
 module.exports = class PearGUI extends ReadyResource {
   constructor ({ API, ctx }) {
@@ -31,6 +32,7 @@ module.exports = class PearGUI extends ReadyResource {
       constructor (ipc, ctx, onteardown) {
         super(ipc, ctx, onteardown)
         this[Symbol.for('pear.ipc')] = ipc
+        this.worker = new Worker({ ipc })
         this.media = {
           status: {
             microphone: () => ipc.getMediaAccessStatus({ id, media: 'microphone' }),
@@ -285,6 +287,27 @@ class IPC {
     return stream
   }
 
+  workerRun (link) {
+    const id = electron.ipcRenderer.sendSync('workerPipeId')
+    electron.ipcRenderer.send('workerRun', link)
+    const stream = new streamx.Duplex({
+      write (data, cb) {
+        electron.ipcRenderer.send('workerPipeWrite', id, data)
+        cb()
+      }
+    })
+    electron.ipcRenderer.on('workerPipeError', (e, stack) => {
+      stream.emit('error', new Error('Worker PipeError (from electron-main): ' + stack))
+    })
+    electron.ipcRenderer.on('workerClose', () => { stream.destroy() })
+    stream.once('close', () => {
+      electron.ipcRenderer.send('workerPipeClose', id)
+    })
+
+    electron.ipcRenderer.on('workerPipeData', (e, data) => { stream.push(data) })
+    return stream
+  }
+
   ref () {}
   unref () {}
 

lib/api.js

@@ -1,5 +1,7 @@
 'use strict'
+const Worker = require('./worker')
 const teardown = global.Bare ? require('./teardown') : Function.prototype
+const program = global.Bare || global.process
 
 class API {
   #ipc = null
@@ -8,14 +10,15 @@ class API {
   #teardowns = null
   #refs = 0
   config = null
-  argv = global.Bare ? global.Bare.argv : process.argv
+  argv = program.argv
 
   constructor (ipc, ctx, onteardown = teardown) {
     this.#ipc = ipc
     this.#ctx = ctx
     this.#refs = 0
     this.key = this.#ctx.key?.z32 || 'dev'
     this.config = ctx.config
+    this.worker = new Worker({ ref: () => this.#ref(), unref: () => this.#unref() })
     this.#teardowns = new Promise((resolve) => { this.#unloading = resolve })
     onteardown(() => this.#unload())
   }
@@ -87,7 +90,7 @@ class API {
     return this.#teardowns
   }
 
-  exit = (code) => Bare.exit(code)
+  exit = (code) => program.exit(code)
 
   // DEPRECATED - assess to remove from Sep 2024
   #preferences = null

lib/constants.js

@@ -4,14 +4,14 @@ const { platform, arch, isWindows, isLinux } = require('which-runtime')
 const { pathToFileURL, fileURLToPath } = require('url-file-url')
 const sodium = require('sodium-native')
 const b4a = require('b4a')
+const CHECKOUT = require('../checkout')
+const { ERR_COULD_NOT_INFER_MODULE_PATH } = require('./errors')
 
 const BIN = 'by-arch/' + platform + '-' + arch + '/bin/'
 
 const url = module.url || electronModuleURL()
 const mount = new URL('..', url)
 
-const CHECKOUT = require('../checkout')
-const { ERR_COULD_NOT_INFER_MODULE_PATH } = require('./errors')
 const LOCALDEV = CHECKOUT.length === null
 
 const swapURL = mount.pathname.endsWith('.bundle/') ? new URL('..', mount) : mount

lib/errors.js

@@ -79,6 +79,10 @@ function ERR_UNABLE_TO_FETCH_MANIFEST (msg) {
   return new PearError(msg, 'ERR_CONNECTION', ERR_UNABLE_TO_FETCH_MANIFEST)
 }
 
+function ERR_INVALID_PACKAGE_JSON (msg) {
+  return new PearError(msg, 'ERR_INVALID_PACKAGE_JSON', ERR_INVALID_PACKAGE_JSON)
+}
+
 function ERR_UNKNOWN_GC_RESOURCE (msg) {
   return new PearError(msg, 'ERR_UNKNOWN_GC_RESOURCE', ERR_UNKNOWN_GC_RESOURCE)
 }
@@ -103,6 +107,7 @@ module.exports = {
   ERR_INVALID_APPLICATION_STORAGE,
   ERR_PACKAGE_JSON_NOT_FOUND,
   ERR_UNABLE_TO_FETCH_MANIFEST,
+  ERR_INVALID_PACKAGE_JSON,
   ERR_UNKNOWN_GC_RESOURCE,
   ERR_ASSERTION
 }

lib/sidecar.js

@@ -45,6 +45,7 @@ const {
   ERR_PLATFORM_ERROR,
   ERR_TRACER_FAILED,
   ERR_SHIFT_STORAGE_ERROR,
+  ERR_INVALID_PACKAGE_JSON,
   ERR_PERMISSION_REQUIRED,
   ERR_UNKNOWN_GC_RESOURCE
 } = require('./errors')
@@ -844,7 +845,7 @@ class Engine extends ReadyResource {
       const currentVersion = bundle.version
       await ctx.initialize({ bundle, dryRun })
       const z32 = hypercoreid.encode(bundle.drive.key)
-      await this.trust({ z32 })
+      await this.trust({ z32 }, client)
       const type = ctx.manifest.pear?.type || 'desktop'
       const terminalBare = type === 'terminal'
       if (terminalBare) bare = true
@@ -1061,7 +1062,6 @@ class Engine extends ReadyResource {
 
       const type = full ? 'full' : 'latest'
       const showChangelog = display(changelog) || full ? type : false
-
       const blank = '[ No Changelog ]'
       const parsed = showChangelog === 'latest'
         ? (await clog.parse(contents).at(0)?.[1]) || blank
@@ -1127,9 +1127,26 @@ class Engine extends ReadyResource {
     yield * client.userData.messages(pattern)
   }
 
-  async trust ({ z32 } = {}) {
+  async trust ({ z32 } = {}, client) {
     const trusted = new Set((await preferences.get('trusted')) || [])
     trusted.add(z32)
+    let pkg = null
+    try {
+      pkg = JSON.parse(await client.userData.bundle.drive.get('/package.json'))
+    } catch (err) {
+      if (err instanceof SyntaxError) throw ERR_INVALID_PACKAGE_JSON('Package.json parsing error, invalid JSON')
+      console.error('Unexpected error while attempting trust', err)
+      return await preferences.set('trusted', Array.from(trusted))
+    }
+    if (typeof pkg?.pear?.links === 'object' && pkg.pear.links !== null) {
+      for (const link of Object.values(pkg.pear.links)) {
+        try {
+          trusted.add(hypercoreid.encode(hypercoreid.decode(link)))
+        } catch {
+          console.error('Invalid link encountered when attempting trust', { link })
+        }
+      }
+    }
     return await preferences.set('trusted', Array.from(trusted))
   }
 
@@ -1320,6 +1337,9 @@ class Engine extends ReadyResource {
     app.linker = linker
     app.bundle = appBundle
 
+    // app is trusted, refresh trust for any updated configured link keys:
+    await this.trust({ z32: ctx.key.z32 }, client)
+
     if (this.swarm) appBundle.join(this.swarm)
 
     try {