Streaming data in Oak

[Fractal-Tess]

Hey guys,

First of all, I'd like to thank you for creating and maintaining oak - for the time that I've been using it, it has been nothing short of amazing.
This issue is aimed primarily to get some answers as to how things work with Oak, Deno, streams, and readers/writer.

Here is a sample of my code

Firstly, the client :

import { readableStreamFromReader } from '@deps'

const inputFile = await Deno.open('./testing/in.mkv')
const inputStream = readableStreamFromReader(inputFile)

const req = new Request('http://localhost:1234', {
  method: 'POST',
  body: inputStream
})

await fetch(req)

And secondly the server

import { Application, copy, readerFromStreamReader } from '@deps'

const app = new Application()

const file = await Deno.open('./out.mkv', {
  write: true,
  read: true,
  create: true
})

app.use(async ctx => {
  console.log('Started writing')
  const r = ctx.request.body({ type: 'stream' }).value.getReader()
  const reader = readerFromStreamReader(r)
  await copy(reader, file)

  console.log('Done writing')
})

const serverListen = app.listen({ port: 1234 })
console.log('Running')
await serverListen

Using this code, I'm able to transfer files between the two processes without any kind of problems, and that was my initial goal to start. But then I look at my RAM and I was surprised to say the least. This method eats through my RAM without any remorese.
But why thought, wasn't the intent of streamers to keep ram as free as possible while waiting on disk I/O?

So I tried chaning some stuff around and looking online for solutions, but this topic isn't too well documented and I also did not spend a vast amount of time hunting for answers since it was confusing to me anyway.

I tried a different setup - something like this:

import { Application, copy, readerFromStreamReader } from '../deps.ts'

const app = new Application()

const file = await Deno.open('./out.mkv', {
  write: true,
  read: true,
  create: true
})

app.use(async ctx => {
  console.log('Started writing')
  const reader = ctx.request.body({ type: 'reader' }).value
  await copy(reader, file)

  console.log('Done writing')
})

const serverListen = app.listen({ port: 1234 })
console.log('Running')
await serverListen

The only change is that I'm doing

  const reader = ctx.request.body({ type: 'reader' }).value
  await copy(reader, file)

Instead of

  const reader = ctx.request.body({ type: 'stream' }).value.getReader()
  const denoReader = readerFromStreamReader(reader)
  await copy(denoReader, file)

And with this solution, my RAM is not being murdered 10 seconds into the transfer.
Why? What is different? Also could you please give a mild explanation to the difference in boody.ts line 320 to 347.

Side question: Is it possible to speed this up? I'm doing those tests between 2 lan pcs that have 10gb network with NVME cache and the speed is not more than 100mbs.

Thank you very much!

[kitsonk]

Ultimately, this is more of a Deno question than specifically an oak one.

At the moment, Deno files use the Deno specific Reader and Writer interfaces, which while they made sense in the early days of Deno, now that web streams are sort of ubiquitous and the fetch API uses them extensively, they don't make a huge amount of sense (see: denoland/deno#11018).

The change in performance you are experiencing has to do with how the stream is offered up in { type: "stream" }. It actually performs a .tee() and it appears that the .tee() implementation isn't very efficient. When you get the { type: "reader" }, the stream isn't tee'ed, and just consumed. I think it is worth looking into and opened #492.

As far as speed issue, it is likely denoland/deno#13608 and/or denoland/deno#10157.