From 00274e23b43334bf2f2aeaf2d9bb053cf8bf05ac Mon Sep 17 00:00:00 2001 From: Martijn Dirkse Date: Fri, 28 Jun 2024 14:45:40 +0200 Subject: [PATCH] First draft about streaming --- .../advancedDevelopment/advancedDevelopment.rst | 2 ++ .../advancedDevelopment/streaming/streaming.rst | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 docs/source/advancedDevelopment/streaming/streaming.rst diff --git a/docs/source/advancedDevelopment/advancedDevelopment.rst b/docs/source/advancedDevelopment/advancedDevelopment.rst index becc2e6a..c32943eb 100644 --- a/docs/source/advancedDevelopment/advancedDevelopment.rst +++ b/docs/source/advancedDevelopment/advancedDevelopment.rst @@ -13,3 +13,5 @@ Chapter :ref:`gettingStarted` provides the basics of Frank development. This cha dockerDevel/dockerDevel narayana/narayana deployment/maven + streaming/streaming + diff --git a/docs/source/advancedDevelopment/streaming/streaming.rst b/docs/source/advancedDevelopment/streaming/streaming.rst new file mode 100644 index 00000000..daeb43a8 --- /dev/null +++ b/docs/source/advancedDevelopment/streaming/streaming.rst @@ -0,0 +1,17 @@ +*Work in progress* + +Streaming +========= + +This section is about an implementation detail of the Frank!Framework. The Frank!Framework handles large messages differently from small messages to save memory usage - large messages are *streamed*. Usually Frank developers do not have to worry about streaming. However, some errors you may encounter have to do with streaming and hence you need some knowledge about it. + +First, large messages are written to a temporary file before they are streamed. Under Linux, they are typically stored in the `/tmp` folder. If this folder is on a small disk partition, you may experience OutOfMemory exceptions. There is a property to configure the threshold for this behavior, `xxx`. If a message is larger than `xxx` bytes, it is written to a temporary file and then streamed. + +TODO: Find the right property name for `xxx` and the right comparison criterion. + +Second, messages that are already in a file are not copied to a temp file before they are streamed. This typically applies to the contents of a file handled by a `LocalFileSystemPipe`. This behavior is more general: the Frank!Framework has intelligence to determine whether a file has been stored already - it is only written to a temp file if it is not available otherwise. + +Ladybug helps you to debug issues with streaming. Checkpoints about streamed messages are clearly shown to be so. + +TODO: Add screenshot here. +TODO: What is different before 7.9 from the behavior from 7.9 and later?