Skip to content

Commit

Permalink
Process feedback
Browse files Browse the repository at this point in the history
TODO: If this is not about streaming, rename the rst file
  • Loading branch information
Martijn Dirkse committed Jul 1, 2024
1 parent 1b17f25 commit 0f56862
Showing 1 changed file with 4 additions and 8 deletions.
12 changes: 4 additions & 8 deletions docs/source/advancedDevelopment/streaming/streaming.rst
Original file line number Diff line number Diff line change
@@ -1,15 +1,11 @@
*Work in progress*

Streaming
=========
About large messages
====================

This section is about an implementation detail of the Frank!Framework. The Frank!Framework handles large messages differently from small messages to save memory usage - large messages are *streamed*. Usually Frank developers do not have to worry about streaming. However, some errors you may encounter have to do with streaming and hence you need some knowledge about it.
The Frank!Framework treats large messages in a special way to save memory. Frank developers sometimes need some knowledge about this to fix issues with their Frank applications. There is a property ``message.max.memory.size`` to configure the threshold: messages larger than this size are treated specially.

First, large messages are written to a temporary file before they are streamed. Under Linux, they are typically stored in the ``/tmp`` folder. If this folder is on a small disk partition, you may experience OutOfMemory exceptions. There is a property to configure the threshold for this behavior, ``xxx``. If a message is larger than ``xxx`` bytes, it is written to a temporary file and then streamed.

TODO: Find the right property name for ``xxx`` and the right comparison criterion.

Second, messages that are already in a file are not copied to a temp file before they are streamed. This typically applies to the contents of a file handled by a ``LocalFileSystemPipe``. This behavior is more general: the Frank!Framework has intelligence to determine whether a file has been stored already - it is only written to a temp file if it is not available otherwise.
First, large messages are written to a temporary file before they are processed. Under Linux, they are typically stored in the ``/tmp`` folder. If this folder is on a small disk partition, an OutOfMemory Java exception may result. As an exception, messages that are already in a file are not copied to a temp file before they are processed. This exception typically applies to the contents of a file handled by a ``LocalFileSystemPipe``. This exception is more general: the Frank!Framework has intelligence to determine whether a file has been stored already - it is only written to a temp file if it is not available otherwise.

Ladybug helps you to debug issues with streaming. Checkpoints about streamed messages are clearly shown to be so.

Expand Down

0 comments on commit 0f56862

Please sign in to comment.