diff --git a/.github/workflows/frank-runner-example-cited-correctly.yml b/.github/workflows/frank-runner-example-cited-correctly.yml index 2d78fcc8..0c36374b 100644 --- a/.github/workflows/frank-runner-example-cited-correctly.yml +++ b/.github/workflows/frank-runner-example-cited-correctly.yml @@ -3,7 +3,6 @@ on: push: branches: - master - pull_request: workflow_dispatch: jobs: testing: diff --git a/.github/workflows/no-console-warnings.js.yml b/.github/workflows/no-console-warnings.js.yml index 3ea2e986..ef3be061 100644 --- a/.github/workflows/no-console-warnings.js.yml +++ b/.github/workflows/no-console-warnings.js.yml @@ -3,7 +3,6 @@ on: push: branches: - master - pull_request: workflow_dispatch: jobs: testing: diff --git a/misc/LADYBUG_USER_STORIES.md b/misc/LADYBUG_USER_STORIES.md index c6d912af..a59d2131 100644 --- a/misc/LADYBUG_USER_STORIES.md +++ b/misc/LADYBUG_USER_STORIES.md @@ -12,6 +12,8 @@ Ladybug is not limited to work with the Frank!Framework; it is a tool that can b When a report has been captured it is a Java object managed by Ladybug. Ladybug then extracts metadata from the report. This helps for searching and browsing reports because the amount of metadata is much less than the amount of data of all reports. The users should have options to define what information captured in reports should be treated as metadata. Ladybug should also be able to transform reports into XML format. Then users can interact with the reports through Xpath and XSLT. +Metadata extracted by ladybug should not be confused with metadata generated by the Frank!Framework. The Frank!Framework generates metadata along with each message. These metadata fields are for example the message length and the character set. For messages that come from a file, a metadata field generated by the Frank!Framework can be the file's location. And the mime type (xml/json/zip) is often available as FF!-generated metadata. FF! generated metadata is different from session keys; session keys appear as separate checkpoints within Ladybug. + Ladybug can be used as a debug tool. Ladybug provides a user interface that shows a table of all captured reports. When the user clicks a report, it is opened in a tree view. Each checkpoint is a node in the tree. Collapsing and expanding nodes is described in more detail in the user stories below. In the tree view, the user can see all intermediate results of processing each message. Each checkpoint has a meaningful name. The Frank!Framework names each checkpoint after the adapter or pipe name. Therefore the user can relate the checkpoint's message to the debugged/tested source code, or to the Frank application being tested. When you click a node in the tree, the corresponding message inside the checkpoint should be shown. Ladybug can be used as a tool for automated testing. Ladybug reports can act as test cases because they can be rerun. Rerunning a report means that the Java code that produced the report is re-executed. The messages inside the related checkpoints are compared to the messages in the original report. The test succeeds if these new messages are considered equivalent to the original ones; otherwise the test fails. Ladybug's user interface has separate tabs for new reports (Debug) and reports that are meant as test cases (Test). In the Debug tab, the user has a button to copy reports to the test tab. The user has options to edit a report to convert the raw capture to a useful testcase. @@ -98,6 +100,10 @@ NOTE: We implement story **120** because Frank developers can write Spring confi NOTE: Story **500** is implemented in the ladybug backend because you can create views in Spring configuration files. Story **510** is implemented because the debug tab has a dropdown box in which you can select a view from the list of available views. +**520:** Given is that I as a user am doing some analysis about my reports and that I have put some filters on the report table. Given is that I select another view during my analysis. For each metadata field that appears in the table for both views, I want that my filter on that field remains. This way I can continue my analysis without having to retype my filters unnecessarily. + +**600:** As a service manager, I want that all reports I see in the report table are original captures. I do not want to see reports that were edited after capture (as can be done according to story **2060**) so that I can be sure about the integrity of my data. + # I want to understand how the message captured by a report was processed **1000:** As a support engineer or Frank developer, I want to see the name of each checkpoint that is shown in the tree view. The name is not required to be unique. @@ -114,15 +120,98 @@ NOTE: Story **500** is implemented in the ladybug backend because you can create * Thread start point. * Thread end point. +TODO: How should InputWrapper, InputValidator, OutputValidator, OutputWrapper, scheduling-related elements and monitoring related elements appear? + **1020:** As a support engineer or Frank developer, I want to see a start point as the parent node of the checkpoints that come after it. This applies recursively: a start node inside a start node causes subsequent nodes to be grand children of the first start node. An end point that is a direct child of a start node is also the last child. Subsequent nodes are siblings of the ended start node. This also applies recursively. +NOTE: The figure below explains story **1020** and **1240** about the existance of a root node: + +![Nesting of nodes and root node](./ladybug-user-stories/nesting-of-report-nodes.jpg) + +**1025:** The following table shows how the elements of a Frank configuration should be shown in a Ladybug report: + +Frank type | Item | Ladybug type | Comment +:--------- | :--- | :----------- | :------ +\- | root node | - | Shown as folder, see story **1056** +Pipeline | message | Start point | Message read by listener +Pipeline | message | End point | Output message of pipeline +Pipeline | error | Abort point | +Pipeline | session key | input point | Additional information like timestamp received or URL on which a http request was received. +Pipeline | session key | output point | Session key related to pipeline output +Receiver | - | not shown | +Listener | - | not shown | Additional information in Pipeline session keys +Sender | message | Start point | Message sent +Sender | message | End point | Response message +Sender | error | Abort point | +Sender | session key | input point | +Sender | parameter | input point | +Sender | session key | output point | +Pipe | message | Start point | Message received +Pipe | message | End point | Output message +Pipe | error | Abort point | +Pipe | session key | Input point | +Pipe | parameter | Input point | TODO: Is this correct? +Pipe | session key | Output point | +Pipe | source code, loaded config syntax (*) | Info point | + +NOTE: About (*). There is a difference between "loaded config syntax", or syntax 1, and "original syntax", or syntax 2. Here is an example of "loaded config syntax": + +``` + + + + +``` + +And here is the same in original config syntax, or syntax 2: + +``` + + + + + +``` + +For completeness, it should be noted that Frank configurations can also be written in syntax 1 directly; mixed syntax 1 and syntax 2 is also allowed. + **1030:** As a support engineer or Frank developer, I want to be able to collapse and to expand each parent node in the tree view. Each parent node can be *expanded* which means that the start node and its children, including the end node, are shown. A parent node can also be *collapsed* which means that its descendants are not shown. This way I can hide details of how a message was processed. -**1033:** As a support engineer or Frank developer, I want to have a checkpoint for each pipe such that I can see how each pipe transformed the incoming message. +**1031:** As a user, I want to click only once to expand or collapse a node. In other words: if I click on the `>` or `v` sign to the left of an expandable node that was not selected yet, then I want the node to become the selected one AND I want to toggle its expanded or collapsed state. + +**1032:** Given is that I am looking at the white box view (story **1310**). As a user, I want to see a start checkpoint and an end checkpoint for each pipeline so that I can see how the pipeline transformed the message. If an error occurred in the pipeline, I do not want an end point but an abort point that shows me information about the error. + +**1033:** Given is that I am looking at the white box view (story **1310**). As a user, I want to have a start checkpoint and an end checkpoint for each pipe so that I can see how each pipe transformed the incoming message. If an error occurred, I do not want to see and end point but an abort point that shows me information about the error. + +**1034:** Given is that I am looking at the white box view (story **1310**). As a user I want to have a checkpoint for each session key that is input or output of a pipeline. + +**1035:** Given is that I am looking at the white box view (story **1310**). As a user I want to have a checkpoint for each parameter of a pipe. + +**1036:** Given is that I am looking at the white box view (story **1310**). As a user I want that pipe checkpoints have child checkpoints for session keys. There should be an input point for each session key that is used by the pipe. There should be an output point for each session key that is generated or changed by the pipe. + +**1037:** Given is that I am looking at the white box view (story **1310**). As a user I want to have a start point and an end point for each sender so that I can see the sent message and the response. If an error occurred, I do not want an end point but an abort point that shows me information about the error. + +**1038:** Given is that I am looking at the white box view (story **1310**). If there are session keys that contain additional information about the message sent by a sender, then I want the start point of the sender to contain child checkpoints (input points) for these session keys. + +**1039:** Given is that I am looking at the white box view (story **1310**). If a there are sessions keys related to the response returned to a sender, then I want the start point of the sender to contain child checkpoints (output points) for these session keys. + +**1040:** Given is that I am looking at the white box view (story **1310**). If a sender has parameters, then I want the start point of the sender to contain child nodes (input points) for these parameters. + +**1041:** Given is that I am looking at the white box view (story **1310**). For each pipe, I want to see an info checkpoint with the source code of that pipe. + +**1048:** As a support engineer or Frank developer, I want a single report when my adapter calls another adapter via a JavaListener. This way, I do not have to browse multiple reports to examine how my incoming message was processed. The sub-adapter checkpoints should have a common ancestor. When I collapse that ancestor, I want to see only one node for everything done in the sub-adapter. -**1037:** As a support engineer or Frank developer, I want a single report when my adapter calls another adapter via a JavaListener. This way, I do not have to browse multiple reports to examine how my incoming message was processed. The sub-adapter checkpoints should have a common ancestor. When I collapse that ancestor, I want to see only one node for everything done in the sub-adapter. +**1050:** As a support engineer or Frank developer, I want to see the value of the selected node in the debug tree. -**1050:** As a support engineer or Frank developer, I want to see the message and the metadata of a checkpoint when I click on it. +**1052:** When I have selected the checkpoint of a session key or a parameter in the tree view, the value I want to see (story **1050**) is the value of the session key or the parameter value. + +**1054:** When I have selected a start point / end point of a pipeline / pipe / sender, I want to see the incoming / outgoing message. Along with that message, the FF!-generated metadata should be shown. + +**1056:** When I have selected the root node of a report, the value I want to see is the report in XML form, the ladybug-generated metadata, the description (story **2040**) and the XSLT transformation (story **2010**). + +**1058:** Given is that I have selected a checkpoint like described in story **1054**. If the message is too large, the message should be shown truncated and a warning should be shown. The warning should make clear that the user is not seeing the complete message. + +**1059:** As a Frank developer, I want to be able to configure a threshold for story **1058**. When a message is longer than the configured threshold, then the message is shown truncated. **1100:** As a support engineer or Frank developer, I want the option to remove a report from the tree view when I am done with it. This does not mean that a report is removed from persistent storage. @@ -136,18 +225,13 @@ NOTE: Story **500** is implemented in the ladybug backend because you can create **1240:** Story **1200** has as a consequence that each report in the tree view should have a single root node, otherwise you could not see which node belonged to each report. -NOTE: Presently, some features of Ladybug are attached to the root node of a report. +**1250:** Given is that I am doing some analysis involving multiple reports. Given is also that I have configured the tree view to allow multiple reports (story **1200**). If the tree view is empty and if I open multiple reports simultaneously, the reports should appear in the tree view in the same sequence as they are in the report table. This way the reports in the tree view have a predictable sequence, helping me to do my analysis more efficiently. -* In the old Echo2 GUI, there is an "Edit" button in the tree view that puts the report in "edit mode". In edit mode, you have an editable message when you click a node in the tree view. There are also metadata fields, some of them editable and some of them read-only. When you select the top level node, you can edit a description. This way you can add a description of a report that exists in the debug tab. -* In the test tab you can "open" a report. This is the user interface for main user story [I want to turn a report into a test case](#i-want-to-turn-a-report-into-a-test-case). Doing so opens a new tab with a tree view with editable node information next to it, like "edit mode" in the debug tab's tree view. Also in this case the root node of a report is relevant in the current implementation of Ladybug. -* When you click the root node of a report you see it as XML. This combines nicely with an edit field to enter an XSLT transformation (story **2010**) -* When you click the root node of a report you have the option there to enter the description (story **2040**). - -**1300:** Given is that I as a service manager am investigating an issue with my application. In the debug tree, I want the option to see only the checkpoints within each report report that are about communicating with external systems. This is currently known as the black box view. +**1300:** Given is that I as a service manager am investigating an issue with my application. In the debug tree, I want the option to see only the checkpoints within each report that are about communicating with external systems. This is currently known as the black box view. **1310:** Given is that I as a support engineer or Frank developer am investigating an issue. In the debug tree, I want the option to see all the checkpoints within each report. This is known as the white box view. -**1320:** Given is that I as a user am investigating an issue. In the debug tree, I want the option to see a selection of the checkpoints within each report. More checkpoints than the black box view. This is known as the gray box view. +**1320:** Given is that I as a user am investigating an issue. In the debug tree, I want the option to see a only the following checkpoints: Checkpoints about communicating with external systems plus checkpoints related to adapters calling each other (typically through `IbisLocalSender`). When adapters call each other, the flow of the message should appear in one report as stated in story **1048**. This is known as the gray box view. **1330:** Given is that multiple users with multiple interests use the debug tree. As a user I want that the view I select (story **510**) also determines which checkpoints of my reports are shown: **1300**, **1310** or **1320**, in other words white box, gray box or black box. @@ -157,32 +241,50 @@ NOTE: Presently, some features of Ladybug are attached to the root node of a rep **2010:** Given is that I am building an automated test from a report. As a Frank developer I want the option to configure an XSLT transformation that is applied to each message inside each checkpoint. When a report is rerun, the XSLT transformation is applied to the produced messages and it is applied to the messages inside the checkpoints. For each checkpoint, the two transformation results are compared. This way, irrelevant differences can be ignored. Irrelevant differences are produced for example if the current time is used by a Frank application. +**2012:** Given is that I am using a report as a test case and that I did not edit an XSLT transformation as indiciated in story **2010**. This means that I am rerunning such a report. As a user, I want the following. If the inputs and the outputs and the behavior of the external systems are the same, then I want my test to succeed because I do not want false negatives. + +NOTE: Story **2012** can be implemented by giving original-capture reports a default XSLT transformation. + **2020:** Given is that I am building an automated test from a report. As a Frank developer I want the option to declare some checkpoints **stubbed**, checkpoints that correspond to calls to external systems. When the report is rerun, the Frank!Framework should not call the external systems again but it should return the results already stored in the stubbed checkpoints. This way, only the logic within the Frank configuration captured in the report is tested, not the behavior of the external systems. Stubbing allows Frank developers to work with a simpler test environment, because the test does not require access to external systems. **2030:** Given is that I am building an automated test from a report. As a Frank developer I want the option to base parameterized tests upon my report. This means that I introduce variable references in my report. I can create a new report by specifying values for the variables. I can do this for multiple possibilities to set the variables, all resulting in a new clone of the report. **2040:** Given is that I am building an automated test from a report. As a Frank developer I want the option to add a *description* to my report. In the description I can document what the report viewed as a test case should test. +**2050:** Given is that I am building an automated test from a report. If I have done some editing and save my results, I want that all views of the report are updated automatically without the need to press a refresh button. + +**2060:** Given is that I as a Frank developer am testing my application in the DTAP test environment. When I have a report in the debug tab for which I do not like the results, I want the option to edit the report there, similar to stories **2000** - **2050**. I do this with the aim of rerunning the report. If the edited report behaves as I want, I can download it and send to my collleague Frank developer so that my colleague can fix our app. + +**2070:** Given is that I as a Frank developer have edited a report in the debug tab (story **2060**). After saving my edits, a report with my edits should be open in the debug tree so that I still have access to it, even though it is not in the report table as is required by story **600**. + +TODO: Should we allow stories **2060** and **2070** when only one report is allowed in the debug tree? We could allow that. The implication is that when you edit a report in the debug tree, that the original report is removed from the debug tree when the edits are saved. + # I want to re-run (test) reports to test my Frank application **3000:** Given is that I am testing my application as a Frank developer. When I enter the test tab I want to see all reports that I have prepared as test cases. +**3005:** For each report I see in the test tab, I want to see its name and its description (story **2040**). + **3010:** Given is that I am testing my application as a Frank developer. I want the option to *rerun* reports. Rerunning a report means that the Frank adapter that produced the report is re-executed. The same input message is supplied. For each pipe, the produced output is compared with the value stored with the corresponding checkpoint. The test succeeds if the new messages are the same after applying the configured XSLT transformation (user story **2010**). -**3020:** Given is that I am testing my application as a Frank developer. I want the option to rerun a single report. +**3020:** Given is that I am testing my application as a Frank developer. I want the option to rerun a single report in the test tab. **3030:** Given is that I am testing my application as a Frank developer. I want the option to organize the reports in the Test tab. I want to create groups of tests that can have sub-groups. A tree structure in which the composite nodes are test groups and the leaf nodes are reports. This structure gives me an overview of the tests I have. -**3040:** Given is that I am testing my application as a Frank developer. I want the option to select test cases and groups of test cases. I want the option to rerun the tests I selected. This gives me fine-grained control over my test runs when I want to test parts of my Frank application. +**3040:** Given is that I am testing my application as a Frank developer. I want the option to select test cases and groups of test cases in the test tab. I want the option to rerun the tests I selected. This gives me fine-grained control over my test runs when I want to test parts of my Frank application. **3050:** Given is that I am testing my application as a Frank developer. I want the option to rerun all reports that are in the Test tab, allowing me to test my Frank application with a single click. -**3060:** Given is that I have reran a few reports and that I want to do new tests. As a user I want the option to reset the report generator, which means that all shown test results are cleared. When I rerun my tests after resetting the report generator, I want that all test results I see come from test started after the moment of resetting. +**3060:** Given is that I have reran a few reports in the test tab and that I want to do new tests. As a user I want the option to reset the report generator, which means that all shown test results are cleared. When I rerun my tests after resetting the report generator, I want that all test results I see come from tests started after the moment of resetting. NOTE: Story **3060** is not trivial because rerunning reports happens in the background. +**3070:** Given is that I as a Frank developer are testing my app in the DTAP test environment. I want the option to rerun a single report in the debug tab. This is relevant in the context of story **2060**. + **3100:** Given is that I have rerun a report as a Frank developer and that this test failed. I want the option to compare the original capture to the new results. I want to see the two datasets next to each other such that I can see what is the same and what is different. +**3108:** Given is that I am comparing a report in the context of story **3100**. On one side I have the checkpoints in the report and on the other side I have the checkpoints produced by the rerun. As a user, I want that the selected checkpoint is synchronized. If I select a checkpoint on one side, I want that the corresponding checkpoint on the other side is selected automatically. This helps me to do my analysis more quickly. + **3110:** Given is that I have rerun a report and that this test failed. Given is also that the number of checkpoints at both sides is different. As a user I want that Ladybug chooses intelligently what checkpoints to the left belong to what checkpoints to the right. When I look at one checkpoint I want to see the corresponding checkpoint next to it. I do not want to search on the other side for the corresponding checkpoint. **3120:** Given is the context of story **3110**. I want that checkpoints corresponding to the same inputs and outputs to external systems are matched. This statement is intended to clarify the "intelligent choice" of story **3110**. @@ -193,9 +295,7 @@ NOTE: Details of the algorithm implied in **3110** and **3120** are not needed h **3210:** As a Frank developer I want to use reports as a means to communicate with my colleagues. I want to construct a report and send it to someone else, asking him to update our application such that the test will pass. -NOTE: Jaco suggested that there should be an edit button in the debug tab. A disadvantage is that if reports can be edited in the debug tab, then you cannot be sure that all reports in the debug tab are original captures. Jaco is also satisfied if there is a disabled edit button in the debug tab. This button can have a mouse-over text that suggests copying to the test tab first. - -TODO: Discuss whether we want that reports can be downloaded and uploaded in the debug tab. It may be better to only have this in the test tab. +**3220:** Because of the context of story **2060**, downloading and uploading reports should be possible both in the debug tab and in the test tab. # I want to configure whether my Frank application does produce reports @@ -229,10 +329,11 @@ The following features should only be available with the DataAdmin role. * Upload report. * Move report. * Run report. -* TODO: Something about resetting a ReportRunner. The following features should be available with the Tester role (and not necessarily the DataAdmin role): # Miscelaneous **5000:** As a service manager I want Ladybug to be self-explanatory. The user interface should help me to understand what is going on, for example by providing mouse-over texts. + +**5010:** Given is that ladybug is not behaving as expected. As a user I want to see a version number for the ladybug frontend. It should point to a unique commit of the ladybug-frontend GitHub project. This way I can read the source code of the ladybug frontend I am not happy with. diff --git a/misc/ladybug-user-stories/nesting-of-report-nodes.jpg b/misc/ladybug-user-stories/nesting-of-report-nodes.jpg new file mode 100644 index 00000000..b0eb09c7 Binary files /dev/null and b/misc/ladybug-user-stories/nesting-of-report-nodes.jpg differ