Feature/tedefo 2944 business entities

modules/fields/pages/business-entities.adoc

@@ -0,0 +1,125 @@
+= Business Entities
+
+== Introduction
+
+In some fields there are patterns like:
+
+[source,json]
+----
+  "pattern" : {
+    "value" : "^(RESULT|((CON|RES|TEN|TPA|TPO|ORG)-\\d{4}))$",
+    "severity" : "ERROR"
+  },
+----
+
+Or this pattern:
+
+[source,json]
+----
+  "pattern" : {
+    "value" : "^(PROCEDURE|BUYER|RESULT|((PAR|LOT|GLO|RES|ORG)-\\d{4}))$",
+    "severity" : "ERROR"
+  }
+----
+
+Values like `LOT` are described in various places outside of the pattern.
+But some values like `PROCEDURE` only existed in the SDK.
+This gave us a hint that machine readable information was missing somewhere in the SDK.
+The information about identifiers was spread across the fields, nodes and patterns.
+
+To solve this we decided to add what we call "business entities" to the SDK.
+We also decided that each field should belong to one and only one business entity.
+The business entities are independent from the XML structure (nodes).
+Thus in `fields.json` the fields have a `businessEntityId` and we decided that nodes of the `xmlStructure` would have none.
+
+The business entities describe what we call the logical structure.
+
+== Business entities in the SDK
+
+Business entities are provided as a JSON array called `businessEntities` in the `fields.json`
+
+[horizontal]
+`id`:: Identifier of the business entity.
+`parentId`:: Identifier of the direct parent business entity (if applicable).
+`name`:: A short name which can be used for display (instead of the id).
+`description`:: A descriptive text, it is longer than the `name`.
+`repeatable`:: Indicates if the business entity can be repeated inside its parent.
+`schemeName`:: Indicates the schemeName (if applicable).
+`identifierFieldId`:: Indicates which field id identifies this business entity instance (if applicable).
+`captionFieldId`:: Indicates which field id can be used to display this business entity instance to a user (if applicable).
+`instanceIdentifier`:: An object to document the instance identifier (if applicable).
+`formatName`:: Found in `instanceIdentifier`, the format name, for now we have two, the "prefixedNumber" or "fixedText".
+`text`:: Found in `instanceIdentifier`, the identifier prefix or the fixed text.
+
+== Legacy data
+
+The business entities make some of the `fields.json` data obsolete.
+We will keep this data for backward compatibility reasons, it could be removed in a future SDK.
+
+For example:
+
+[source,json]
+----
+    "businessEntityId" : "groupOfLots",
+    "idScheme" : "GLO",
+    "schemeName" : "LotsGroup",
+----
+
+The field `schemeName` is coming from the business entity, it should ideally be retrieved via the business entity of the field.
+We will keep this `schemeName` in the `fields.json`.
+The same is true for the `idScheme`.
+
+There are similar cases for the nodes `identifierFieldId` and `captionFieldId`, both of which are provided by the business entities but the relationship is more indirect.
+
+== Usage
+
+=== User interfaces
+
+This can be used in a UI to group or filter fields by their business entity.
+It can be more natural to group them by business entity than to group them by XML structure node.
+In that case the fields can be first sorted by business entity and second by alphabetical order.
+
+=== Hardcoded identifiers
+
+Some identifiers like for example `PROCEDURE` were not present in the SDK except in the `pattern` of some fields.
+These identifiers are made of fixed text, in the business entities they have `"formatName"` with value `"fixedText"`.
+The identifier value is given by the `"text"`, for example `"text" : "PROCEDURE"`.
+
+=== Detecting changes
+
+The process of detecting changes could be automated inside of applications.
+In order to populate the data of the field `BT-13716-notice` the following logic can be used: 
+
+* If the data of a field was changed
+* Read the `businessEntityId` of the field via `fields.json`
+* Get the business entity by business entity id via `fields.json`
+
+After this there are two cases.
+
+==== Fixed identifiers
+
+The simplest case is if the business entity has the fixed identifier format `fixedText`.
+In that case the `text` gives the identifier that changed.
+
+==== Prefixed identifiers
+
+A more complex case is when the identifier has the prefixed number format `prefixedNumber`.
+In that case the `instanceIdentifier` field can be found via the business entity.
+The field business entity id of a field always points to the most precise business entity.
+But if the field is repeatable the application must pick the correct instance, the one to which the changed field belongs.
+
+=== XML generation and schemeName attribute
+
+In simple cases fields of type `id-ref` can only reference one type of identifier, in that case the scheme name is given.
+But in some cases for fields of type `id-ref` the user can select between multiple identifier types.
+For example `ORG` or `TPO`, this means the user will decide the `schemeName`, it cannot be determined in advance.
+
+Assuming the user selected `Touch point one (TPO-0001)`:
+
+* The application puts the correct id `TPO-0001` into the field
+* From the value `TPO-0001` the application can extract the `TPO` prefix 
+* In `fields.json` the application can find the business entity that has the prefix `TPO`
+* From the business entity the application can infer the `schemeName` which is `touchpoint`
+
+The `schemeName` is an XML attribute which must be set on some XML elements and the value must be `touchpoint` in this example.
+It is possible to achieve this without the business entities but the business entities make the link more obvious.

modules/fields/pages/index.adoc

@@ -65,7 +65,7 @@ We are using this file to provide the definition of all fields to our
 form-filling application (eNotices2). Each form is then made of a specific
 subset of those fields.
 
-The field repository is mainly an array of fields, preceded by some version information and the general structure of XML notices:
+The field repository is mainly an array of fields, preceded by some version information, the business entities and the general structure of XML notices:
 
 [source,json]
 ----
@@ -76,11 +76,15 @@ The field repository is mainly an array of fields, preceded by some version info
     "version": "0.2.53",
     "createdOn": "2021-11-20T16:46:39"
   },
-  "xmlStructure" : [ // <4>
+  "businessEntities" : [ // <4>
+    { ... },
+    ...
+  ]
+  "xmlStructure" : [ // <5>
     { ... },
     ...
   ],
-  "fields": [ // <5>
+  "fields": [ // <6>
     { ... },
     ...
   ]
@@ -89,8 +93,9 @@ The field repository is mainly an array of fields, preceded by some version info
 <1> Version of the UBL standard used.
 <2> Version of the eForms SDK the file belongs to.
 <3> Version number and date of the data used to create this file.
-<4> General structure of XML notices. See the xref:xml-structure.adoc[] page.
-<5> List of fields.
+<4> Logical structure defined by the business entities. See the xref:business-entities.adoc[] page.
+<5> General structure of XML notices. See the xref:xml-structure.adoc[] page.
+<6> List of fields.
 
 == Field properties
 
@@ -108,16 +113,19 @@ The various characteristics of each field are indicated as properties of the JSO
     "xpathRelative" : "cbc:RegulatoryDomain", // <6>
     "xsdSequenceOrder" : [ { "cbc:RegulatoryDomain" : 18 } ], // <7>
     "type" : "code", // <8>
-    "legalType" : "CODE", // <9>
-    "maxLength" : 400, // <10>
-    "repeatable" : { // <11>
-      "value" : false
+    "businessEntityId" : "procedure", // <9>
+    "legalType" : "CODE", // <10>
+    "maxLength" : 400, // <11>
+    "repeatable" : { // <12>
+      "value" : false,
+      "severity" : "ERROR"
     },
-    "codeList" : { // <12>
+    "codeList" : { // <13>
       "value" : {
-        "id" : "legal-basis",
+        "id" : "eforms-legal-basis",
         "type" : "flat"
-      }
+      },
+      "severity" : "ERROR"
     }
   }, {
     "id" : "BT-137-Lot",
@@ -128,13 +136,15 @@ The various characteristics of each field are indicated as properties of the JSO
     "xpathRelative" : "cbc:ID",
     "xsdSequenceOrder" : [ { "cbc:ID" : 2 } ],
     "type" : "id",
-    "idScheme" : "LOT", // <13>
+    "attributes" : [ "BT-137-Lot-Scheme" ],
+    "businessEntityId" : "lot",
+    "idScheme" : "LOT", // <14>
     "legalType" : "IDENTIFIER",
     "repeatable" : {
       "value" : false,
       "severity" : "ERROR"
     },
-    "forbidden" : { // <14>
+    "forbidden" : { // <15>
       "value" : false,
       "severity" : "ERROR",
       "constraints" : [ {
@@ -143,7 +153,7 @@ The various characteristics of each field are indicated as properties of the JSO
         "severity" : "ERROR"
       } ]
     },
-    "mandatory" : { // <15>
+    "mandatory" : { // <16>
       "value" : false,
       "severity" : "ERROR",
       "constraints" : [ {
@@ -152,7 +162,7 @@ The various characteristics of each field are indicated as properties of the JSO
         "severity" : "ERROR"
       } ]
     },
-    "pattern" : { // <16>
+    "pattern" : { // <17>
       "value" : "^LOT-\\d{4}$",
       "severity" : "ERROR"
     }
@@ -165,7 +175,9 @@ The various characteristics of each field are indicated as properties of the JSO
     "xpathRelative" : "efac:TenderLot/cbc:ID",
     "xsdSequenceOrder" : [ { "efac:TenderLot" : 14 }, { "cbc:ID" : 1 } ],
     "type" : "id-ref",
-    "idSchemes" : [ "LOT", "GLO" ], // <17>
+    "attributes" : [ "BT-13714-Tender-Scheme" ],
+    "businessEntityId" : "tender",
+    "idSchemes" : [ "LOT", "GLO" ], // <18>
     "legalType" : "IDENTIFIER",
     "repeatable" : {
       "value" : false,
@@ -192,16 +204,17 @@ The various characteristics of each field are indicated as properties of the JSO
 <6> Location of the field in an XML notice, relative to its parent node.
 <7> Position of each XML element relative to its siblings. <<xsdSequenceOrder,See below for details>>. 
 <8> Technical data type of the field.
-<9> Data type of the business term, as indicated in the eForms Regulation.
-<10> Maximum number of characters allowed in the value of the field, optional. 
-<11> Indicates if the field can appear more than once inside its container
-<12> Identifier of the code list from which the field value must belong.
-Applicable only for fields of type "code" or "internal-code"
-<13> Identifier scheme used by a field with `"type" : "id"`.
-<14> This property provides information on when a field's presence is forbidden.
-<15> This property provides information on when a field's value is mandatory.
-<16> A RegEx pattern the field's value must match.
-<17> Array indicating the valid `idScheme` values which this identifier reference field (`"type" : "id-ref"`) can reference.
+<9> The id of the business entity the field belongs to.
+<10> Data type of the business term, as indicated in the eForms Regulation.
+<11> Maximum number of characters allowed in the value of the field, optional. 
+<12> Indicates if the field can appear more than once inside its container
+<13> Identifier of the code list from which the field value must belong.
+Applicable only for fields of type "code".
+<14> Identifier scheme used by a field with `"type" : "id"`.
+<15> This property provides information on when a field's presence is forbidden.
+<16> This property provides information on when a field's value is mandatory.
+<17> A RegEx pattern the field's value must match.
+<18> Array indicating the valid `idScheme` values which this identifier reference field (`"type" : "id-ref"`) can reference.
 
 
 === Property Values
@@ -221,15 +234,16 @@ The properties that are assigned with static values are listed below:
 `xpathRelative`:: The XPath of the field relative to its parent `node`. See `parentNodeId`.
 [#xsdSequenceOrder]`xsdSequenceOrder`:: This indicates, for each XML element that is part of `xpathRelative`, the position that this element should have relative to its siblings in order to have an XML instance valid against the XML Schema. This information is extracted from the definition of the corresponding complex type in the XSD files.
 `type`:: The data type of the field. xref:#data-types[See Field data types].
-`attributeOf`:: If the field represents an XML attribute this indicates the identifier of the field representing the XML element which has the attribute.
 `attributeName`:: The name of the XML attribute represented by the field.
+`attributeOf`:: If the field represents an XML attribute this indicates the identifier of the field representing the XML element which has the attribute.
 `attributes`:: An array of one or more identifiers of the fields representing the XML attributes the XML element represented by the field can have.
 `presetValue`:: The value used to pre-fill a field. This property is always a string, but the value should be converted to the same type as the field. The value "\{NOW\}" corresponds to the current date or time.
-`legalType`:: The data type of the Business Term associated with the field as defined by the eForms Regulation.
-`maxLength`:: The maximum number of characters that the field can hold.
+`businessEntityId`:: The id of the business entity the field belongs to. See xref:fields:business-entities.adoc[Business Entities]. 
+`idSchemes`:: Applicable only to fields of type `id-ref`. Provides an array of valid identifier schemes that this `id-ref` field can reference.
 `idScheme`:: Applicable only to fields of type `id`. Indicates the identifier scheme for this `id` field (e.g. "TPO" for Touch Point).
 `schemeName`:: Applicable only to fields of type `id`. Indicates the value that should be indicated for this field in the `schemeName` attribute in the XML.
-`idSchemes`:: Applicable only to fields of type `id-ref`. Provides an array of valid identifier schemes that this `id-ref` field can reference.
+`legalType`:: The data type of the Business Term associated with the field as defined by the eForms Regulation.
+`maxLength`:: The maximum number of characters that the field can hold.
 
 Static properties are always assigned with a scalar value (a string, a boolean, a number, array of strings, etc.). If there is no value defined, the property is omitted.
 
@@ -417,7 +431,7 @@ In the following snippet, a dynamic value is assigned to the `forbidden` dynamic
   } ]
 },
 ----
-<1> The default value of the property will be false in this example
+<1> The default value of the property will be false in this example.
 <2> List of constraints for this property.
 <3> The first constraint in this example specifies a different value than the default one in the case that the field is used in one of the notice types indicated.
 <4> The value of this dynamic property in the case that the constraint applies is indicated here.
@@ -458,7 +472,7 @@ The value of the `condition` property of a constraint is a string representing a
 
 This expression is made of two parts:
 
-* The context under which the expression is evaluated. It's often the parent node of the field.
+* The context under which the expression is evaluated. It is often the parent node of the field.
 * The boolean expression itself.
 
 

modules/fields/pages/xml-structure.adoc

@@ -2,7 +2,7 @@
 
 == Introduction
 
-The field metadata in the fields repository file (fields.json) describe the properties and behaviour of each field. Each field, representing an element in an XML document, is ultimately part of a greater tree-like structure. Actually, there is more than one such structure that each field is a part of:
+The field metadata in the fields repository file (fields.json) describes the properties and behaviour of each field. Each field, representing an element in an XML document, is ultimately part of a greater tree-like structure. Actually, there is more than one such structure that each field is a part of:
 
 * The physical structure in which each field is stored inside the notice XML file.
 This page documents how this physical structure is represented in the SDK.
@@ -15,7 +15,7 @@ This physical structure (XML structure) is included in the eForms SDK to allow i
 
 == XML structure in the field repository
 
-The XML structure is provided as a flat list of "nodes" at the end of the field repository file (fields.json). Every node represents an XML element. Each node can "contain" other nodes and/or fields. Every field provides a `parentNodeId` property which points to its parent node in the XML structure. Every node provides a `parentId` property which points to its parent node. Using this information you can reconstruct the entire XML tree. The absolute XPath of each node is also provided as well as the XPath relative to its parent node.
+The XML structure is provided as an array called "xmlStructure" near the top of field repository file (fields.json). Every item represents an XML element. Each item (a node) can "contain" other nodes and/or fields. Every field provides a `parentNodeId` property which points to its parent node in the XML structure. Every node provides a `parentId` property which points to its parent node (except the root). Using this information you can reconstruct the entire XML tree. The absolute XPath of each node is also provided as well as the XPath relative to its parent node.
 
 NOTE: The `xmlStructure` list provided in fields.json is a flat list. The logical hierarchy of the nodes is represented using the `parentId` property.
 

modules/fields/partials/nav.adoc

@@ -1,2 +1,3 @@
 * xref:{page-component-version}@eforms:fields:index.adoc[Field metadata]
-** xref:{page-component-version}@eforms:fields:xml-structure.adoc[XML Structure]
+** xref:{page-component-version}@eforms:fields:xml-structure.adoc[XML Structure]
+** xref:{page-component-version}@eforms:fields:business-entities.adoc[Business Entities]