Feature/tedefo 2944 business entities

modules/fields/pages/business-entities.adoc

@@ -0,0 +1,118 @@
+= Business Entities
+
+== Introduction
+
+Here we will cover historical reasons for the addition of the business entities to the SDK.
+In some fields there was a pattern like:
+
+[source,json]
+----
+  "pattern" : {
+    "value" : "^(RESULT|((CON|RES|TEN|TPA|TPO|ORG)-\\d{4}))$",
+    "severity" : "ERROR"
+  },
+----
+
+Or this pattern:
+
+[source,json]
+----
+  "pattern" : {
+    "value" : "^(PROCEDURE|BUYER|RESULT|((PAR|LOT|GLO|RES|ORG)-\\d{4}))$",
+    "severity" : "ERROR"
+  }
+----
+
+Values like `LOT` are described in the SDK in various places outside of the pattern.
+But some values like `PROCEDURE`, `BUYER` only existed in these patterns.
+This gave us a hint that machine readable information was missing somewhere in the SDK.
+Also information about identifiers was spread across the fields, nodes and patterns, nothing centralised this information.
+
+To solve this we decided to add what we call "business entities" to the SDK.
+We also decided that each field should belong to one and only one business entity.
+The business entities are independent from the XML structure (nodes) even if there are common points.
+In `fields.json` the fields have a `businessEntityId` and we decided that nodes of the `xmlStructure` nodes would only have a business entity if repeatable.
+
+The business entities describe what we call the logical structure.
+
+== Business entities in the SDK
+
+Business entities are provided as a JSON array called `businessEntities` in the `fields.json`
+
+* `id` Identifier of the business entity.
+* `name` A short name which can be used for display (instead of the id).
+* `description` A descriptive text, it is longer than the `name`.
+* `repeatable` Indicates if the business entity can be repeated.
+* `repeatsWithNodeId` Indicates the node the business entity repeats with, present if it repeats.
+
+* `changeIdentification` An object to document the what should be done when something changed in the business entity.
+**  `identifyInChangeNotice` A boolean to determine if a change in this business entity should be reported
+**  `useInstanceIdentifier` A boolean to determine if the instanceIdentifier should be used
+**  `changeIdentifier` A string to use to identify the change, if present it should be used
+
+* `instanceIdentifier` An object containing instance identifier information (if applicable).
+**  `referencedBusinessEntityId` A referenced business entity identifier, if present it means this business entity relies on another business entity for the identifiers.
+**  `prefix` Found in `instanceIdentifier`, the identifier prefix.
+**  `schemeName` Indicates the scheme name (if applicable).
+**  `identifierFieldId` Indicates which field id identifies this business entity instance (if applicable).
+**  `captionFieldId` Indicates which field id can be used to display this business entity instance to a user (if applicable).
+
+== Future
+
+The business entities could make some of the `fields.json` data obsolete.
+We will keep this data for backward compatibility reasons, but it could be removed in a future SDK.
+
+For example:
+
+[source,json]
+----
+    "businessEntityId" : "groupOfLots",
+    "idScheme" : "GLO",
+    ...
+      "schemeName" : "LotsGroup",
+----
+
+The field `schemeName` is coming from the business entity, it should ideally be retrieved via the business entity of the field.
+We will keep the `schemeName` in the `fields.json` for backward compatibility.
+The same is true for the `idScheme`, `identifierFieldId` and `captionFieldId`.
+
+== Usage
+
+=== User interfaces
+
+This can be used in a UI to group or filter fields by their business entity.
+It can be more natural to group them by business entity than to group them by XML structure node.
+In that case the fields can be first sorted by business entity and second by alphabetical order.
+
+=== Detecting changes
+
+The process of detecting changes could be automated inside of applications.
+In order to populate the data of the field `BT-13716-notice` the following logic can be used: 
+
+* If the data of a field was changed
+* Read the `businessEntityId` of the field via `fields.json`
+* Get the business entity by business entity id via `fields.json`
+* Read the `identifyInChangeNotice` boolean
+
+After this there are several scenarios.
+
+* If `identifyInChangeNotice` is false, then do nothing.
+* If `identifyInChangeNotice` is true:
+** If `useInstanceIdentifier` is true, then the identifier of the modified instance must be found via the `instanceIdentifier`
+** If `useInstanceIdentifier` is false, then the `changeIdentifier` gives the identifier that changed
+
+=== XML generation and schemeName attribute
+
+In simple cases fields of type `id-ref` can only reference one type of identifier, in that case the scheme name is given.
+But in some cases for fields of type `id-ref` the user can select between multiple identifier types.
+For example `ORG` or `TPO`, this means the user will decide the `schemeName`, it cannot be determined in advance.
+
+Assuming the user selected `Touch point one (TPO-0001)`:
+
+* The application puts the correct id `TPO-0001` into the field
+* From the value `TPO-0001` the application can extract the `TPO` prefix 
+* In `fields.json` the application can find the business entity that has the prefix `TPO`
+* From the business entity the application can infer the `schemeName` which is `touchpoint`
+
+The `schemeName` is an XML attribute which must be set on some XML elements and the value must be `touchpoint` in this example.
+It is possible to achieve this without the business entities but the business entities make the link more obvious.

modules/fields/pages/index.adoc

@@ -65,7 +65,7 @@ We are using this file to provide the definition of all fields to our
 form-filling application (eNotices2). Each form is then made of a specific
 subset of those fields.
 
-The field repository is mainly an array of fields, preceded by some version information and the general structure of XML notices:
+The field repository is mainly an array of fields, preceded by some version information, the business entities and the general structure of XML notices:
 
 [source,json]
 ----
@@ -76,11 +76,15 @@ The field repository is mainly an array of fields, preceded by some version info
     "version": "0.2.53",
     "createdOn": "2021-11-20T16:46:39"
   },
-  "xmlStructure" : [ // <4>
+  "businessEntities" : [ // <4>
+    { ... },
+    ...
+  ]
+  "xmlStructure" : [ // <5>
     { ... },
     ...
   ],
-  "fields": [ // <5>
+  "fields": [ // <6>
     { ... },
     ...
   ]
@@ -89,8 +93,9 @@ The field repository is mainly an array of fields, preceded by some version info
 <1> Version of the UBL standard used.
 <2> Version of the eForms SDK the file belongs to.
 <3> Version number and date of the data used to create this file.
-<4> General structure of XML notices. See the xref:xml-structure.adoc[] page.
-<5> List of fields.
+<4> Logical structure defined by the business entities. See the xref:business-entities.adoc[] page.
+<5> General structure of XML notices. See the xref:xml-structure.adoc[] page.
+<6> List of fields.
 
 == Field properties
 
@@ -108,16 +113,19 @@ The various characteristics of each field are indicated as properties of the JSO
     "xpathRelative" : "cbc:RegulatoryDomain", // <6>
     "xsdSequenceOrder" : [ { "cbc:RegulatoryDomain" : 18 } ], // <7>
     "type" : "code", // <8>
-    "legalType" : "CODE", // <9>
-    "maxLength" : 400, // <10>
-    "repeatable" : { // <11>
-      "value" : false
+    "businessEntityId" : "procedure", // <9>
+    "legalType" : "CODE", // <10>
+    "maxLength" : 400, // <11>
+    "repeatable" : { // <12>
+      "value" : false,
+      "severity" : "ERROR"
     },
-    "codeList" : { // <12>
+    "codeList" : { // <13>
       "value" : {
-        "id" : "legal-basis",
+        "id" : "eforms-legal-basis",
         "type" : "flat"
-      }
+      },
+      "severity" : "ERROR"
     }
   }, {
     "id" : "BT-137-Lot",
@@ -128,13 +136,15 @@ The various characteristics of each field are indicated as properties of the JSO
     "xpathRelative" : "cbc:ID",
     "xsdSequenceOrder" : [ { "cbc:ID" : 2 } ],
     "type" : "id",
-    "idScheme" : "LOT", // <13>
+    "attributes" : [ "BT-137-Lot-Scheme" ],
+    "businessEntityId" : "lot",
+    "idScheme" : "LOT", // <14>
     "legalType" : "IDENTIFIER",
     "repeatable" : {
       "value" : false,
       "severity" : "ERROR"
     },
-    "forbidden" : { // <14>
+    "forbidden" : { // <15>
       "value" : false,
       "severity" : "ERROR",
       "constraints" : [ {
@@ -143,7 +153,7 @@ The various characteristics of each field are indicated as properties of the JSO
         "severity" : "ERROR"
       } ]
     },
-    "mandatory" : { // <15>
+    "mandatory" : { // <16>
       "value" : false,
       "severity" : "ERROR",
       "constraints" : [ {
@@ -152,7 +162,7 @@ The various characteristics of each field are indicated as properties of the JSO
         "severity" : "ERROR"
       } ]
     },
-    "pattern" : { // <16>
+    "pattern" : { // <17>
       "value" : "^LOT-\\d{4}$",
       "severity" : "ERROR"
     }
@@ -165,7 +175,10 @@ The various characteristics of each field are indicated as properties of the JSO
     "xpathRelative" : "efac:TenderLot/cbc:ID",
     "xsdSequenceOrder" : [ { "efac:TenderLot" : 14 }, { "cbc:ID" : 1 } ],
     "type" : "id-ref",
-    "idSchemes" : [ "LOT", "GLO" ], // <17>
+    "attributes" : [ "BT-13714-Tender-Scheme" ],
+    "businessEntityId" : "tender",
+    "idSchemes" : [ "GLO", "LOT" ], // <18>
+    "referencedBusinessEntityIds" : [ "groupOfLots", "lot" ], // <19>
     "legalType" : "IDENTIFIER",
     "repeatable" : {
       "value" : false,
@@ -192,17 +205,18 @@ The various characteristics of each field are indicated as properties of the JSO
 <6> Location of the field in an XML notice, relative to its parent node.
 <7> Position of each XML element relative to its siblings. <<xsdSequenceOrder,See below for details>>. 
 <8> Technical data type of the field.
-<9> Data type of the business term, as indicated in the eForms Regulation.
-<10> Maximum number of characters allowed in the value of the field, optional. 
-<11> Indicates if the field can appear more than once inside its container
-<12> Identifier of the code list from which the field value must belong.
-Applicable only for fields of type "code" or "internal-code"
-<13> Identifier scheme used by a field with `"type" : "id"`.
-<14> This property provides information on when a field's presence is forbidden.
-<15> This property provides information on when a field's value is mandatory.
-<16> A RegEx pattern the field's value must match.
-<17> Array indicating the valid `idScheme` values which this identifier reference field (`"type" : "id-ref"`) can reference.
-
+<9> The id of the business entity the field belongs to.
+<10> Data type of the business term, as indicated in the eForms Regulation.
+<11> Maximum number of characters allowed in the value of the field, optional. 
+<12> Indicates if the field can appear more than once inside its container
+<13> Identifier of the code list from which the field value must belong.
+Applicable only for fields of type "code".
+<14> Identifier scheme used by a field with `"type" : "id"`.
+<15> This property provides information on when a field's presence is forbidden.
+<16> This property provides information on when a field's value is mandatory.
+<17> A RegEx pattern the field's value must match.
+<18> Array indicating the valid `idScheme` values which this identifier reference field (`"type" : "id-ref"`) can reference.
+<19> Array indicating the valid `schemeName` values which this identifier reference field (`"type" : "id-ref"`) can reference.
 
 === Property Values
 
@@ -221,15 +235,17 @@ The properties that are assigned with static values are listed below:
 `xpathRelative`:: The XPath of the field relative to its parent `node`. See `parentNodeId`.
 [#xsdSequenceOrder]`xsdSequenceOrder`:: This indicates, for each XML element that is part of `xpathRelative`, the position that this element should have relative to its siblings in order to have an XML instance valid against the XML Schema. This information is extracted from the definition of the corresponding complex type in the XSD files.
 `type`:: The data type of the field. xref:#data-types[See Field data types].
-`attributeOf`:: If the field represents an XML attribute this indicates the identifier of the field representing the XML element which has the attribute.
 `attributeName`:: The name of the XML attribute represented by the field.
+`attributeOf`:: If the field represents an XML attribute this indicates the identifier of the field representing the XML element which has the attribute.
 `attributes`:: An array of one or more identifiers of the fields representing the XML attributes the XML element represented by the field can have.
 `presetValue`:: The value used to pre-fill a field. This property is always a string, but the value should be converted to the same type as the field. The value "\{NOW\}" corresponds to the current date or time.
-`legalType`:: The data type of the Business Term associated with the field as defined by the eForms Regulation.
-`maxLength`:: The maximum number of characters that the field can hold.
+`businessEntityId`:: The id of the business entity the field belongs to. See xref:fields:business-entities.adoc[Business Entities]. 
+`idSchemes`:: Applicable only to fields of type `id-ref`. Provides an array of "prefixedNumber" identifier schemes that this `id-ref` field can reference (this will be deprecated later, referencedBusinessEntityIds should be used as it has more complete data).
 `idScheme`:: Applicable only to fields of type `id`. Indicates the identifier scheme for this `id` field (e.g. "TPO" for Touch Point).
+`referencedBusinessEntityIds`:: Applicable only to fields of type `id-ref`. Provides an array of business entities that this `id-ref` field can reference.
 `schemeName`:: Applicable only to fields of type `id`. Indicates the value that should be indicated for this field in the `schemeName` attribute in the XML.
-`idSchemes`:: Applicable only to fields of type `id-ref`. Provides an array of valid identifier schemes that this `id-ref` field can reference.
+`legalType`:: The data type of the Business Term associated with the field as defined by the eForms Regulation.
+`maxLength`:: The maximum number of characters that the field can hold.
 
 Static properties are always assigned with a scalar value (a string, a boolean, a number, array of strings, etc.). If there is no value defined, the property is omitted.
 
@@ -417,7 +433,7 @@ In the following snippet, a dynamic value is assigned to the `forbidden` dynamic
   } ]
 },
 ----
-<1> The default value of the property will be false in this example
+<1> The default value of the property will be false in this example.
 <2> List of constraints for this property.
 <3> The first constraint in this example specifies a different value than the default one in the case that the field is used in one of the notice types indicated.
 <4> The value of this dynamic property in the case that the constraint applies is indicated here.
@@ -458,7 +474,7 @@ The value of the `condition` property of a constraint is a string representing a
 
 This expression is made of two parts:
 
-* The context under which the expression is evaluated. It's often the parent node of the field.
+* The context under which the expression is evaluated. It is often the parent node of the field.
 * The boolean expression itself.
 
 
@@ -468,8 +484,8 @@ This expression is made of two parts:
 The possible technical values for a field type are:
 
 [horizontal]
-`id`:: string representing an identifier (may have an associated `idScheme`)
-`id-ref`:: string representing a reference to an identifier (has an associated `idSchemes`)
+`id`:: string representing an identifier (may have an associated `idScheme` and `schemeName`)
+`id-ref`:: string representing a reference to an identifier (has an associated `idSchemes` and `referencedBusinessEntityIds`)
 `indicator`:: boolean (true or false)
 `integer`:: whole-valued positive number
 `number`:: numerical value, with optional decimal places.
@@ -484,4 +500,3 @@ The possible technical values for a field type are:
 `text`:: language-independent string
 `text-multilingual`:: string that can be translated into multiple languages
 
-

modules/fields/pages/xml-structure.adoc

@@ -2,7 +2,7 @@
 
 == Introduction
 
-The field metadata in the fields repository file (fields.json) describe the properties and behaviour of each field. Each field, representing an element in an XML document, is ultimately part of a greater tree-like structure. Actually, there is more than one such structure that each field is a part of:
+The field metadata in the fields repository file (fields.json) describes the properties and behaviour of each field. Each field, representing an element in an XML document, is ultimately part of a greater tree-like structure. Actually, there is more than one such structure that each field is a part of:
 
 * The physical structure in which each field is stored inside the notice XML file.
 This page documents how this physical structure is represented in the SDK.
@@ -15,7 +15,7 @@ This physical structure (XML structure) is included in the eForms SDK to allow i
 
 == XML structure in the field repository
 
-The XML structure is provided as a flat list of "nodes" at the end of the field repository file (fields.json). Every node represents an XML element. Each node can "contain" other nodes and/or fields. Every field provides a `parentNodeId` property which points to its parent node in the XML structure. Every node provides a `parentId` property which points to its parent node. Using this information you can reconstruct the entire XML tree. The absolute XPath of each node is also provided as well as the XPath relative to its parent node.
+The XML structure is provided as an array called "xmlStructure" near the top of field repository file (fields.json). Every item represents an XML element. Each item (a node) can "contain" other nodes and/or fields. Every field provides a `parentNodeId` property which points to its parent node in the XML structure. Every node provides a `parentId` property which points to its parent node (except the root). Using this information you can reconstruct the entire XML tree. The absolute XPath of each node is also provided as well as the XPath relative to its parent node.
 
 NOTE: The `xmlStructure` list provided in fields.json is a flat list. The logical hierarchy of the nodes is represented using the `parentId` property.
 
@@ -44,7 +44,8 @@ The following snippet shows the information provided for each node.
     "xpathRelative" : "cac:ProcurementProjectLot[cbc:ID/@schemeName='Lot']", // <4>
     "repeatable" : true, // <5>
     "identifierFieldId" : "BT-137-Lot", // <6>
-    "captionFieldId" : "BT-21-Lot" // <7>
+    "captionFieldId" : "BT-21-Lot", // <7>
+    "businessEntityId" : "lot" // <8>
   }
 ----
 <1> Identifier of the node.
@@ -54,6 +55,7 @@ The following snippet shows the information provided for each node.
 <5> Indicates if the node can be repeated inside its parent.
 <6> Indicates the field of type `id` that identifies specific instances of this node, only present if applicable.
 <7> Indicates the field that can be used to help a user recognise a specific instance of this node, like a name or title. Only present if applicable.
+<8> Indicates the business entity this node repeats with, present if repeatable.
 
 The absolute XPath of a node can be constructed from the XPaths of its parent and successive ancestors. We provide it for each node as a convenience.
 

modules/fields/partials/nav.adoc

@@ -1,2 +1,3 @@
 * xref:{page-component-version}@eforms:fields:index.adoc[Field metadata]
-** xref:{page-component-version}@eforms:fields:xml-structure.adoc[XML Structure]
+** xref:{page-component-version}@eforms:fields:xml-structure.adoc[XML Structure]
+** xref:{page-component-version}@eforms:fields:business-entities.adoc[Business Entities]