Position of the fields in subType.xml

[rengit62]

Hi all,
in 4.xml ( for example ) there a list of fields in Metadata section and from each id of them one can get its definition in fields.json.
OK, but I do know see the 'position' of each field in the final xml.
For example : field "BT-02-notice" is the first in 4.xml , but it must put in xml , after , for example, of "BT-701-notice" , otherwise xsd validation fails.
Are you going to set this information in the future, in eForms-SDK , or one has to read xsd before reading 4.xml ?
Thanks a lot
Renato

[rousso]

Hi Renato,

There is no missing information. Here is how to get a grasp of it:

• The notice-type-definition files contain a visual representation of the notice.
• The XML file that you want to create will contain a physical representation of the same notice.
• In order to go from the visual representation to the physical representation you need an intermediate step.
  You need a conceptual representation.

Let me explain.

• The visual representation (or visual model) contains stuff that is displayed on a screen (and the values the user has entered).
• The conceptual representation (or conceptual model) should contain only nodes and fields (see fields.json).
  First, you need to traverse the visual model to create a conceptual model. All the information that is needed is already in the visual model as every repeatable group is associated with a node (and every field on the screen with a field in fields.json of course).
• You can then traverse the conceptual model to create the physical model (or physical representation) that should contain only XML elements. When you have the physical representation it is easy to serialize it to an XML file.

So think of it like this: visual -> conceptual -> physical
If you break the problem into these concrete steps you will see that it is much easier to figure it out.

Let me know if this helps.

Regards,
Yannis

c: @rouschr @bertrand-lorentz

[bertrand-lorentz]

To complement the answer from @rousso :
The XSD files do indeed impose a specific order between the various XML elements, which must be respected when you create an XML file:
Under the document root, the element "cbc:NoticeTypeCode" (BT-02-notice) must be located after the element "cbc:ID" (BT-701-notice)

As this information is defined in a standard way in the XSD files, we currently have no plans to provide it in a different way.

[rengit62]

First if all , thanks for all you replies, now I would like to know more about the step conceptual -> physical
Ok , from fields.json I now all informations correspondent to visual stuff known as ( for example ) BT-5101(a)-Part

"id" : "BT-5101(a)-Part",
"parentNodeId" : "ND-PartPlacePerformance",
"name" : "Place Performance Street",
"btId" : "BT-5101",
"xpathAbsolute" : "/*/cac:ProcurementProjectLot[cbc:ID/@schemeName='Part']/cac:ProcurementProject/cac:RealizedLocation/cac:Address/cbc:StreetName",
"xpathRelative" : "cac:Address/cbc:StreetName",
..................

Now how can I create the corrispondent xml tag in the final xml ?
Should I get 3 general xml correspondent to the 3 xsd (UBL-PriorInformationNotice-2.3.xsd, UBL-ContractAwardNotice-2.3.xsd, ... ) and then apply xPath expression ? ( for the first 40 notices)

And another question, can you give me any suggestion about to create html forms from notice-type-definition files ( and 'detail' in fields,json ? ) ?

Thanks
Renato

[rousso]

Hi again Renato,

First I will address a couple of points on the specific things you ask, and then I will give you more details on how to go about creating the XML file.

Now how can I create the corrispondent xml tag in the final xml ?

See my explanation of the process later in this answer.

Should I get 3 general xml correspondent to the 3 xsd (UBL-PriorInformationNotice-2.3.xsd, UBL-ContractAwardNotice-2.3.xsd, ... ) and then apply xPath expression ? ( for the first 40 notices)

If I understand well your question, the idea you are considering is to start from the XSD to generate a "generic" XML file inside which you can later inject the notice data collected from the user (and remove any unused XML elements). The idea is valid. A similar idea was our first attempt at this (before we started creating the SDK) and I think that this is more or less the approach that our eNotices2 application still uses. I think it should work, although you will still need to juggle the transposition of the data from the visual model which you already have on your user's screen, to the physical model that you create from the XSD. In my head it is simpler and safer to go through the visual -> conceptual -> physical model transformation because it is a rules based approach that uses an intermediate representation and will require far fewer mind bending if-then-elses. In my mind the approach we try to take in the Notice Editor Sample Application will be easier to implement and to maintain. It's your choice.

And another question, can you give me any suggestion about to create html forms from notice-type-definition files ( and 'detail' in fields,json ? ) ?

See the implementation we have so far for this in the Notice Editor Sample Application.

Going back to the topic of visual -> conceptual -> physical model transformation.
This is still a concept for us too. I will hook you up with @rouschr who just started working on this so that he may give you more information on which branch we will be working on so that you can follow his commits.

Quick clarification before I continue: I use the term "model" (visual, conceptual, physical) but I really refer to a "representation" (an "instance" of a model). It will be an in-memory object at the end of the day that follows some rules for its structure. I hope this makes sense.

Let's take them one by one:

Visual model:

This should be a collection of objects that are either display-groups or input-fields. Notice that I am making a distinction between input-fields and fields to make sure we all understand that a field is a conceptual model element whereas an input-field is a visual model element.

The elements you need for the visual model are all inside the notice-type-definition file (inside the notice-types folder of the SDK). They are provided in a hierarchical structure and you will use them to create a form for the user to fill-in the notice. With the term form I mean a visual layout of sections, groups and user-controls with which the user of your application will interact to enter notice data. For every element you put on the screen, you need to maintain some reference to the corresponding element in your visual model.

When the user submits the form, you will have to traverse your visual model and for each element that the user has provided data you need to create a corresponding element in the conceptual model.

Conceptual model:

The conceptual model is a collection of objects that are either nodes or fields. The elements that you can use in the conceptual model are all provided in fields.json. There will be no display-groups or input-fields in the conceptual model. The reason that you are creating the conceptual model is because you need to "escape" from the visual structure and move closer to the physical one while still being able to have a one-to-one correspondence in the concepts our are working with.

While you are creating your conceptual model you will be actually traversing your visual model element by element, adding to the conceptual model a corresponding element for each element in your visual model.

• For each display-group that is associated with a node you must add that corresponding node to the conceptual model.
• For each input-field in the visual model, you must add a field in your conceptual model.
• Every time you add a node in the conceptual model you need to ensure that all its parent nodes are also added.
  The same for each field you add.
• You will pay attention to reuse the nodes that have already been created.
• You will also pay attention to create multiple nodes as needed for repeatable display-groups.
  The same for repeatable input-fields.

In short, the structure of the elements in your conceptual model will follow the structure provided in fields.json (but, of course, will only contain the nodes and fields for which the user has entered data in your form).

At the end of this process all the data that the user has entered on the screen will be in a conceptual model structure that is much closer to the final physical model structure. The next thing you need to do is derive the physical model from the conceptual one, which should be much easier now that you have "escaped" the visual structure.

Physical model:

The physical model needs to correspond to the actual XML structure. The physical model therefore can only contain xml-elements.

To create the physical model, you will traverse your conceptual model. Each node in the conceptual model has an associated relative-xpath. This relative XPath will indicate one or more XML elements that need to be added to the physical model. Therefore, as you traverse your conceptual model:

• For each node you need to create all necessary xml-elements
• For each field you will also need to create the corresponding xml-elements
• If a field points to an XML attribute you will have to also properly indicate that in your physical model
• You will pay attention to reuse xml-elements that have already been added to the physical model as needed.
  Your guide for this is the structure that you have already created in the conceptual model.
• You will use the XSD (which formally defines each xml-element) to properly order the elements in your physical model as needed.

At the end of this process all the data entered by the user will be in a structure that has a one-to-one correspondence with the structure of the XML file that you want to generate. Your final step is to traverse this physical model that you just generated and write the XML output file element by element.

I hope this helps to clarify the idea a bit further.

Maybe an idea that combines your suggestion to "start with the XSD" with the idea of transforming from a visual to a conceptual model first, is an even better approach that keeps the best of both worlds. In this case you would still do the work of transforming from the visual to the conceptual model, but, instead of generating the XML element by element, you can create it from the XSD and then fill it in from the conceptual model where the correspondence between nodes, fields and xml-elements would be easier to find. As a final step ou would remove all the empty elements from your XML.

Regards
Yannis

cc: @bertrand-lorentz, @rouschr

[JasperBeaurain]

Thanks for this explanation Yannis! I think this is info a lot of us were looking for.

We have come up with the same approach in the meantime. However, I am still facing some issues with it.

There seem to be elements in the SDK where you cannot escape from the visual model like you describe. Let me give you an example. (this relates to the thread I started: #132 )

In the visual model you have the following structure (simplified):

{
    "id": "GR-Procedure-SProvider",
    "nodeId": "ND-ServiceProviderParty",
    "repeatable": true,
    "content": [
        {
            "id": "OPT-300-Procedure-SProvider",
            "repeatable": true,
            "parentNodeId": "ND-ServiceProvider"
        },
        {
            "id": "OPT-030-Procedure-SProvider",
            "repeatable": false,
            "parentNodeId": "ND-ServiceProviderParty"
        }
    ]
}

Note that OPT-300-Procedure-SProvider has node ND-ServiceProvider as a parent and not ND-ServiceProviderParty!

In my form, I happen to repeat GR-Procedure-SProvider twice. In one of those groups, I also repeat OPT-300-Procedure-SProvider.

In JSON this would look like this:

{
    "id": "GR-Procedure-SProvider",
    "nodeId": "ND-ServiceProviderParty",
    "repeatable": true,
    "content": [
        {
            "id": "OPT-300-Procedure-SProvider",
            "repeatable": true,
            "parentNodeId": "ND-ServiceProvider"
        },
        {
            "id": "OPT-300-Procedure-SProvider",
            "repeatable": true,
            "parentNodeId": "ND-ServiceProvider"
        },
        {
            "id": "OPT-030-Procedure-SProvider",
            "repeatable": false,
            "parentNodeId": "ND-ServiceProviderParty"
        }
    ]
},
{
    "id": "GR-Procedure-SProvider",
    "nodeId": "ND-ServiceProviderParty",
    "repeatable": true,
    "content": [
        {
            "id": "OPT-300-Procedure-SProvider",
            "repeatable": true,
            "parentNodeId": "ND-ServiceProvider"
        },
        {
            "id": "OPT-030-Procedure-SProvider",
            "repeatable": false,
            "parentNodeId": "ND-ServiceProviderParty"
        }
    ]
}

Now when I convert this to my conceptual model, I end up with this:

ND-ServiceProvider[0]    -   cac:Party
    OPT-300-Procedure-SProvider[2]    -   cac:ServiceProviderParty/cac:Party/cac:PartyIdentification/cbc:ID
    OPT-300-Procedure-SProvider[1]    -   cac:ServiceProviderParty/cac:Party/cac:PartyIdentification/cbc:ID
    ND-ServiceProviderParty[1]    -   cac:ServiceProviderParty
        OPT-030-Procedure-SProvider[1]    -   cbc:ServiceTypeCode
    OPT-300-Procedure-SProvider[0]    -   cac:ServiceProviderParty/cac:Party/cac:PartyIdentification/cbc:ID
    ND-ServiceProviderParty[0]    -   cac:ServiceProviderParty
        OPT-030-Procedure-SProvider[0]    -   cbc:ServiceTypeCode

I also displayed the indices of the nodes (as they were repeated) and the relative xpaths of these nodes/fields.
You can now see that OPT-300-Procedure-SProvider is below ND-ServiceProvider in the conceptual model, while in XML it belongs below ND-ServiceProviderParty (cac:ServiceProviderParty).

An example of the expected xml is shown below (from cn_24_multilingual.xml).

<cac:Party>
    <cac:PartyIdentification>
        <cbc:ID schemeName="organization">ORG-0001</cbc:ID>
    </cac:PartyIdentification>
    <cac:ServiceProviderParty>
        <cbc:ServiceTypeCode listName="organisation-role">ted-esen</cbc:ServiceTypeCode>
        <cac:Party>
            <cac:PartyIdentification>
                <cbc:ID schemeName="organization">ORG-0004</cbc:ID>
            </cac:PartyIdentification>
        </cac:Party>
    </cac:ServiceProviderParty>
</cac:Party>

In the conceptual model, the information about having to put OPT-300 inside the cac:ServiceProvderParty created by the group GR-Procedure-SProvider is lost.
In my implementation I would either always create new cac:ServiceProvderParty elements (you end up with 5x cac:ServiceProviderParty, not what we want) or try to merge them (you end up combining too much).

Do you agree that this is an issue? Or am I missing something?

In my opinion, this would have been way easier if OPT-300-Procedure-SProvider had a parentNodeId of ND-ServiceProviderParty and a corresponding xpath ( cac:ServiceProviderParty/ cac:Party/cac:PartyIdentification/cbc:ID).
Similar field/group/node combinations exist in the SDK. (This is what I asked about in #132 )

Thanks for your help!
Jasper

[rousso]

Hi @Jasper9041,

Having looked at this carefully with our team yesterday, the conclusion is that you are right about the specific fields and you correctly point out that the conceptual model looks wrong. Indeed it contains some systematic mistakes in this specific case as well as in some other similar ones.

We identified the cause of the issue in our internal metadata store and we will fix this in the coming days. The corrected metadata will be released in version 1.3.0 of the SDK that will come in 2 or 3 weeks (28 October at the latest).

For a bit more transparency, I will give you some background information. We encode and manage eForms metadata internally in a relational database. From there we generate all the files we publish in the SDK, Like everything else in the world, our database and encoding are not perfect either. Certainly not perfect yet!

So, the issue you reported (as well as some other issues that were recently puzzling us related with repeatable fields and nodes) helped us realize and pinpoint the cause of the problem which was actually a general assumption about the context of execution of all business rules. It turns out that the assumption with witch our encoding and export algorithm were operating until today, overlooked this case where the conceptual structure of the notice does not coincide with the logical structure necessary for schematron validation.

Luckily of course, this problem is easy to fix in our metadata database. The whole purpose of encoding, managing and publishing the metadata in this way was to be able to fix issues quickly and transparently for everyone else.

The result is that there is nothing you need to do from your side. You can keep working with the same assumptions that we discussed about the transformation from visual to conceptual to physical model. The metadata in the next release of the SDK (1.3.0) will be fixed and your algorithm will not encounter this issue again.

Thank you for your help and feedback.

Regards,
Yannis

cc: @rouschr, @rengit62, @bertrand-lorentz, @YvesJo

[JasperBeaurain]

Hi @rousso

Thanks for your quick response. I'm very happy to hear that the source of the issue has been found!

Looking forward to the 1.3.0 release!

Best regards,
Jasper

[JasperBeaurain]

@rengit62 FYI for converting the conceptual model to XML (building the xml nodes one by one by using/splitting the XPath):
There is a xpath ANTLR grammar and example code in eforms-toolkit-java. You can use it to parse the xpath correctly.
You could try doing it with regex or a rudimentary DIY parser too, but that is not as easy to get 100% correct in our experience.
If you are using Java, this might save you some time 😄

[rengit62]

@Jasper9041 of course is useful ! Thank you Jasper ! I made 3 xml from the main 3 xsd but they are really huge !
by the way , did you have problems starting the eforms-notice-editor ? It searches eform-sdk snapshot from an host unreacheble from my workstation :
Could not transfer artifact eu.europa.ted.eforms:eforms-sdk:jar:0.7.0-SNAPSHOT from/to central (http://midevtl01:8080/artifactory/esop-release): midevtl01
I you did not .. I will watch my network configuration, ( proxies, firewall ... )
Ciao !
Renato

[JasperBeaurain]

@rengit62 This looks like an issue with your maven settings.xml pointing to an internal repo. I would search in that direction

[rousso]

@rengit62 eForms SDK 0.7.0-SNAPSHOT is no longer referenced anywhere from what I can see in our source code. You probably have an outdated version of the sources. Make sure you have the latest develop version of the notice editor and dependent projects. When you work with develop versions always make sure you "pull" the latest commits from git.

In general the -SNAPSHOT suffix means that the dependency is on a "develop" version (the source of which is on the "develop" branch of the project). Which in turn means that to resolve such dependency you need get the sources from the develop branch and build locally so that the dependency can be resolved from your local maven repository.

Let me know if that helped.
Yannis

[rengit62]

Hi Jannis, nice idea the conceptual model , I did it and began making the xml checking the xsd files.
Just a question, when one meets an xpath like //ext:UBLExtensions/ext:UBLExtension/ext:ExtensionContent/efext:EformsExtension ( id :ND-RootExtension) it is easy : splitting by "/" to make the tag.
Less clear are the xPath Predicate, beyond using ELX or regex, what do you mean for example by :
//cac:ProcurementProjectLot[cbc:ID/@schemeName='LotsGroup'] ( ND-LotsGroup ) ?
as Xpath expression, it should be an xml tag cac:ProcurementProjectLot with the child cbc:ID that has got the attribute schemeName='LotsGroup', and son of ND-LotsGroup, the field BT-137-LotsGroup which add an other tag cbc:ID ?
Watching cn_24_cumbria.xml, cn_24_maximal, ... it is not true, can you explain ?
Thanks
Renato

[rengit62]

Ok Jannis, perhaps I understood, BT-137-LotsGroup has got the value of the tag cbc:ID
Thanks !
Renato

[rengit62]

Right ! Thanks Jasper !