Support seperate presentation model & data model

[TheTechArch]

Description

To be able to support an optimal UI, we might end up in a suboptimal data model.
Not all data models can be mapped to a sensible view model or UI setup for the front end.

We then end up with a UI that is not 100%, or we need to change the data model in the app to something else defined for the core system.

It is requested that we support having one data model used for UI support, and we convert the data to a more data-centric model.

An example of a data model where this was done to support UI is Altinn/app-migration#1

Considerations

• Hide data object for end-user
• Where and when should this conversion happen. For every save or when a data task is completed? Both?
• Do we risk more errors since mapping can cause errors
• Manuell mapping we need to support, but do we need to support automatic mapping for simple cases? The assumption is no; when the case is simply, the need for two models is probably low.

Ops requirements

Are there any requirements related to backup? -->

Acceptance criteria

• Support separate data model used for supporting UI and downloading to ORG
• Models are in sync

Specification tasks

• Development tasks are defined

Development tasks

Test

Definition of done

• Documentation (docs.altinn.studio) is updated (if relevant)
  • Technical documentation
  • User documentation
• QA
• Manual test is complete (if relevant)
• Automated test is implemented (if relevant)
• All tasks in this userstory are closed (i.e. remaining tasks are moved to other user stories or marked obsolete)

[altinnadmin]

@TheTechArch This issue needs to be cleaned up. I'm not able to understand what you mean.

[TheTechArch]

@altinnadmin some of my text was hidden because of comment text. Available now. More clear?

[altinnadmin]

@TheTechArch This sounds like something we should try to avoid, sounds like a lot of complexity and extra work for all parties.
It's not clear to me what functionality in Altinn/app-migration#1 is triggering this?

[ivarne]

A somewhat less confusing solution (for those that don't need separate models), would be to have a separate model for the fields that we need to store while filling out the form, but that we don't want to leak into the backend systems. As I understand it, Altinn2 has this with the @my prefix.

I would assume this could be implemented in a backwards compatible way too, with the two models being merged in frontend, but (de)serialize the frontend model separately at the backend.

[nkylstad]

@TheTechArch This might be solved (at least partially) by something like Altinn/app-frontend-react#744

[FinnurO]

The @my prefix is feature complete in A2. There is an issue on shadow fields. @olemartinorg

Should look into if DTO (Fowler) could be used ad a method for the presentation model

[olemartinorg]

Reads to me like this and shadow fields tries to solve the exact same problem (i.e., the "presentation model" == "shadow fields"). I would assume backend needs/wants to access this (generated?) presentation model, and possibly prefill data into it - but it should be deleted when the instance/process ends.

An example here is filters causing updates to option lists. Say you create a dropdown, and the choice you make there affects which options you see in a later dropdown. It might make sense that you don't want to store the first choice, but only store the second in the data model that is sent along when the instance ends.

App-frontend currently has no support for working with multiple data models in the same data task, but that's been a wish (especially during the work on on the new PDF generator). It might make sense to refactor app-frontend to support multiple data models at once, and having one of them be some auto-generated model for "presentation" (aka "shadow fields").

Another issue is the List component, which currently has internal state in redux for presentation (i.e. the current sort order, current page, and number of items to show on a page). If you reload the page, the current position in the list is lost (it was only stored client-side), so the user might get disoriented. Having a place to store such things would let us solve that problem as well - but it certainly does not make sense to store these things in the data model for that component.

[ivarne]

@olemartinorg Another related issue is what to do when the user enters data into a field that is not representable in the backend data type. Common examples of that is decimal values that contain letters, have more than 28 digits, or DateTime that contains dates in a format that is not recognized. These should be preserved (maybe not across reload), so that the user can see the error and fix the issue.

[FinnurO]

Tagger @TheTechArch (og kanskje @allinox ) som kan si noe mer om de konkrete hindringene de traff på som gir begrensninger i brukeropplevelsen p.g.a. datamodellen. Jeg ser f.eks at Alexandra ber om nye felt som kanskje kunne vært løst av hjelpefelt.

Kan behovet komme fra:

• I en tabell så har jeg lyst til å vise navnet som en konkatenering av fornavn+mellomnavn+etternavn, men at detaljene viser hvert felt for seg? Altså at man presenterer data på en annen måte enn datamodellen? Det kan kanskje løses i dag, men da må man spesifisere det hver gang fremfor å referere til en variabel fullt_navn?
• Er det bedre/enklere å forholde seg til en array i f.eks grid selvom den bakenforliggende datamodellen består av enkeltfelt, f.eks kan det være enklere å summere en kolonne i en array fremfor å spesifisere summen av enkeltfelt?
• Samme med checkbox grupper som kan gå mot enkeltfelt i datamodellen (Analsysis: Should we allow (in a standardised way) checkbox support for individual data model fields for each alternative app-frontend-react#789)

Tror det er viktig å finne ut av levetiden til disse feltene; leve i mens instansen lever og slettes ved innsending, være en egen datamodell eller delmodell i hovedmodellen. Spørsmålet er vel om behovet utelukkende er for GUI tjenester eller om API tjenester også vil ha bruk for de?

Noen relevante(?) issues: Altinn/altinn-studio#725

Altinn/app-frontend-react#273

Altinn/app-frontend-react#789

[altinnadmin]

It is requested that we support having one data model uses for UI support and we convert the data to a more data-centric model.

@TheTechArch After adding support for shadow fields and dynamic expressions, and a ton of improvements in various UI components, do you still think this issue is relevant and should be implemented as described?

[allinox]

@FinnurO
The cases that I have come across are usually related to managing dynamic in the app.

For example, in the case that we want to display an error (as a panel with info and as a validation error) based on some feedback from an API, so we need one or more fields to work as flags or states for the error. This field isn't relevant for anything but the logic of the app itself, but we still need to add it to the data model. In some cases, you can add a specific section to the data model (e.g. under the name AppLogikk) for the fields that are needed to make the app function but no further. However, in other cases, you might need the flag to be related to a specific index in a repeating group and thus, the error-field needs to be located inside the list of objects.

Another case (an app made for DiHe) is an app that fetches all the users children form an API and presents them as options in a check-box component for the user to choose from. This app could be an application form where the user can apply for one or several of their children and will have to check which children are relevant in their case.
The chosen children are then transferred to a repeating group where more information should be provided by the user regarding each chosen child.
In this case, we still need to store all children in some data model field and this field will be forwarded to the case manager (unless explicitely removed in custom code by the end of the event) even if only the chosen children (and their information) is actually of relevance to the case.

Other cases might be related to sharing data between the specific handlers, such as storing prefill data during instansiation that may or may not be relevant for an application later depending on what flow the user takes, or validating emails in a text area and informing the users which emails are faulty (something that will require a check in DataProcessingHandler, storing the faulty emails in a field for use in the error message, and checking if this field is non-empty in ValidationHandler.)

There might be even more cases, but these are the ones that come to me right now 🙂

[olemartinorg]

Another case (an app made for DiHe) is an app that fetches all the users children form an API and presents them as options in a check-box component for the user to choose from. This app could be an application form where the user can apply for one or several of their children and will have to check which children are relevant in their case. The chosen children are then transferred to a repeating group where more information should be provided by the user regarding each chosen child. In this case, we still need to store all children in some data model field and this field will be forwarded to the case manager (unless explicitely removed in custom code by the end of the event) even if only the chosen children (and their information) is actually of relevance to the case.

Sounds a lot like the use-case for improved filter functionality in repeating groups, where you could fill all rows in the repeating group by default, and hide the ones not checked in the previous checkboxes. The advantage of doing it that way is you can check an item, fill out some custom data, and then uncheck that item again - hiding it from the repeating group without deleting the custom data.

[TheTechArch]

@altinnadmin @olemartinorg I dont belive this is the same as presentation fields. For reelle rettighetshavere the internal model is something like this

{
  "Person": {
    "Positions": [
      {
        "PositionType": "A",
        "PositionReason": [
          {
            "ResonType": "Owner",
            "Category": "A"
          },
          {
            "ResonType": "Fan",
            "Category": "B"
          }
        ]
      }
    ]
  }
}

The gui that was designed wanted to expand all possible options for each level in the tree. Differnt positions tpes, Poisition Reasons and so on. This was not possible to do in GUI because that would force to populate all data in datamodel and then remove stuff that was not selected. So the person object got fields for every possible options and BRG had to change the model

[olemartinorg]

@TheTechArch I took the liberty of formatting your JSON example to help me make sense of this thing. I can't say I understand everything here - from what I can tell this looks like a nested repeating group setup in the data model (there are two levels of arrays, each with objects in them, which is the data model expected by a nested repeating group setup). However, it's not clear to me what you mean by expand all possible options for each level in the tree in the GUI. Did nested repeating groups not work for you for some reason, and how does this suggestion solve it?

The way I read this issue (or, the original description of it), it is still unclear for me how we should solve it. Most issues are described in one of two ways (either I have a problem, and I want you to solve in some way of your choosing or I want you to implement this exact solution, but I'm not going to tell you why I want it). This one is falling short somewhere in-between for me - I have a limited understanding of the root problem and I struggle to figure out how you wanted this to be solved.

I have a feeling that we already have plans for solving the underlying problem here, but in a different way. As I said, this looks like a data model structure for nested repeating groups, and from this part of your comment:

because that would force to populate all data in datamodel and then remove stuff that was not selected

How about:

1. If you wanted to make rows of a repeating group (or nested repeating group) invisible when (for some reason) not relevant, via dynamic expressions - that is the use-case for the improved filter functionality I linked above.
2. If you wanted to construct some options (for displaying in a dropdown, checkboxes, etc), but not pollute the data model with it - maybe you wanted dynamic options? Even dynamic options based on some selection? Or, possibly even options based on repeating groups from the data model?
3. If dropdown or checkboxes are too simple for this use-case, maybe the new List component is what you wanted?

[nkylstad]

@RonnyB71 transferring this issue to app-lib repo.