Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task] SQL generator: Handling Collections of Entities #622

Open
atextor opened this issue Jul 18, 2024 · 1 comment
Open

[Task] SQL generator: Handling Collections of Entities #622

atextor opened this issue Jul 18, 2024 · 1 comment
Labels
acknowledged Has been viewed by one of the maintainers and is ready for further work, discussion or other steps. task

Comments

@atextor
Copy link
Contributor

atextor commented Jul 18, 2024

Is your task related to a problem? Please describe.
Currently, for a Property that has a Collection/List/Set/SortedSet Characteristic, the Databricks SQL generator will create one column with an ARRAY<> datatype, with a STRUCT for the nested Entity type. This makes querying difficult and the approach is not applicable to other SQL dialects.
Instead, such collections of Entities should be flattened into columns for the Entity's type, with additional discriminator columns that allow retrieving the original dataset. Additionally,
for Aspects with one Property called items (a "collection Aspect"), flattening to tables should happen for the Entity which is the type of the collection, not for the Aspect itself. As an example, for AspectWithErrorCollection.ttl, the resulting table should have columns for errorNo, errorText and startTimestamp instead of an items column.

Describe the solution you'd like
Adjust the SQL Databricks generator to flatten collections of entities into columns. Handle a top-level items Property separately if necessary. Make names of additional discriminator columns configurable.

@atextor atextor added task acknowledged Has been viewed by one of the maintainers and is ready for further work, discussion or other steps. labels Jul 18, 2024
@atextor
Copy link
Contributor Author

atextor commented Nov 25, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
acknowledged Has been viewed by one of the maintainers and is ready for further work, discussion or other steps. task
Projects
None yet
Development

No branches or pull requests

1 participant