Support dataset identifier with schema name #18
Conversation
imtaos
force-pushed
the
custom-ingestion-schema
branch
from
f4b5158
to
5e16046
Compare
|
|
||
|
|
||
| @dataclass | ||
| class AffirmArtifact: | ||
| schema_name: str |
There was a problem hiding this comment.
Looking later in the PR I get that this is to display the path in the repo for the corresponding artifact. Are we going to do the same for the datasets?
There was a problem hiding this comment.
hmm mysql does not have schema concept so we don't have it.
after re-think it over, I feel it's better not to use schema to represent the nested directory structure of artifacts. e.g. logs/application_log.yml, over here we represent logs as database shema-like - and it isn't very much accurate. we can simply name it as logs.application_log
wdyt
There was a problem hiding this comment.
What is the purpose/value for the end-user we plan to get out of this? Can we just provide the file path in the datahub-metadata that corresponds to the dataset/artifact in DataHub? That may be useful to cross-compare, update, etc? I suppose it is not possible to have links in properties, but we could also consider adding a link to the GitHub file from elsewhere in DataHub.
There was a problem hiding this comment.
Btw, this is all going away as we will be moving it to the export-to-datahub script, correct?
There was a problem hiding this comment.
it wont' go away. it is part of the dataset urn. say urn:li:dataset:(urn:li:dataPlatform:affirmInfra,development.bluejays,PROD).
the difference is either consider the whole development.bluejays as the schema.table format, or it as development folder, bluejays table.
| def get_schema(dir: str): | ||
| relative_dir = dir.replace(os.path.abspath(directory), '').strip() | ||
| relative_dir = relative_dir if not relative_dir.startswith(os.sep) else relative_dir[len(os.sep):] | ||
| relative_dir = relative_dir if not relative_dir.endswith(os.sep) else relative_dir[len(os.sep):] | ||
| return '.'.join(relative_dir.split(os.sep)) |
There was a problem hiding this comment.
Can we expect a git directory and get the git root to get the relative path instead?
metadata-ingestion/examples/affirm_artifact/3rd-party.recipe.yml
Outdated
Show resolved
Hide resolved
Support
schema.tablein custom affirm artifacts ingestion source.e.g.
logsbelow is a schema and it can contain a few datasets inside.Checklist