Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat background task nf #765

Merged
merged 37 commits into from
Aug 9, 2024
Merged

Feat background task nf #765

merged 37 commits into from
Aug 9, 2024

Conversation

nazarfil
Copy link
Contributor

@nazarfil nazarfil commented Jul 9, 2024 •
edited
Loading

Creates dataset snapshot model and the functional flow to generate snapshot object in the task
Exposes snapshot object through GraphQL api

@nazarfil nazarfil force-pushed the feat-background_task-nf branch 6 times, most recently from 14857c0 to 41b9015 Compare July 10, 2024 14:04
@nazarfil nazarfil requested a review from qgerome July 11, 2024 13:49
@nazarfil nazarfil marked this pull request as ready for review July 11, 2024 13:49
@nazarfil nazarfil marked this pull request as draft July 11, 2024 13:50
@nazarfil
Copy link
Contributor Author

wip: adding tests

@nazarfil nazarfil force-pushed the feat-background_task-nf branch from e15438d to 2f8e666 Compare July 12, 2024 08:18
@nazarfil nazarfil force-pushed the feat-background_q_dataset-nf branch from 6950da9 to 3770808 Compare July 12, 2024 08:18
@nazarfil nazarfil marked this pull request as ready for review July 12, 2024 08:30
@nazarfil nazarfil marked this pull request as draft July 12, 2024 08:39
@nazarfil nazarfil force-pushed the feat-background_q_dataset-nf branch from 3770808 to eb337b6 Compare July 12, 2024 08:53
@nazarfil nazarfil force-pushed the feat-background_task-nf branch 3 times, most recently from f5a9e31 to da4c070 Compare July 12, 2024 09:05
@nazarfil nazarfil marked this pull request as ready for review July 12, 2024 09:05
@nazarfil nazarfil force-pushed the feat-background_task-nf branch 2 times, most recently from d8bda21 to 4d51db1 Compare July 15, 2024 08:09
qgerome
qgerome reviewed Jul 15, 2024
View reviewed changes
Copy link
Member

@qgerome qgerome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add the metadata object directly on the DatasetVersionFile in the graphql schema

hexa/datasets/graphql/schema.graphql Outdated Show resolved Hide resolved
hexa/datasets/graphql/schema.graphql Outdated Show resolved Hide resolved
hexa/datasets/graphql/schema.graphql Outdated Show resolved Hide resolved
hexa/datasets/models.py Outdated Show resolved Hide resolved
hexa/datasets/queue.py Outdated Show resolved Hide resolved
hexa/datasets/queue.py Outdated Show resolved Hide resolved
hexa/datasets/queue.py Outdated Show resolved Hide resolved
hexa/datasets/queue.py Outdated Show resolved Hide resolved
hexa/datasets/queue.py Outdated Show resolved Hide resolved
hexa/datasets/schema/queries.py Outdated Show resolved Hide resolved
qgerome
qgerome requested changes Jul 16, 2024
View reviewed changes
hexa/datasets/queue.py Outdated Show resolved Hide resolved
hexa/datasets/schema/queries.py Outdated Show resolved Hide resolved
@nazarfil nazarfil requested a review from qgerome July 16, 2024 14:42
@nazarfil nazarfil force-pushed the feat-background_task-nf branch 2 times, most recently from ed3061a to e2c1e5d Compare July 17, 2024 07:30
@nazarfil nazarfil force-pushed the feat-background_q_dataset-nf branch from 898c651 to 400056b Compare July 17, 2024 07:33
@nazarfil nazarfil force-pushed the feat-background_task-nf branch 3 times, most recently from 1fa774a to ecf23dd Compare July 18, 2024 07:03
nazarfil added 19 commits August 7, 2024 12:57
@nazarfil
test(Dataset): attempt to test the version file
31ed9af
@nazarfil
chore: rename snapshot to file_metadata
a5951dc
@nazarfil
refactor: removed not needed file metadata query
af85aad
@nazarfil
test(Dataset): add unittest for the task flow
9526eaa
@nazarfil
fix(Dataset): fixed parquet file reading
06b427c
@nazarfil
fix(Dataset): added corner cases for failures
0901809
@nazarfil
fix(Dataset): move from head to sample
5817d6f
@nazarfil
fix(Dataset): tests handled exceptions
d03d879
@nazarfil
fix(Dataset): tests file with no columns and no rows
e2f1ad7
@nazarfil
fix(Dataset): fixes grpahql exposed api
ab38d4d
@nazarfil
test(Dataset): add unittest for xlsx file
a9be661
@nazarfil
chore: rename worker to dataset-worker
aa1b240
@nazarfil
chore: ignore local files
9aaef85
@nazarfil
chore: document how to run dataset worker locally
fe817e8
@nazarfil
fix(Dataset): fixes grpahql exposed metadata and exceptions
95f6b49
@nazarfil
fix(Dataset): fix typo, add internationalisation
57e05c3
@nazarfil
fix(Dataset): adds property to fethc latest metadata for a file
a7abfdc
@nazarfil
fix(Dataset): fixes typo and dict argument
6f7b71c
@nazarfil
fix: renames sample in metadata
be80552
@nazarfil nazarfil force-pushed the feat-background_task-nf branch 2 times, most recently from 6b0e3f9 to ecd80c5 Compare August 7, 2024 11:09
@nazarfil nazarfil force-pushed the feat-background_task-nf branch from ecd80c5 to a347e39 Compare August 7, 2024 12:37
@nazarfil nazarfil force-pushed the feat-background_task-nf branch from b449ff1 to 0f7fd7c Compare August 8, 2024 09:54
@qgerome qgerome merged commit 8838e15 into main Aug 9, 2024
4 checks passed
@qgerome qgerome deleted the feat-background_task-nf branch August 9, 2024 12:26
@qgerome qgerome mentioned this pull request Aug 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qgerome qgerome qgerome approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nazarfil @qgerome