Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit Testing for DataLoaders #23

Open
hjabbot opened this issue Feb 5, 2024 · 1 comment
Open

Unit Testing for DataLoaders #23

hjabbot opened this issue Feb 5, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@hjabbot
Copy link
Collaborator

hjabbot commented Feb 5, 2024

Moving here from project repo #151

Watch-dog scripts to check data is collected correctly and is in the correct format. Gives a warning if unexpected changes


Move this issue to mesh builder repo when it is split out after 0.2.x


The GRF and CSV dataloaders can get this implemented no worries, and realistically they should be done ASAP as our regression tests are dependant on the GRF dataloaders working consistently.

Need to figure out issues with EULA's of the data products we're ingesting first. Perhaps we could create fake datasets and attach real header information just to see if the functionality still works.

@hjabbot hjabbot added the enhancement New feature or request label Feb 5, 2024
@hjabbot hjabbot self-assigned this Feb 6, 2024
@hjabbot
Copy link
Collaborator Author

hjabbot commented Jun 11, 2024

The working plan is to create a suite of unit tests for the abstract dataloaders, and then it is up to the end user to make sure that each individual dataloader is correctly formatting the data to be handled by the abstract dataloaders.

A separate suite of unit tests can be saved in a private repo that will check the header info of each data source to flag any changes that occur. I don't think that anything beyond the header info is worth testing, as data changes and so there is no 'ground truth' that the data itself can be compared against.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant