Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

background tasks #61

Open
jesteria opened this issue May 3, 2021 · 2 comments
Open

background tasks #61

jesteria opened this issue May 3, 2021 · 2 comments
Labels
optimization Smaller/faster/better question Further information is requested

Comments

@jesteria
Copy link
Collaborator

jesteria commented May 3, 2021

The pipeline might be sped up by performing in the background sub-tasks which are necessary for completion but not for the pipeline itself.

For example, the user may have requested that nPrint data and/or features are saved to disk; however, the pipeline needn't necessarily wait for these to be serialized and written to disk before continuing to the next step in the pipeline.

Instead, such work might be performed by a background thread/process (off of a standard queue), while the pipeline proceeds.

@jesteria jesteria added question Further information is requested optimization Smaller/faster/better labels May 3, 2021
@jesteria
Copy link
Collaborator Author

jesteria commented May 3, 2021

This should be considered and tested to ensure we're able to measurably improve performance this way and to ensure that whatever we do doesn't interfere with the performance of the pipeline itself.

@JordanHolland
Copy link
Collaborator

Agreed on the testing -- this seems significantly less important given the results in #55, though as the files grow in size, it could become more important to consider this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimization Smaller/faster/better question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants