Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache/store chardet results per file #4

Open
aparrish opened this issue Aug 12, 2018 · 0 comments
Open

cache/store chardet results per file #4

aparrish opened this issue Aug 12, 2018 · 0 comments

Comments

@aparrish
Copy link
Owner

The chardet detection works pretty well but also takes a long time—I didn't time it exactly, but it felt like it added at least an hour to the time it took the corpus-building process to run on my MacBook Air. Since these files aren't going to change, it makes sense to pre-build and cache the results so that subsequent runs of the corpus-building script don't need to re-run the detection process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant