Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use async io where possible to improve runtime performance #163

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

netomi
Copy link

@netomi netomi commented Nov 29, 2023

This PR fixes #162 .

Things that this PR will modify:

  • use aiohttp whereever appropriate to improve performance
  • cleanups of existing code where appropriate (e.g. get_pypi_data_from_purl which returned duplicate data for each package)

@netomi netomi marked this pull request as ready for review November 30, 2023 12:31
@TG1999
Copy link
Contributor

TG1999 commented Nov 30, 2023

@netomi thanks! please check the errors in the CI and also do not forget to regen the tests just in case dependencies in tests have been updated https://github.com/nexB/python-inspector#testing

@netomi
Copy link
Author

netomi commented Nov 30, 2023

There are a couple of things that I still need to fix, but the failing CI runs are unrelated to this PR afaict. They also exist for other PRs and look like that the setup is broken.

@TG1999
Copy link
Contributor

TG1999 commented Nov 30, 2023

@netomi please rebase your PR with latest main, tests are fixed in this PR #165

@netomi
Copy link
Author

netomi commented Nov 30, 2023

ok so there are some unit test failure that I need to address. Most of them are from the fact that python-inspector returned duplicate package entries in its result up to now, which I fixed, so the expected test results need to be adjusted to take this change into account.

@netomi
Copy link
Author

netomi commented Nov 30, 2023

I could fix most tests (weird thing is that 4 tests fail locally when run in pycharm, but work on the command line so that might be a pycharm problem), there are some weird test failures on macos and windows only. Do you have any idea about that?

@netomi
Copy link
Author

netomi commented Dec 1, 2023

still need to take a look at etc/scripts if the there also need to be adjustments after the changes.

@pombredanne
Copy link
Member

@netomi BTW, do not bother with the etc/scripts ... these come from the shared skeleton repo at https://github.com/nexB/skeleton and should not be of concern for you.

@sschuberth
Copy link
Contributor

Can this be moved forward, please? 😬

@TG1999
Copy link
Contributor

TG1999 commented Jan 26, 2024

@sschuberth tests are falling, tests needs to be regen

@sschuberth
Copy link
Contributor

@sschuberth tests are falling, tests needs to be regen

Would you have time to do this @netomi?

@netomi
Copy link
Author

netomi commented Nov 19, 2024

hmm I can take a look, but I am quite discouraged from contributing to a repo that has to constantly rebuilding its tests and they are failing most of the time when creating a PR. You still need to inspect if its related to your changes or the test data is just out of date.

@pombredanne
Copy link
Member

hmm I can take a look, but I am quite discouraged from contributing to a repo that has to constantly rebuilding its tests and they are failing most of the time when creating a PR. You still need to inspect if its related to your changes or the test data is just out of date.

@netomi I agree this is not great. What do you think we could do as an alternative?

@netomi
Copy link
Author

netomi commented Nov 20, 2024

please dont actually call live servers during tests but rather mock their responses with data you might have retrieved from the server at one point in time. I understand that is a bit harder to do, but ultimately results in much more stable and reliable tests imho.

PyGitHub has a nice system for their tests where they replay data:

https://github.com/PyGithub/PyGithub

@sschuberth
Copy link
Contributor

At least one of the errors seems to be

ImportError: cannot import name 'cached_property' from 'functools' (/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/functools.py)

@netomi
Copy link
Author

netomi commented Nov 28, 2024

no idea where this error is coming from. This decorator is not used anywhere in the code. Probably from a dependency? This error btw also occurs for the default branch, so its not related to this PR afaict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use asyncio to gain performance
4 participants