BUG: fetch_string_and_headers compat: raise in and out of pyodide #129
Conversation
tests/test_compat.py
Outdated
| from micropip._compat import HttpStatusError, fetch_string_and_headers | ||
|
|
||
|
|
||
| @pytest.mark.asyncio |
There was a problem hiding this comment.
Do I just need to throw a @run_in_pyodide here to make sure this is tested in and out of pyodide ? Or is that a bit more complicated ?
There was a problem hiding this comment.
It's a bit more complicated. Honestly, the easiest thing might be just to copy paste the code. At some point we'll have to think about a better way...
There was a problem hiding this comment.
Yep, testing something both in and out of Pyodide is a bit complex. Actually, I think you don't need to test out-of-pyodide case. We don't care about using micropip out of Pyodide and not_in_pyodide exists mostly for test purpose and typing.
|
Moving to draft, I think HttpStatusError is not yet in latest released pyodide. |
Carreau
changed the title
|
Ok, I've added relevant fixes now that PyPI properly set CORS on 404s, and added conditional for wether we are on pyodide 0.26.x or 0.27+. |
Carreau
changed the title
|
SO now some of the fixture that return 500 now need to return 404... I'm not sure how to get the httpserver fixture to return 404 on wildcard url without expecting requests. |
|
|
Let's release Pyodide 0.27.0a1 and use it in the micropip CI so we don't need to put complex version checks. Hood released 0.27.0a1 yesterday but the deploy was failed because of the CI timeout (pyodide/pyodide@5231ac0). Let me see if rerunning the job would work. |
I was also just concern micropip might be broken with current pyodide/warehouse, but a new version of pyodide would be great :-) |
|
Ok, I've tried multiple variation of the test to |
tests/test_compat.py
Outdated
|
|
||
|
|
||
| @run_in_pyodide | ||
| async def _inner_test_404_raise(httpserver): |
There was a problem hiding this comment.
| async def _inner_test_404_raise(httpserver): | |
| async def _inner_test_404_raise(selenium, httpserver): |
There was a problem hiding this comment.
Also the httpserver fixture isn't going to work with @run_in_pyodide. I'm not sure what the best way is to do this, @ryanking13 do you have any suggestions?
There was a problem hiding this comment.
You can do this way (passing only string not httpserver fixture)
def test_404(httpserver):
@run_in_pyodide
async def _inner_test_404_raise(url):
from ... import fetch_string_and_headers
import pytest
with pytest.raises(HttpStatusError):
await fetch_string_and_headers(url, {})
httpserver.expect_request("/404").respond_with_data(
"Not found", status=404, content_type="text/plain"
)
url_404 = httpserver.url_for("/404")
_inner_test_404_raise(url_404)There was a problem hiding this comment.
Ah... let me try that, thanks for the pointers.
|
I released 0.27.0a2 instead. |
Carreau
force-pushed
the
status-raises
branch
5 times, most recently
from
a502d5e to
6d73dd2
Compare
BTW should CI be updated - pyodide-version: [0.24.1, 0.25.0]
+ pyodide-version: [0.24.1, 0.25.0, 0.26.0, 0.27.0a2]? |
Carreau
force-pushed
the
status-raises
branch
from
a757ef3 to
7a6db4b
Compare
|
Any idea on how to actually change the timeout to something else the 10 minutes... I've tried various options with no success. |
I think this is a chrome (+selenium) issue.. for now I just pinned the version of the chrome to some older version to see if it pass. #135 |
Carreau
force-pushed
the
status-raises
branch
from
d1c6313 to
b4afd0a
Compare
Carreau
force-pushed
the
status-raises
branch
from
6ce80ba to
06cd694
Compare
Carreau
force-pushed
the
status-raises
branch
5 times, most recently
from
1523d5a to
11f61d2
Compare
Currently only the not_in_pyodide will raise on non-success, because this is the default behavior of urllib, the in_pyodide will not, so I added a raise_for_status. It is better to raise, as otherwise the package parser will potentially get proper URL and not manage to parse it, and decide there is no wheels, while we actually just got an error (404, or maybe 500). In addition wraps both case in a custom local HttpStatusError, so that we can actually catch these errors in the right places when we encounter them. Also add handling for PyPI 404 Now that warehouse set cors to 404, (pypi/warehouse#16339) we need to change the checked exceptions as there is no more network errors.
Carreau
force-pushed
the
status-raises
branch
from
11f61d2 to
5c98d90
Compare
| if parse(pyodide.__version__) > parse("0.27"): | ||
| from pyodide.http import HttpStatusError, pyfetch | ||
| else: | ||
|
|
||
| class HttpStatusError(Exception): # type: ignore [no-redef] | ||
| """we just want this to be defined, it is never going to be raised""" | ||
|
|
||
| pass |
There was a problem hiding this comment.
I removed Pyodide versions < 0.27 from CI. I am fine with removing these conditions and drop backward compatibility for versions < 0.27 @hoodmane WDYT?
micropip/package_index.py
Outdated
| try: | ||
| import pyodide | ||
| from packaging.version import parse | ||
|
|
||
| if parse(pyodide.__version__) > parse("0.27"): | ||
| # reraise on more recent pyodide. | ||
| raise | ||
| continue | ||
| except ImportError: | ||
| # not in pyodide. | ||
| raise |
|
ok, I pushed a commit that remove compat with old versions. |
for more information, see https://pre-commit.ci
Currently only the not_in_pyodide will raise on non-success, because this is the default behavior of urllib, the in_pyodide will not, so I added a raise_for_status.
It is better to raise, as otherwise the package parser will potentially get proper URL and not manage to parse it, and decide there is no wheels, while we actually just got an error (404, or maybe 500).
In addition wraps both case in a custom local HttpStatusError, so that we can actually catch these errors in the right places when we encounter them.
I think we could define HttpStatusError on the ABC and not make it an ABC itself, but it small enough that I think duplication is ok.