Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigating hanging on Travis CI #2113

Closed
wants to merge 9 commits into from

Conversation

takluyver
Copy link
Member

This is probably not meant to be merged, I'm just poking Travis CI to figure out why builds are hanging.

@codecov
Copy link

codecov bot commented Jun 13, 2022

Codecov Report

Merging #2113 (bc5a59c) into master (3badee4) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2113   +/-   ##
=======================================
  Coverage   90.00%   90.00%           
=======================================
  Files          17       17           
  Lines        2350     2350           
=======================================
  Hits         2115     2115           
  Misses        235      235           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3badee4...bc5a59c. Read the comment docs.

@takluyver
Copy link
Member Author

OK, I think I've figured it out. The codecov uploader gets an upload URL from codecov which is either on S3 (according to the logs) or Google Cloud (according to https://github.com/seantis/pytest-codecov/blob/master/src/pytest_codecov/codecov.py ). It tries to upload the coverage XML file to there with a PUT request. This appears to be blocked on the arm64 machines, in a way that causes the .connect() call on the socket to hang rather than failing. I might have a wild guess that this is part of some effort to mitigate cryptocurrency mining on free CI resources (so once again, crypto is the reason we can't have nice things 😠 ).

I thought about moving to the new codecov uploader, but if my guess above is correct, this wouldn't fix it - a connection to either S3 or google cloud is not going to work from typescript if it fails from Python. And the new uploader doesn't have an arm64 binary anyway (codecov/uploader#523 ).

I suspect there's no way round that, so all we can really do is ignore the failure, or skip codecov altogether on the arm64 jobs. 😞


Traceback:

    Pinging Codecov...
    Uploading to S3...
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.13/bin/codecov", line 8, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/codecov/__init__.py", line 1177, in main
    "Content-Encoding": "gzip",
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/codecov/__init__.py", line 301, in retry_upload
    res = request_method(url, **kwargs)
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/requests/api.py", line 130, in put
    return request("put", url, data=data, **kwargs)
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
    timeout=timeout,
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/travis/virtualenv/python3.7.13/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
KeyboardInterrupt

@takluyver
Copy link
Member Author

I've rebased the changes to work around this as #2114.

@takluyver takluyver closed this Jun 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant