Run stubtest on stubs on different platforms #8923

sobolevn · 2022-10-18T11:43:13Z

Open problems:

How to install deps on other platforms. brew works well on macos, but no idea how to use windows (I've never touched it since 2008)
Listing platform specific packages. Which ones should we add?
I still need to change daily.yml to do the same thing for all stubs
How to test this? Not quite clear for now. I will need to tweak some values

sobolevn · 2022-10-18T12:02:17Z

I have no idea how to fix this Windows problem:


Run STUBS=$(
ParserError: D:\a\_temp\4a20ead6-174f-4640-96c9-b47d87c97584.ps1:8
Line |
   8 |  if [ -n "$STUBS" ]; then
     |    ~
     | Missing '(' after 'if' in if statement.

It is valid sh, what more do you need? 🤣
Probably we need two different test runners: one for win and one for posix

AlexWaygood · 2022-10-18T12:04:27Z

It is valid sh, what more do you need? 🤣
Probably we need two different test runners: one for win and one for posix

The Windows shell is a whole different beast.

Can we use WSL in GitHub Actions? That might be an easy workaround.

sobolevn · 2022-10-18T12:13:28Z

Looks like it works! https://github.com/python/typeshed/actions/runs/3273273447/jobs/5385277526

Avasam · 2022-10-18T16:33:29Z

It is valid sh, what more do you need?

But not valid powershell ! Although you already figured out that you can specify the shell in github actions. Which is what I would've recommended over:

Can we use WSL in GitHub Actions? That might be an easy workaround.

How to install deps on other platforms

Chocolatey is what I see most often on windows (at least when developing with Node and Python).
Brew for MacOS.

Listing platform specific packages. Which ones should we add?

Whatever's needed as you develop/test this I guess?
Taking a look at stubtest_allowlist.txt files, and skip=true in METADATA.toml, seems to be a good starting point to get an idea of which stubs have os-specific needs.

tests/stubtest_third_party.py

Akuli · 2022-10-19T07:59:47Z

You can use shell: bash on Windows github actions.

sobolevn · 2022-10-19T08:00:11Z

I already do that, thanks!

AlexWaygood · 2022-10-19T09:55:26Z

Could you make this change to the METADATA.toml file for D3DShot as part of this PR, to test that the Windows part of this is working?

  [tool.stubtest]
- # The library only works on Windows; we currently only run stubtest on Ubuntu for third-party stubs in CI.
- # See #8660
- skip = true
+ platform = ["win32"]

Akuli · 2022-10-19T09:59:40Z

Nit: I would prefer if the metadata key was named platforms instead of platform.

.github/workflows/stubtest_stubs.yml

AlexWaygood · 2022-10-19T10:49:25Z

You might have to add a @tests/requirements-stubtest.txt file to D3DShot, and pin Pillow to a specific revision, in order to get it to install.

sobolevn · 2022-10-19T10:54:32Z

Output from windows daily run: https://github.com/sobolevn/typeshed/actions/runs/3280905801/jobs/5402262235

…/sobolevn/typeshed into run-stubs-on-different-platforms

sobolevn · 2022-10-19T17:37:25Z

Since we now have platform-specific stubtest failures: https://github.com/sobolevn/typeshed/actions/runs/3283716889/jobs/5408788907

I will introduce platform-specific allowlists. Something similar to what we have in stdlib/

stubs/pywin32/@tests/stubtest_allowlist_win32.txt

AlexWaygood

Looks great to me! Thanks for all the work you've done on this!

github-actions · 2022-11-11T00:21:58Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

CONTRIBUTING.md

JelleZijlstra

Slight wording suggestion because many stubs may be "supported" on other platforms, we just don't need to run stubtest multiple times.

Co-authored-by: Jelle Zijlstra <[email protected]>

github-actions · 2022-11-11T10:21:33Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

AlexWaygood

A few more tiny nits I spotted while looking over again

.github/workflows/stubtest_third_party.yml

CONTRIBUTING.md

sobolevn · 2022-11-11T13:04:43Z

stubs/D3DShot/METADATA.toml

@@ -2,6 +2,6 @@ version = "0.1.*"
 requires = ["types-Pillow"]

 [tool.stubtest]
-# The library only works on Windows; we currently only run stubtest on Ubuntu for third-party stubs in CI.
-# See #8660
+# TODO: re-enable


Suggested change

# TODO: re-enable

Wait, this TODO comment I did like -- it would be good if we could figure out how to enable stubtest for this package!

Co-authored-by: Alex Waygood <[email protected]>

github-actions · 2022-11-11T13:17:43Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

stubs/D3DShot/METADATA.toml

github-actions · 2022-11-11T13:36:27Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

AlexWaygood · 2022-11-11T13:39:53Z

Time for this to go in.

Avasam · 2022-11-12T17:05:42Z

@sobolevn & @AlexWaygood There's an unforseen issue: Now I can't run stubtest locally on any stub that doesn't explicitely specify my non-linux OS!

If no platform is specified in METADATA.toml, it should be allowed to be run on any OS, but the CI should only run linux.

sobolevn · 2022-11-12T17:31:52Z

I think we can add --local-run flag to be sure that it is indeed runing on a local env.

Avasam · 2022-11-12T17:41:15Z

Proposed solution: #9173

sobolevn added 3 commits October 18, 2022 14:39

Run stubtest on stubs on different platforms

61deba6

Fix CI

0a96d99

Fix CI

8149e3c

This comment has been minimized.

Sign in to view

Fix CI

7d364f7

This comment has been minimized.

Sign in to view

AlexWaygood reviewed Oct 18, 2022

View reviewed changes

tests/stubtest_third_party.py Outdated Show resolved Hide resolved

sobolevn added 2 commits October 19, 2022 13:35

Address review

1a9af3a

Run daily on differeny platforms

f00714a

AlexWaygood reviewed Oct 19, 2022

View reviewed changes

.github/workflows/stubtest_stubs.yml Outdated Show resolved Hide resolved

Fix daily run

3f31bed

This comment has been minimized.

Sign in to view

Avasam mentioned this pull request Oct 19, 2022

pywin32: Remove IID that doesn't exist at runtime #8930

Merged

Merge branch 'master' into run-stubs-on-different-platforms

8eb0c83

This comment has been minimized.

Sign in to view

sobolevn added 2 commits October 19, 2022 20:29

Rename stubtest files

c8650b3

Merge branch 'run-stubs-on-different-platforms' of https://github.com…

770c5aa

…/sobolevn/typeshed into run-stubs-on-different-platforms

This comment has been minimized.

Sign in to view

Add platform-specific allowlists for stubs

fdf13bb

AlexWaygood reviewed Nov 11, 2022

View reviewed changes

stubs/pywin32/@tests/stubtest_allowlist_win32.txt Outdated Show resolved Hide resolved

Delete unused allowlist entries

7df5a1f

AlexWaygood approved these changes Nov 11, 2022

View reviewed changes

AlexWaygood requested a review from JelleZijlstra November 11, 2022 00:15

JelleZijlstra reviewed Nov 11, 2022

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

JelleZijlstra reviewed Nov 11, 2022

View reviewed changes

Update CONTRIBUTING.md

949bf46

Co-authored-by: Jelle Zijlstra <[email protected]>

AlexWaygood reviewed Nov 11, 2022

View reviewed changes

.github/workflows/stubtest_third_party.yml Outdated Show resolved Hide resolved

.github/workflows/stubtest_third_party.yml Outdated Show resolved Hide resolved

.github/workflows/stubtest_third_party.yml Show resolved Hide resolved

CONTRIBUTING.md Outdated Show resolved Hide resolved

sobolevn commented Nov 11, 2022

View reviewed changes

Apply suggestions from code review

faac1be

Co-authored-by: Alex Waygood <[email protected]>

AlexWaygood reviewed Nov 11, 2022

View reviewed changes

stubs/D3DShot/METADATA.toml Show resolved Hide resolved

Add TODO comment back

9464335

AlexWaygood merged commit 9cd9f6f into python:main Nov 11, 2022

This was referenced Nov 11, 2022

Do not run stubtest_third_party on requirements change #9160

Merged

Run stubtest under different OS environments #8660

Closed

This was referenced Nov 12, 2022

Run stubtest on windows for colorama #9168

Merged

Run stubtest for psutil on all platforms #9169

Merged

Avasam mentioned this pull request Nov 12, 2022

stubtest_third_party.py: Allow running non-listed platforms locally #9173

Merged

This was referenced Nov 12, 2022

cffi: run stubtest on Windows #9178

Merged

Fix paramiko stubtest on Windows #9179

Merged

Run stubtest on Windows for waitress #9181

Merged

AlexWaygood mentioned this pull request Aug 16, 2023

Bump aiofiles to 23.2.* #10584

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run stubtest on stubs on different platforms #8923

Run stubtest on stubs on different platforms #8923

sobolevn commented Oct 18, 2022 •

edited

Loading

sobolevn commented Oct 18, 2022

AlexWaygood commented Oct 18, 2022 •

edited

Loading

This comment has been minimized.

sobolevn commented Oct 18, 2022

This comment has been minimized.

Avasam commented Oct 18, 2022 •

edited

Loading

Akuli commented Oct 19, 2022

sobolevn commented Oct 19, 2022

AlexWaygood commented Oct 19, 2022

Akuli commented Oct 19, 2022

AlexWaygood commented Oct 19, 2022

sobolevn commented Oct 19, 2022

This comment has been minimized.

This comment has been minimized.

sobolevn commented Oct 19, 2022

This comment has been minimized.

AlexWaygood left a comment

github-actions bot commented Nov 11, 2022

JelleZijlstra left a comment

github-actions bot commented Nov 11, 2022

AlexWaygood left a comment

sobolevn Nov 11, 2022 •

edited

Loading

AlexWaygood Nov 11, 2022

github-actions bot commented Nov 11, 2022

github-actions bot commented Nov 11, 2022

AlexWaygood commented Nov 11, 2022

Avasam commented Nov 12, 2022 •

edited

Loading

sobolevn commented Nov 12, 2022

Avasam commented Nov 12, 2022

Run stubtest on stubs on different platforms #8923

Run stubtest on stubs on different platforms #8923

Conversation

sobolevn commented Oct 18, 2022 • edited Loading

sobolevn commented Oct 18, 2022

AlexWaygood commented Oct 18, 2022 • edited Loading

This comment has been minimized.

sobolevn commented Oct 18, 2022

This comment has been minimized.

Avasam commented Oct 18, 2022 • edited Loading

Akuli commented Oct 19, 2022

sobolevn commented Oct 19, 2022

AlexWaygood commented Oct 19, 2022

Akuli commented Oct 19, 2022

AlexWaygood commented Oct 19, 2022

sobolevn commented Oct 19, 2022

This comment has been minimized.

This comment has been minimized.

sobolevn commented Oct 19, 2022

This comment has been minimized.

AlexWaygood left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 11, 2022

JelleZijlstra left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 11, 2022

AlexWaygood left a comment

Choose a reason for hiding this comment

sobolevn Nov 11, 2022 • edited Loading

Choose a reason for hiding this comment

AlexWaygood Nov 11, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 11, 2022

github-actions bot commented Nov 11, 2022

AlexWaygood commented Nov 11, 2022

Avasam commented Nov 12, 2022 • edited Loading

sobolevn commented Nov 12, 2022

Avasam commented Nov 12, 2022

sobolevn commented Oct 18, 2022 •

edited

Loading

AlexWaygood commented Oct 18, 2022 •

edited

Loading

Avasam commented Oct 18, 2022 •

edited

Loading

sobolevn Nov 11, 2022 •

edited

Loading

Avasam commented Nov 12, 2022 •

edited

Loading