Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing dependencies for: _khash_primitive_helper #55795

Merged
merged 3 commits into from
Nov 6, 2023

Conversation

vaikas
Copy link
Contributor

@vaikas vaikas commented Nov 1, 2023

Thanks to @kaniini for helping me find where to add the dependencies.

@vaikas vaikas requested a review from WillAyd as a code owner November 1, 2023 18:13
@WillAyd
Copy link
Member

WillAyd commented Nov 1, 2023

Might be overlooking it but why do these modules need this as a source? Can you provide the original error you were getting?

@vaikas
Copy link
Contributor Author

vaikas commented Nov 1, 2023

Thank you! Yeah, the original errors are in the issue under Installation Logs under Details, maybe I put them in the wrong section 🤣 . I did also dig deeper that has more details in this issue:
wolfi-dev/os#7334

@kaniini
Copy link

kaniini commented Nov 1, 2023

When building with samurai, the dependency rules are sorted after loading, while the original implementation of ninja orders the dependency rules based on when they were malloc'd. This can result in underdefined dependencies appearing to work simply as a side effect of when the dependency rule was evaluated.

In this case, both arrays.pyx and index.pyx are pulling in pandas._libs via an import (not cimport) statement, which results in khash.pyx being evaluated as part of the cython generation process. But khash.pyx requires generated code, so it needs to be generated before these modules so they can safely pull in pandas._libs.

As an aside, see #51875 which is pointing out basically the same problem.

@WillAyd
Copy link
Member

WillAyd commented Nov 1, 2023

Do these need to be listed in sources or can you add them as deps? I see a few other modules already use deps and that seems like a more natural fit

@kaniini
Copy link

kaniini commented Nov 1, 2023

Yes, probably. Testing it now!

@mroeschke mroeschke added the Build Library building on various platforms label Nov 1, 2023
@kaniini
Copy link

kaniini commented Nov 1, 2023

Sadly, Meson reports when using deps:

../pandas/_libs/meson.build:106:7: ERROR: Argument is of an unacceptable type 'CustomTarget'.

So I think we need to keep it as a source due to limitations of meson.

@WillAyd
Copy link
Member

WillAyd commented Nov 1, 2023

Did you declare that dependency against _khash_primitive_helper or _khash_primitive_helper_dep? Looks like the latter is used in a few places already so would be surprised if that didn't work

@vaikas vaikas force-pushed the fix-deps branch 5 times, most recently from 824f595 to 4f86857 Compare November 1, 2023 22:00
@vaikas
Copy link
Contributor Author

vaikas commented Nov 1, 2023

Ok, phew that was easy 😆 PTAL, looking better?

Copy link
Member

@WillAyd WillAyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm. Thanks for fixing up algos too.

@mroeschke @lithomas1 any objections?

@mroeschke mroeschke added this to the 2.2 milestone Nov 2, 2023
Copy link
Member

@mroeschke mroeschke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No objections on my side

'algos': {'sources': ['algos.pyx', _algos_common_helper, _algos_take_helper, _khash_primitive_helper]},
'arrays': {'sources': ['arrays.pyx']},
'algos': {'sources': ['algos.pyx', _algos_common_helper, _algos_take_helper], 'deps': _khash_primitive_helper_dep},
'arrays': {'sources': ['arrays.pyx'], 'deps': _khash_primitive_helper_dep},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems weird that this would be needed here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with Brock, doesn't arrays not cimport anything from pandas?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right you are! 👍 I reckon it hit that in 'earlier' tries, and the proper fix was in index? 🤷

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it does seem to build, but breaks the unit tests?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that's unrelated.

#55804 should fix it.

Copy link
Member

@lithomas1 lithomas1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@vaikas
Copy link
Contributor Author

vaikas commented Nov 5, 2023

I don't think the failing test is related to this PR looking at other PRs that are failing similarly?

@mroeschke mroeschke merged commit e31a686 into pandas-dev:main Nov 6, 2023
71 of 72 checks passed
@mroeschke
Copy link
Member

Thanks @vaikas

@vaikas vaikas deleted the fix-deps branch November 10, 2023 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build Library building on various platforms
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUILD: Building with samurai fails while ninja works.
6 participants