Add first array draft #1090
Merged
Add first array draft #1090
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
lets merge this |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to get #471 passing.
Apart from missing implementation, I encountered a lot of trouble in the assertion code. For example
https://github.com/dask/dask/blob/36e9d7c84c619abcfc498e3fdf7b37f7fb8fd400/dask/array/utils.py#L240
is accessing a key of a HLG and is expecting this to resolve to the key in the low level graph, see custom getitem logic https://github.com/dask/dask/blob/36e9d7c84c619abcfc498e3fdf7b37f7fb8fd400/dask/highlevelgraph.py#L508-L528
This works... at times. Especially, the later part of the graph expects this result, i.e. the low level graph value to be an actual array (since it was persisted a couple of lines further above) which breaks since we have to attach a dummy layer in the
FromGraph(to avoid name collisions). Ideally, this would use a "Get Nth chunk" API but I'm not sure if that exists.There are likely plenty of other problems around here but this is what I found so far