Adding a to_spatial method on ParquetSource #11
Conversation
intake_parquet/source.py
Outdated
| @@ -106,5 +106,32 @@ def _to_dask(self): | |||
| self._load_metadata() | |||
| return self._df | |||
|
|
|||
| def to_spatial(self): | |||
| """ | |||
| Create a datashader spatial object from the parquet data | |||
There was a problem hiding this comment.
I think this needs a lot more description, and certainly a link to wherever the general idea is described (e.g., how do you make these special parquet datasets?)
intake_parquet/source.py
Outdated
| # Load metadata | ||
| props = json.loads(pf.key_value_metadata['SpatialPointsFrame']) | ||
| else: | ||
| props = None |
intake_parquet/source.py
Outdated
| props = None | ||
|
|
||
| # Call DataFrame constructor with the internals of frame | ||
| return SpatialPointsFrame(frame.dask, frame._name, frame._meta, |
There was a problem hiding this comment.
Presumably this can fail in various way?
I suppose the point is to pass this immediately to graphics, would it make sense to (optionally) construct a datashader/holoviews object right here?
There was a problem hiding this comment.
Maybe I should have made this clearer, I copied all of this straight from: https://github.com/pyviz/datashader/blob/master/datashader/spatial/points.py#L297-L312
There was a problem hiding this comment.
OK, so we just defer everything to whatever that interface does... I guess then the docstring should be caveated to say that whatever goes wrong, it's not Intake's fault!
conda/meta.yaml
Outdated
| @@ -30,6 +30,7 @@ test: | |||
| - pytest | |||
| - pytest-cov | |||
| - coverage | |||
| - datashader==0.7.0 | |||
There was a problem hiding this comment.
This doesn't seem right. The test skips without datashader, so it's not required; the things here are those needed to be able to run the tests. You did this because there is no separate "testing environment" and tests run via conda-build?
There was a problem hiding this comment.
Right. I was trying to run the tests on CI. We should probably set up this repo properly so that there is separate testing env.
There was a problem hiding this comment.
Can that fit in here? Could be as simple as
conda instal ...
pytest ...
in travis file, instead of the conda build line. Since we don't need to auto-upload, makes sense to me.
intake_parquet/source.py
Outdated
|
|
||
| frame = self._df or self.to_dask() | ||
| urlpath = self._get_cache(self._urlpath)[0] | ||
| pf = fp.ParquetFile(urlpath) |
There was a problem hiding this comment.
I guess this probably doesn't work for remote. Is there a good way to get key_value_metadata off the dask version?
There was a problem hiding this comment.
No there isn't. You need slightly different things to work with FP directly, but it can be done.
|
I didn't notice the new commits, sorry. Any idea why this isn't testing? |
|
I suppose just travis flakiness. |
|
I was doing a little spring cleaning and came across this PR. Closing since it's 4 years old, although it looks like another option might be to merge it? |
|
Thanks @jsignell . I think "spatial parquet" is being replaced by geopandas related things and/or awkward array, so there is no need for this any more. Plus, you MUST check out intake 2.0, which will make all these |
|
I can't wait! Intake-stac is dying to be rewritten once intake 2.0 is ready :) |
Fixes #10