(feat): Backed zarr and hdf5 sparse matrices (separate from backed mode) #765

ivirshup · 2022-04-29T15:41:26Z

Initial draft of backed sparse array support for zarr.

I intend to export sparse_dataset, CSRDataset, and CSCDataset class from experimental.
Get tests passing
Figure out if I'm going to expose any arguments through read_zarr

codecov · 2022-04-29T15:46:13Z

Codecov Report

Merging #765 (ad016a6) into main (88dd129) will decrease coverage by 2.09%.
The diff coverage is 92.74%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #765      +/-   ##
==========================================
- Coverage   84.88%   82.79%   -2.09%     
==========================================
  Files          36       36              
  Lines        5153     5197      +44     
==========================================
- Hits         4374     4303      -71     
- Misses        779      894     +115

Flag	Coverage Δ
gpu-tests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
anndata/_core/anndata.py	`83.60% <66.66%> (ø)`
anndata/_core/file_backing.py	`90.32% <87.50%> (-1.15%)`	⬇️
anndata/_io/specs/methods.py	`87.52% <90.00%> (-0.19%)`	⬇️
anndata/_core/sparse_dataset.py	`92.73% <93.15%> (+1.06%)`	⬆️
anndata/_core/raw.py	`79.28% <100.00%> (-4.29%)`	⬇️
anndata/_io/h5ad.py	`92.89% <100.00%> (ø)`
anndata/_io/utils.py	`76.47% <100.00%> (ø)`
anndata/experimental/__init__.py	`100.00% <100.00%> (ø)`
anndata/experimental/merge.py	`87.56% <100.00%> (ø)`
anndata/experimental/multi_files/_anncollection.py	`70.56% <100.00%> (ø)`
... and 2 more

... and 3 files with indirect coverage changes

📢 Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scverse).

for more information, see https://pre-commit.ci

ilan-gold · 2023-03-21T13:00:01Z

@ivirshup It seems that we settled on friday on not exposing anything for read_zarr. beyond that, i was not clear what the goal of assignment would be here - did you want to include it or not? i remember you saying that writing in backed mode probably wasn't so great anyway so is the goal here to get rid of it?

ilan-gold · 2023-07-31T19:56:59Z

Initial draft of backed sparse array support for zarr.

I intend to export sparse_dataset, CSRDataset, and CSCDataset class from experimental.

Get tests passing

Figure out if I'm going to expose any arguments through read_zarr

At the moment, the first and last are still open ended as is the question of setting the data. Otherwise, tests pass and you can convert from a draft IMO. Thanks again for getting this started!

ilan-gold · 2023-08-01T13:37:49Z

Decisions:

don't expose arguments to read_zarr
do export the classes from experimental.
DeprecationWarning for sparse setting - eventually remove probably, but leave space for a new implementation.

ivirshup

@ilan-gold looking very good!

I've added a few comments, but otherwise can we add a changelog entry?

ivirshup · 2023-08-28T12:53:26Z

anndata/_core/sparse_dataset.py


    @property
    def value(self) -> ss.spmatrix:
        return self.to_memory()

    def __repr__(self) -> str:
        return (
-            f"<HDF5 sparse dataset: format {self.format_str!r}, "
+            f"<Backed sparse dataset: format {self.format_str!r}, "


How about we put the type name here, so it's more like:

<CSRDataset: backend zarr, shape (10000, 10000), data_dtype '<f8'>

ivirshup · 2023-08-28T12:54:32Z

anndata/_core/sparse_dataset.py

+    # def __setitem__(self, index: Union[Index, Tuple[()]], value):
+
+    #     row, col = self._normalize_index(index)
+    #     mock_matrix = self._to_backed()
+    #     mock_matrix[row, col] = value
+


That sounds good.

I think I prefer deprecation to "Instability". What were you thinking for instability warning?

ivirshup · 2023-08-28T12:57:54Z

anndata/tests/test_backed_sparse.py

+    path = (
+        tmp_path / f"test.{diskfmt.replace('ad', '')}"
+    )  # diskfmt is either h5ad or zarr


I think it would be worth adding the negative test too. These should be quite fast as they can be done with small data, right?

ivirshup · 2023-08-28T12:59:25Z

anndata/_core/sparse_dataset.py

+    format_str = "csc"
+
+
+def sparse_dataset(group) -> BaseCompressedSparseDataset:


Since this is now exported in experimental could it get a docstring with at least one Usage example?

Good point, will do.

ilan-gold · 2023-08-29T11:38:10Z

@ivirshup Where is the changelog? Is it the release-notes? And if so, which release?

for more information, see https://pre-commit.ci

ivirshup · 2023-08-29T11:42:07Z

Where is the changelog?

release-notes/0.10.0.md

… into zarr-sparse-array

ilan-gold · 2023-09-05T09:57:01Z

@ivirshup have look at the deprecation warning! thanks!

… into zarr-sparse-array

ivirshup · 2023-09-07T16:40:44Z

anndata/_core/sparse_dataset.py

+            PendingDeprecationWarning,
+        )
+        row, col = self._normalize_index(index)
+        mock_matrix = self._to_backed()


Suggested change

mock_matrix = self._to_backed()

mock_matrix = self.to_backed()

This is currently throwing an error, but I do like the idea of making it private.

It's used publicly.....

ilan-gold · 2023-09-08T08:24:31Z

I'm not sure where I got the idea in my head that this branch was passing CI but in any case, should we make zarr a dep of anndata now? It's only used for type checking right now in sparse_dataset.py

This is what is causing the failures.

ivirshup · 2023-09-08T11:32:29Z

In anndata.compat there are dummy classes defined for this. Basically replace zarr.Array and zarr.Group with anndata.compat.ZarrArray and anndata.compat.ZarrGroup

ivirshup · 2023-09-08T11:33:04Z

anndata/_core/file_backing.py

-from .sparse_dataset import SparseDataset
-from ..compat import ZarrArray, DaskArray, AwkArray
+from .sparse_dataset import BaseCompressedSparseDataset
+from ..compat import ZarrArray, ZarrGroup, DaskArray, AwkArray


@ilan-gold like this

Start backed sparse support for zarr

2f73576

ivirshup added topic: backed backend: zarr topic: exported functions Area - IO labels Apr 29, 2022

ivirshup added 3 commits October 31, 2022 17:56

Merge branch 'master' into zarr-sparse-array

df160f0

Merge branch 'master' into zarr-sparse-array

7983291

Fix sparse_to_dense

a5e0311

ivirshup added topic: io and removed Area - IO labels Dec 1, 2022

ivirshup and others added 8 commits February 8, 2023 18:38

Merge branch 'master' into zarr-sparse-array

b28448c

[pre-commit.ci] auto fixes from pre-commit.com hooks

5e3cb02

for more information, see https://pre-commit.ci

(chore): remove large comment

d159805

(style): _to_backed -> to_backed

ff2fdfa

(fix): revert backed test

01089fc

Merge branch 'main' into zarr-sparse-array

e038314

(fix): add basic zarr backed reading for test

28f0218

[pre-commit.ci] auto fixes from pre-commit.com hooks

cc51a65

for more information, see https://pre-commit.ci

ilan-gold added 2 commits July 31, 2023 15:01

Merge branch 'main' into zarr-sparse-array

c81f819

(fix): merge.py behavior

d0da485

ilan-gold changed the title ~~Start backed sparse support for zarr~~ (feat): backed support for zarr Jul 31, 2023

ivirshup added this to the 0.10.0 milestone Aug 1, 2023

(chore): add tests for zarr append

4187f51

ilan-gold force-pushed the zarr-sparse-array branch from 4727f97 to 4187f51 Compare August 1, 2023 15:24

(chore): add exports to experimental

9d293ba

ilan-gold force-pushed the zarr-sparse-array branch from 612f781 to 9d293ba Compare August 1, 2023 15:26

ivirshup changed the title ~~(feat): backed support for zarr~~ (feat): Backed zarr and hdf5 sparse matrices (separate from backed mode) Aug 28, 2023

ivirshup commented Aug 28, 2023

View reviewed changes

ilan-gold added 2 commits August 29, 2023 12:53

(chore): change __repr__

1cb7be1

(feat): add append negative test for zarr

78998c1

ilan-gold and others added 2 commits August 29, 2023 13:39

Merge branch 'main' into zarr-sparse-array

ec20ae7

[pre-commit.ci] auto fixes from pre-commit.com hooks

a045c3b

for more information, see https://pre-commit.ci

ilan-gold added 4 commits September 4, 2023 15:08

(docs): add changelog

f926680

Merge branch 'zarr-sparse-array' of https://github.com/ivirshup/anndata…

b0edc03

… into zarr-sparse-array

(chore): add PendingDepreciationWarning for __setitem__

4f7ee76

Merge branch 'main' into zarr-sparse-array

cb1ddcd

ilan-gold added 2 commits September 5, 2023 12:01

(chore): add __future__ import for union

e0a97c3

Merge branch 'zarr-sparse-array' of https://github.com/ivirshup/anndata…

1b58a64

… into zarr-sparse-array

ivirshup commented Sep 7, 2023

View reviewed changes

ilan-gold added 2 commits September 7, 2023 19:08

(fix): _to_backed -> to_backed

49dceb2

Merge branch 'main' into zarr-sparse-array

53b1fde

ivirshup commented Sep 8, 2023

View reviewed changes

ivirshup added 5 commits September 8, 2023 14:11

Make _to_backed private, type fixes

d3161a3

Deprecations: .value -> .to_memory(), .format_str -> .format

aa9324d

Update deprecations

18bee74

Merge branch 'main' into zarr-sparse-array

cc3d403

Docs

ad016a6

ivirshup added the run-gpu-ci label Sep 8, 2023

ivirshup enabled auto-merge (squash) September 8, 2023 13:54

ivirshup merged commit 8c00a29 into scverse:main Sep 8, 2023
15 of 16 checks passed

ivirshup mentioned this pull request Oct 11, 2023

Export SparseDataset #327

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(feat): Backed zarr and hdf5 sparse matrices (separate from backed mode) #765

(feat): Backed zarr and hdf5 sparse matrices (separate from backed mode) #765

ivirshup commented Apr 29, 2022 •

edited

Loading

codecov bot commented Apr 29, 2022 •

edited

Loading

ilan-gold commented Mar 21, 2023

ilan-gold commented Jul 31, 2023 •

edited by flying-sheep

Loading

ilan-gold commented Aug 1, 2023

ivirshup left a comment

ivirshup Aug 28, 2023

ivirshup Aug 28, 2023

ivirshup Aug 28, 2023

ivirshup Aug 28, 2023

ilan-gold Aug 29, 2023

ilan-gold commented Aug 29, 2023

ivirshup commented Aug 29, 2023

ilan-gold commented Sep 5, 2023

ivirshup Sep 7, 2023

ivirshup Sep 7, 2023

ilan-gold Sep 7, 2023

ilan-gold commented Sep 8, 2023 •

edited

Loading

ivirshup commented Sep 8, 2023

ivirshup Sep 8, 2023

		format_str = "csc"


		def sparse_dataset(group) -> BaseCompressedSparseDataset:

	mock_matrix = self._to_backed()
	mock_matrix = self.to_backed()

(feat): Backed zarr and hdf5 sparse matrices (separate from backed mode) #765

(feat): Backed zarr and hdf5 sparse matrices (separate from backed mode) #765

Conversation

ivirshup commented Apr 29, 2022 • edited Loading

codecov bot commented Apr 29, 2022 • edited Loading

Codecov Report

ilan-gold commented Mar 21, 2023

ilan-gold commented Jul 31, 2023 • edited by flying-sheep Loading

ilan-gold commented Aug 1, 2023

ivirshup left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilan-gold commented Aug 29, 2023

ivirshup commented Aug 29, 2023

ilan-gold commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilan-gold commented Sep 8, 2023 • edited Loading

ivirshup commented Sep 8, 2023

Choose a reason for hiding this comment

ivirshup commented Apr 29, 2022 •

edited

Loading

codecov bot commented Apr 29, 2022 •

edited

Loading

ilan-gold commented Jul 31, 2023 •

edited by flying-sheep

Loading

ilan-gold commented Sep 8, 2023 •

edited

Loading