Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data transfer resumption for large data transfers exceeds message size limits #235

Open
aarshkshah1992 opened this issue Aug 19, 2021 · 1 comment
Open

Data transfer resumption for large data transfers exceeds message size limits #235

aarshkshah1992 opened this issue Aug 19, 2021 · 1 comment
Assignees
@hannahhoward
Labels
x/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs

Comments

@aarshkshah1992
Copy link
Collaborator

To support transfer resumption, a data-transfer requestor sends a Graphsync DoNotSendCids message to the responder with ALL the CIDs received so far. This solution does NOT scale as sending ALL cids recieved for large data transfers causes the message size to exceed the message size limits set by the Graphsync protocol.

We need to redesign how we resume large data transfers.
@aarshkshah1992
Copy link
Collaborator Author

cc @whyrusleeping who has been running into this with Estuary resuming 32G data transfers.

@jacobheun jacobheun added the x/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs label Aug 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hannahhoward hannahhoward

Labels
x/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jacobheun @hannahhoward @aarshkshah1992