fh.reader.ReadAt: readFull: context canceled

[carbocation]

I'm using dsub 0.3.7 which uses gcsfuse to mount a Google Storage bucket. Recently, I've started running into a lot of io error messages. I've added some detail on the dsub issue, but it sounds like this is likely more closely related to gcsfuse than dsub so I wanted to start a conversation here.

Is there information about what is happening when we get errors such as:

fuse: 2020/06/18 21:07:50.890584 *fuseops.ReadFileOp error: fh.reader.ReadAt: readFull: context canceled
fuse: 2020/06/18 21:07:50.926473 *fuseops.ReadFileOp error: fh.reader.ReadAt: readFull: context canceled

And, in particular, is this something that we should expect to see frequently? Or is it totally dependent on Google Storage, and is gcsfuse dealing with this as expected? As of now, it seems to occur stochastically, but frequently enough that it's fairly crippling to mount with gcsfuse.

[kovan]

I am getting this also. It appears randomly and with a different file each time.
This is the log with --debug_gcsfuse and --debug_fuse:

fuse_debug: 2021/02/10 11:15:03.858249 Op 0x00002cc6 connection.go:479] -> OK ()
fuse_debug: 2021/02/10 11:15:03.859866 Op 0x00002cc8 connection.go:396] <- ReadFile (inode 10, handle 10, offset 184156160, 131072 bytes)
fuse_debug: 2021/02/10 11:15:03.859900 Op 0x00002cc9 connection.go:396] <- interrupt (fuseid 0x00002cc8)
fuse_debug: 2021/02/10 11:15:03.860117 Op 0x00002cc8 connection.go:481] -> Error: "fh.reader.ReadAt: readFull: context canceled"
fuse: 2021/02/10 11:15:03.860136 *fuseops.ReadFileOp error: fh.reader.ReadAt: readFull: context canceled

[avidullu]

a. strace would help find the source of the interrupt
b. There are other issues #288 which seem to indicate that the interrupt might be coming from the application using gcsfuse and causing the error. So tracking the source of interrupt might be helpful.

[sethiay]

This is similar to #1016.

[bhack]

@sethiay Any news on this? I had something similar in the last two days on GKE GoogleCloudPlatform/gcs-fuse-csi-driver#36

[sethiay]

Hey @bhack, Thanks for reaching out. We suggest to follow up on the GoogleCloudPlatform/gcs-fuse-csi-driver#36.

[sethiay]

Closing the issue as it is transferred to https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver and is closed there.

[charith87]

In case this issue is still open, some of the logs look similar to #1016

We recently released GCSFuse v2.1.0 yesterday which supports disabling these interrupts thereby context not getting cancelled. Pls use the "--ignore-interrupts=true" flag when mounting GCSFuse to enable the new behavior.