Range requests for encrypted blobs

[clehner]

As reported in thread %YvyRh0zP/W+Lk5g9aysGM1ecMm4T9lE/JjmBaO6IDcs=.sha256 about playing videos from encrypted blobs.

Demo

We have the same file, added as an (unencrypted) blob, and as a boxed (encrypted) blob:

$ curl -s 'http://localhost:8989/blobs/get/&HAKR34cROYxgirpU7wLhdUt2/8EJkcRb6bXNJEU7L4o=.sha256' | sha256sum
1c0291df8711398c608aba54ef02e1754b76ffc10991c45be9b5cd24453b2f8a  -
$ curl -s 'http://localhost:8989/blobs/get/&fNYYzSwokE4DSZsqJxkoSPAWiFXJzGQBt+LY9Mcfgmg=.sha256?unbox=HAKR34cROYxgirpU7wLhdUt2/8EJkcRb6bXNJEU7L4o=.boxs' | sha256sum
1c0291df8711398c608aba54ef02e1754b76ffc10991c45be9b5cd24453b2f8a  -

Playing the unencrypted one works:

$ mpv 'http://localhost:8989/blobs/get/&HAKR34cROYxgirpU7wLhdUt2/8EJkcRb6bXNJEU7L4o=.sha256'
 (+) Video --vid=1 (*) (h264 1080x1080 30.000fps)
 (+) Audio --aid=1 (*) (aac 1ch 44100Hz)
AO: [pulse] 44100Hz mono 1ch float
VO: [gpu] 1080x1080 yuv420p
AV: 00:00:00 / 00:00:10 (7%) A-V:  0.000 Cache: 9.7s/1MB


Exiting... (Quit)

Playing the encrypted one does not:

$ mpv 'http://localhost:8989/blobs/get/&fNYYzSwokE4DSZsqJxkoSPAWiFXJzGQBt+LY9Mcfgmg=.sha256?unbox=HAKR34cROYxgirpU7wLhdUt2/8EJkcRb6bXNJEU7L4o=.boxs'
[ffmpeg] Seek failed (to 1121777, size -38)
[ffmpeg] Seek failed (to 1121777, size -38)
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
[ffmpeg] http: Will reconnect at 1134338 in 0 second(s), error=End of file.
[ffmpeg] http: Will reconnect at 1134338 in 1 second(s), error=End of file.
[ffmpeg] http: Will reconnect at 1134338 in 3 second(s), error=End of file.
[ffmpeg] http: Will reconnect at 1134338 in 7 second(s), error=End of file.
[ffmpeg] http: HTTP error 416 Range Not Satisfiable
[ffmpeg] Seek failed (to 9223372036854775799, size -38)
 (+) Video --vid=1 (*) (h264 1080x1080 30.000fps)
 (+) Audio --aid=1 (*) (aac 1ch 44100Hz)
[ffmpeg] Seek failed (to 36, size -38)
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: stream 0, offset 0x24: partial file
[lavf] error reading packet: Invalid data found when processing input.
[ffmpeg] Seek failed (to 2800, size -38)
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: stream 0, offset 0xaf0: partial file
[lavf] error reading packet: Invalid data found when processing input.
[ffmpeg] Seek failed (to 4770, size -38)
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: stream 1, offset 0x12a2: partial file
[lavf] error reading packet: Invalid data found when processing input.


Exiting... (Errors when loading file)

I suspect the problem is that the video is being played using a HTTP Range request, which we are not handling correctly. multiblob-http handles Range requests. But it doesn't know about the blob unboxing. It takes a transform function which we use for the blob unboxing, but it applies that function after calling blobs.getSlice. The result is that the range is applied to a slice of the encrypted blob and then we attempt to unbox that, which fails.

To fix this, I think we have to either disable support for range requests for boxed blobs, or fix the implementation.

To make range requests for boxed streams work, the easy inefficient way would be to decrypt the blob stream from the start (i.e. using blobs.get instead of blobs.getSlice), and then slice the resulting transformed stream to the ranges requested (i.e. discard the stream up until the offset we want.

To efficiently support random access on a boxed blob, we could cache the decryption state: for each box-stream packet in the blob stream, record a mapping of its offset in the decrypted stream to the offset in the encrypted stream stream and the nonce needed to decrypt it. This mapping could be cached in memory and/or on disk. To answer a range request for a boxed blob, we look up in the mapping the nonce and ciphertext offset for the closest offset ≤ the requested cleartext offset, get the data starting there using getSlice, decrypt it with the passed unbox key and nonce, and then slice it to fit the requested range exactly.