Range requests browser caching not working in Firefox (browser bug)

[daniel-j-h]

Hey folks, I noticed earlier in my experimentation with maplibre and pmtiles that range requests against a .pmtiles file are not getting cached by browsers (I'm talking about Firefox and Chrome specifically).

I just ran across @bdon's https://bdon.github.io/overture-tiles/places.html#10.28/52.4901/13.3301 on mastodon and used this example to drill further down into what's happening because I can reproduce the issue there.

The summary is that currently neither Firefox nor Chrome can cache the range requests against .pmtiles, not even Chrome even tho it sends an If-Range with an ETag and the ETag matches (Firefox sends a regular Range request).

Firefox

Check this out. On second page load (warm cache)

1. In red: the individual .mvt tiles are getting cached because these are individual files
2. In blue: the range requests against the single .pmtiles are not getting cached and the requests always go through

Here are the request and response headers; notice

1. Firefox sends a Range request
2. The .pmtiles have an ETag attached to it

Chrome

Check this out. On second page load (warm cache)

1. In red: again the .mvt files are getting cached
2. In blue: the range requests against the single .pmtiles are again not getting cached and requests always go through

Here are the request and response headers; notice

1. Chrome sends an If-Range request and attached the ETag
2. The response returns the very same ETag, so even though the ETags are matching it's not getting cached

---

What's happening here?

I looked around it and it seems like both Firefox as well as Chrome's support for caching range requests is very limited.

For example for Firefox

• https://searchfox.org/mozilla-central/source/netwerk/protocol/http/nsHttpChannel.cpp#3617-3625
• https://searchfox.org/mozilla-central/source/netwerk/protocol/http/nsHttpChannel.cpp#3658-3662

they only allow to cache a range request for the very first byte range starting at byte zero.

// Don't cache byte range requests which are subranges, only cache 0-
// byte range requests.
if (IsSubRangeRequest(mRequestHead)) {
  return NS_OK;
}

How to move forward

With limited support of caching range requests in the browsers, I see the following ways forward

1. We don't care about caching and always re-run requests against the .pmtiles file
2. We have a lambda or similar in front of the static file turning range requests into individual file get requests
3. We implement caching ourselves in the pmtiles lib here in this repo
4. We implement caching ourselves e.g. as a maplibre plugin

I wanted to flag this with you since you might have thoughts on this and might have encountered this before.

What I wanted to add: pmtiles is such an amazing format for dropping a single file onto a static host that not having caching working by default is a bit unfortunate, because otherwise it shines and is such a cool idea 👌

[bdon]

Thanks for investigating this thoroughly

I don't think 3 and 4 are viable options because they would require interacting with the browser's storage APIs. MapLibre will already cache things internally I believe, but not across page loads. If caching across page loads is absolutely mandatory then there is many options such as lambda, cloudflare or caddy deployment to make it work as a ZXY tile endpoint.

The best solve here is for Firefox to behave the same way as the other browsers, of course.

[daniel-j-h]

I don't think 3 and 4 are viable options because they would require interacting with the browser's storage APIs. MapLibre will already cache things internally I believe, but not across page loads. If caching across page loads is absolutely mandatory then there is many options such as lambda, cloudflare or caddy deployment to make it work as a ZXY tile endpoint.

Why is interacting with the browser storage APIs not an option until the browsers catch up and properly cache range requests? Would love to understand your thinking behind this.

If we wanted to create a plugin, where would it best go, in the maplibre ecosystem and intercepting transformRequest?

And Maplibre does cache tiles internally once they are loaded, but the use cases I'm looking at are e.g.

• react components which are then re-running requests against the server every time they unmount/mount
• tiles that don't change at all e.g. date- or hash stamped where you would normally set a very long cache-control max-age and immutable header. For pages users go to multiple times a day but some tiles stay the same for the whole year.

The best solve here is for Firefox to behave the same way as the other browsers, of course.

Hey by the way, it's both Firefox and Chrome that I tested, both have this issue.

[bdon]

Why is interacting with the browser storage APIs not an option until the browsers catch up and properly cache range requests? Would love to understand your thinking behind this.

It is a viable option, just outside the scope of this decoding library - you could write something that uses Service Workers, for example, but that would not be general among all the runtimes that the JS library supports: browser, deno, lambda, cloudflare workers, etc.

I guess the MapLibre adapter is only browser runtimes, so we could interact with local storage APIs inside the addProtocol bits, but would that storage behavior happen only on firefox/chrome?

I don't have access to Google Chrome right now but Arc (chromium) caches 206s in the browsers fine (screenshot)

[bdon]

Here is the Firefox bug tracker where there's discussion of how this affects Cloud Optimized GeoTIFF in the same way:

https://bugzilla.mozilla.org/show_bug.cgi?id=1615698

[daniel-j-h]

Quick correction: range request caching in Chrome works fine except in incognito windows (most likely by design).

I just re-tested this in a normal Chrome window and it works indeed. Let me change the title to reflect that.

[bdon]

In js v3 we're planning to move to always sending a conditional request If-Match: ${etag}. It turns out that this makes all range requests other than the header uncacheable. Even if If-Match matches a browser-cached resource it will always send a server request to get the updated etag. Reproduce like this:

<script>
  fetch("https://build.protomaps.com/20240130.pmtiles",{headers:{range:'bytes=0-1'}}).then(resp => {
    console.log(resp);
  })
  fetch("https://build.protomaps.com/20240130.pmtiles",{headers:{range:'bytes=0-1','if-match':'fbad7cc5b8c7510e8301d8adb6b12335-442'}}).then(resp => {
    console.log(resp);
  })
  fetch("https://build.protomaps.com/20240130.pmtiles",{headers:{range:'bytes=0-1','if-match':'fbad7cc5b8c7510e8301d8adb6b12335-442'}}).then(resp => {
    console.log(resp);
  })
</script>

(substitute in your own file and etag if necessary). if browser cache is enabled in developer mode, at most the first request without if-match succeeds, the 2nd and 3rd will never be returned from browser cache.

So this change will effectively make all browsers act like Firefox. The trade-off is that the JS client will work correctly with archives updated in-place, which is useful for many applications.

[daniel-j-h]

Do I get this right - in v3 we want to send the If-Match header not to allow for caching (because not a single browser supports it) but to force cache busting unconditionally? For .pmtiles updating in-place would a Cache-Control: no-cache header not work?

In that case a middleware (e.g. lambda) turning the uncacheable byte-range requests into trivially cacheable

/{z}/{x}/{y}.mvt

requests will become more and more unavoidable.

[bdon]

Well, I spent today testing it out and I'm rolling back this change :P

It turns out If-Match causes not only browser cache misses, but also a preflight CORS for every request, which can add 200ms+ to each tile. I'm implementing a different approach that will use cache: 'reload' in fetch conditionally only when an ETag change is detected. So v3 should be 100% as cacheable as v2, but correct the cases around in-place updates which are broken right now.

[bdon]

(We are still going to use If-Match for Lambda, Cloudflare and go-pmtiles, just not for the browser Fetch API which interacts with CORS and the browser cache)

[daniel-j-h]

I appreciate your work here, this is really tricky and touching browser specific edge cases! 🚀

In terms of preflight CORS we had similar issues with our pmtiles deployment internally where browsers differed and Safari was the only one issuing a preflight CORS request. Definitely a bit tricky to make the setup work for all browsers when the byte-range requests are so finicky.

[bdon]

JS 3.0.0-alpha.2 is on NPM implementing this, see changelog here: https://github.com/protomaps/PMTiles/blob/main/js/CHANGELOG.md

It may require you change from import pmtiles from "pmtiles" to named imports import {PMTiles} from "pmtiles"

This should be the final RC of 3.0.0, planning to publish the new major version this week.

[rmcf]

I got a strange error "Decoding failed" in Firefox v.124 with .pmtiles files deployed to Firebase hosting, while with Google Chrome or Chromium everything works without any additional configuration.

Firefox Response Headers:

header	value
access-control-allow-origin:	*
cache-control:	max-age=3600
content-encoding:	br
content-type:	application/octet-stream
etag:	"7c0cc1d15fb812f73b7413a191d6f415e5de1ea9be47d1409a900917e79053fe-br"
last-modified:	Mon, 25 Mar 2024 11:49:30 GMT
strict-transport-security:	max-age=31556926; includeSubDomains; preload
accept-ranges:	bytes
content-range:	bytes 0-16383/9147232
date:	Mon, 25 Mar 2024 13:03:03 GMT
x-served-by:	cache-lhr7333-LHR
x-cache:	HIT
x-cache-hits:	0
x-timer:	S1711371783.071773,VS0,VE1
vary:	x-fh-requested-host, accept-encoding
alt-svc:	h3=":443";ma=86400,h3-29=":443";ma=86400,h3-27=":443";ma=86400
content-length:	16384
X-Firefox-Spdy:	h2

Chrome Response Headers:

header	value
Accept-Ranges:	bytes
Access-Control-Allow-Origin:	*
Alt-Svc:	h3=":443";ma=86400,h3-29=":443";ma=86400,h3-27=":443";ma=86400
Cache-Control:	max-age=3600
Content-Length:	16384
Content-Range:	bytes 0-16383/9167043
Content-Type:	application/octet-stream
Date:	Mon, 25 Mar 2024 13:06:46 GMT
Etag:	"7c0cc1d15fb812f73b7413a191d6f415e5de1ea9be47d1409a900917e79053fe"
Last-Modified:	Mon, 25 Mar 2024 11:49:30 GMT
Strict-Transport-Security:	max-age=31556926; includeSubDomains; preload
Vary:	x-fh-requested-host, accept-encoding
X-Cache:	HIT
X-Cache-Hits:	0
X-Served-By:	cache-lhr7372-LHR
X-Timer:	S1711372007.888898,VS0,VE1

It seems that I am missing something but changing the response headers in Firebase to different values did not fix the problem:

// firebase.json
{
  "hosting": {
    "target": "atlas-prototype",
    "public": "dist/spa",
    "ignore": ["firebase.json", "**/.*", "**/node_modules/**"],
    "headers": [
      {
        "source": "**/*.pmtiles",
        "headers": [
          {
            "key": "Content-Type",
            "value": "application/octet-stream"
          },
          {
            "key": "Access-Control-Allow-Origin",
            "value": "*"
          }
        ]
      }
    ]
  }
}

[bdon]

Can you reproduce this on storage that isn’t firebase, like Amazon s3?

[rmcf]

No, I cannot. It seems that in my case problem is only with Firebase web hosting. I have deployed the same web application to Storj cloud (AWS S3 compatible) and everything is fine there for both web browsers.

[bdon]

Thanks for reporting, this is worth digging into. Since I haven't used Firebase, can you provide a brief explanation as to how we can reproduce this? Are you embedding the pmtiles into a firebase application bundle or using a storage product?

[rmcf]

Sure. I have a single page application developed with Vue.js (Quasar framework) where in Map component I read the .pmtiles file with MapLibre library accordingly to official documentation. File .pmtiles is located just in public folder of the application. To deploy this application to Firebase hosting I used Firebase CLI and GitHub Actions to configure CI/CD. After successful deployment that problem appeared in Firefox. Should I try Firebase storage for storing .pmtiles files?

[bdon]

Yes, if you can please try it with Firebase Storage that would be helpful. It's possible that including it as a static file in the Firebase app is transforming it somehow but it doesn't explain why Firefox would behave differently (I can also reproduce).

[rmcf]

@bdon thank you for advice! That solved my problem. Now everything works fine: https://atlas-prototype.web.app/#/districts in both web browsers: Firefox and Chrome.

The solution in my case was:

1. upload reg_ua.pmtiles file to Firebase Storage
2. create reference for public access to .pmtiles file with getDownloadURL() method
3. use received URL as a source of vector layer in MapLibre map:

sources: {
  uatiles: {
    type: "vector",
    url: "pmtiles://https://firebasestorage.googleapis.com/v0/b/kinburn-land.appspot.com/o/vector-tiles%2Freg_ua.pmtiles?alt=media&token=3aadf1ee-df47-4e50-8006-660e08b0145e",
  }
}

Important! This part of URL ?alt=media&token=3aadf1ee-df47-4e50-8006-660e08b0145e is necessary
4. deploy web-application to Firebase Hosting

Also, we should keep in mind settings of CORS, AppCheck and Rules while working with Firebase Storage

[gvkhna]

@daniel-j-h

Quick correction: range request caching in Chrome works fine except in incognito windows (most likely by design).

I just re-tested this in a normal Chrome window and it works indeed. Let me change the title to reflect that.

I'm encountering this issue in chrome, not able to cache range requests. The example you originally linked to is now gone. What's the status of this issue. I'm curious if it's now generally working or there are some configuration settings that need to be made for it work.

For ref, serving pmtiles from cloudflare r2 bucket, it is returning etag. I also tried changing the content type from application/octet-stream to application/x-protobuf but that didn't seem to make a difference.

[bdon]

Caching is disabled completely on Chromium-based browsers because of this bug: #442