GitHub Actions workflow to automatically update map tiles in Cloudflare R2

[shawaj]

I created a GitHub workflow that periodically (currently set to once every two months) downloads the latest tileset and uploads it to Cloudflare R2. It also updates the cloudflare worker to point to the new tileset after it is uploaded and then deletes the old tileset so you are not paying for extra storage of old tilesets unnecessarily...there should be essentially no downtime as the worker is only updated after the new tileset successfully uploads.

The only "gotchas" here are:

• you have to use the 8core runner available from GitHub Larger Runners service, because the disk space on the smaller ones is not big enough. The label labels: 8core-ubuntu is just what I have called the larger runner I set up
• you also may need to edit the r2 bucket name - I have called mine protomaps
• you also want to make sure that there is nothing else in this bucket other than files with filename structure YYYYMMDD/world.pmtiles
• you will need to have a fork of this repo (public or private) where you have added your own wrangler.toml at serverless/cloudflare/wrangler.toml and the below action (or copy the needed files to a different repo)

The cron is set to run once every two months on the 1st at 21:45 ... as it seems the tiles are usually updated in late morning at https://maps.protomaps.com/builds/ but sometimes there are some later timestamps. You can also trigger it manually with the workflow_dispatch

You need to set up a few environment variables in the repo as well:

• R2_API_ENDPOINT - this is just your Cloudflare account ID which is the long string after https://dash.cloudflare.com/ in the domain on your account
• R2_ACCESS_KEY_ID - this is the access key id as described here
• R2_SECRET_ACCESS_KEY - this is the secret access key as described here
• CLOUDFLARE_WORKERS_API_TOKEN - this is an API token created as described here. You can remove all permissions except for Account -> Workers Scripts -> Edit and make sure to scope to the account the worker is in
• CLOUDFLARE_ACCOUNT_ID - this is actually the same as the R2_API_ENDPOINT I just created another secret so it made more sense name wise

Happy to document it in more detail if you would like to add it to the repo (feel free to ping me), or you can just close this issue and have it available from google search :-)

Just thought it might be useful for some people in the future!

name: Update protomaps tiles on Cloudflare R2

on:
  schedule:
    - cron:  '45 21 1 */2 *'
  workflow_dispatch:

jobs:
  update-tiles:
    runs-on: 
      labels: 8core-ubuntu
    
    steps:
      - name: Set AWS credentials
        env:
          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
        run: |
          echo "Setting AWS Credentials"
          aws configure set aws_access_key_id "$R2_ACCESS_KEY_ID"
          aws configure set aws_secret_access_key "$R2_SECRET_ACCESS_KEY"
          aws configure set default.region auto

      - name: Download latest tileset
        run: |
          TODAY="$(date +'%Y%m%d')"
          echo "TODAY=$TODAY" >> $GITHUB_ENV
          echo "Today's date for filename is $TODAY"
          echo "Downloading protomaps pmtiles file."
          wget https://build.protomaps.com/"${TODAY}".pmtiles

      - name: Upload tileset to Cloudflare R2
        env:
          R2_API_ENDPOINT: ${{ secrets.R2_API_ENDPOINT }}
        run: |
          echo "Getting existing tileset filename"
          aws s3api list-objects-v2 --bucket protomaps --endpoint-url https://"$R2_API_ENDPOINT".r2.cloudflarestorage.com > output.json
          OLD_MAP_TILES=$(jq -r '.Contents[0].Key' output.json)
          echo "OLD_MAP_TILES=$OLD_MAP_TILES" >> $GITHUB_ENV
          echo "Uploading file to Cloudflare R2"
          aws s3 cp "${{ env.TODAY }}".pmtiles s3://protomaps/"${{ env.TODAY }}"/world.pmtiles --endpoint-url https://"$R2_API_ENDPOINT".r2.cloudflarestorage.com

      - name: Checkout repo
        uses: actions/checkout@v4
        with:
          sparse-checkout: |
            js
            serverless/cloudflare
            serverless/shared
          sparse-checkout-cone-mode: false

      - name: Update PMTILES_PATH environment variable in wrangler.toml
        run: |
          echo "Updating PMTILES_PATH environment variable in wrangler.toml"
          PMTILES_PATH="${{ env.TODAY }}/{name}.pmtiles"
          sed -i -E 's|PMTILES_PATH =.*|PMTILES_PATH = "'"$PMTILES_PATH"'"|g' serverless/cloudflare/wrangler.toml

      - name: Install dependencies
        run: |
          echo "Installing JS dependencies"
          cd js
          npm install
          echo "Installing cloudflare worker packages"
          cd ../serverless/cloudflare
          npm install

      - name: Build & Deploy Worker
        uses: cloudflare/wrangler-action@v3
        with:
          apiToken: ${{ secrets.CLOUDFLARE_WORKERS_API_TOKEN }}
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          workingDirectory: "serverless/cloudflare"

      - name: Push updated wrangler.toml to repo
        uses: test-room-7/action-update-file@v1
        with:
          file-path: serverless/cloudflare/wrangler.toml
          commit-msg: Update wrangler.toml to latest version
          github-token: ${{ secrets.GITHUB_TOKEN }}

      - name: Delete old map tiles file from Cloudflare R2
        env:
          R2_API_ENDPOINT: ${{ secrets.R2_API_ENDPOINT }}
        run: |
          echo "Deleting old map tiles from Cloudflare R2"
          aws s3 rm s3://protomaps/"${{ env.OLD_MAP_TILES }}" --endpoint-url https://"$R2_API_ENDPOINT".r2.cloudflarestorage.com

[bdon]

(moved to discussion) cool, thanks for sharing! I wonder if this could run as a Cloudflare Workers scheduled job instead of on GitHub Actions, that would let you eliminate needing the credentials.

[shawaj]

Sorry didn't see the discussions feature. Thanks for converting 👍

Oh that's a good idea. I didn't think about that. I'll have a go and see what I can come up with. I guess it would have to do some kind of streaming download to R2 but I guess that should be possible.

[shawaj]

Apologies, I played around with this but never got round to working this out sufficiently and then ran out of time