Ways to automatically update own robot.txt with latest update from here

[Andre601]

Would like to automate the process of updating my website's robot.txt based on latest changes made to this repository's robot.txt, but I'm not sure what a good aproach would be to make this as easy and effective.

Before suggesting anything, let me clear some stuff up:

• My website is open source, but its source is not on GitHub, but Codeberg
• I have access to Woodpecker-CI and Forgejo Actions (Similar to GitHub Actions but for Forgejo)
• I only have experience in Java and some very basic experience in Python and even less experience in Javascript, so you may need to explain things to me if it gets complicated.

Right now, what I would assume I could do is regularely pull the robots.txt and compare it with my own. If they differ, update the file and commit the change.
Tho, idk how good of an aproach this is, or if there could be issues.

[glyn]

Do you want to be sure to get the latest released version of robots.txt or just the current latest version on the main branch? The process you describe would appear to pull the latest version on the main branch. So, if a bug was introduced and later corrected before release, you could end up committing the buggy version.

[Andre601]

It wouldn't be much of an issue for me if that was the case. I would at most have it updated once a week, so I doubt the file would have typos or alike when it gets updated. Also, I would have a manual trigger too, so I could simply re-run it if necessary.

Tho, maybe it could be good to have it commit to a separate branch and make a PR... But the question would then be, how difficult this would be to set up with my current environment (Remember. No GitHub Actions here and the one I use does not guarantee compat with any GH Action).

[glyn]

See also: https://stackoverflow.com/questions/24987542/is-there-a-link-to-github-for-downloading-a-file-in-the-latest-release-of-a-repo

[Andre601]

Managed to setup the following Actions file:

name: "Update robots.txt"

on:
  schedule:
    - cron: "0 12 * * 0" # Once a week
  workflow_dispatch:

jobs:
  updateRobotsFile:
    runs-on: codeberg-tiny-lazy
    container:
      image: ghcr.io/catthehacker/ubuntu:act-latest
    steps:
      - name: "Checkout website repository"
        uses: https://code.forgejo.org/actions/checkout@v4
        with:
          token: ${{ secrets.BOT_TOKEN }}
          path: source
      - name: "Checkout ai.robots.txt Repository"
        run: "git clone https://github.com/ai-robots-txt/ai.robots.txt remote"
      - name: "Update file if different"
        run: |
          if git diff --no-index --exit-code "source/docs/robots.txt" "remote/robots.txt"; then
            echo "No difference found."
          else
            echo "Difference found. Updating file."
            cp "remote/robots.txt" "source/docs/robots.txt"
            cd source
            git config --global user.email "[email protected]"
            git config --global user.name "Andre601's Bot"
            git remote set-url origin https://${{ secrets.BOT_TOKEN }}@codeberg.org/Andre601/website
            git add docs/robots.txt
            git commit -m "Update robots.txt from ai.robots.txt"
            git push
            echo "Done!"
          fi

This seems to do the job for what I need.