Skip to content

Releases: ai-robots-txt/ai.robots.txt

v1.20: adds PanguBot

02 Dec 23:57
24666e8
Compare
Choose a tag to compare

What's Changed

  • Clarify our rationale by @glyn in #54
  • Create workflow to upload robots.txt file as release artefact by @norwd in #55
  • Allow facebookexternalhit by @glyn in #56
  • Restore attribution by @fabianegli in #58

New Contributors

  • @norwd made their first contribution in #55

Full Changelog: v1.19...v1.20

v1.19: adds DuckAssistBot, improves automation

08 Nov 22:54
Compare
Choose a tag to compare

What's Changed

  • Update robots.json with Claude respect link by @lxjv in #43
  • Sort the content of robots.json by keys, case-insensitively by @isagalaev in #46
  • Update the FAQ by @glyn in #45
  • log the diff in the update actions by @fabianegli in #47
  • PHP diagnostics by @glyn in #49
  • Fix typo and trigger rerun of main job by @glyn in #50
  • PHP to Python plus tests and stuff by @fabianegli in #51

New Contributors

Full Changelog: v1.18...v1.19

v1.18: adds ISSCyberRiskCrawler

01 Oct 00:38
Compare
Choose a tag to compare

Adds ISSCyberRiskCrawler. Thanks to Brian at Skewray Research for calling this one out!

v1.17: add Sidetrade indexer bot + other updates

28 Sep 21:03
Compare
Choose a tag to compare

What's Changed

  • Add Sidetrade indexer bot (Thank you to Brian at Skewray Research for calling this one out.)
  • Add instructions for AI bot blocking on Vercel by @cramforce in #36
  • Update README.md by @cityrolr in #41
  • feat: make CCBot entry more accurate by @wumpus in #42
  • Add an additional resource - README.md by @urvish-p80 in #38

New Contributors

Full Changelog: v1.16...v1.17

v1.16: adds iaskspider/2.0

07 Sep 02:06
Compare
Choose a tag to compare

Thank you to Brian at Skewray Research for calling this one out.

What's Changed

Full Changelog: v1.15...v1.16

v1.15: adds AI2Bot

29 Aug 03:09
Compare
Choose a tag to compare

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.14...v1.15

v1.14: adds Webzio-Extended

27 Aug 22:42
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.13...v1.14

v1.13: corrects agent name

04 Aug 21:54
Compare
Choose a tag to compare

What's Changed

  • Fix Imagesift user agent by @jsheard in #27
  • Add Cloudflares first-party scraper blocking to FAQ by @jsheard in #28

New Contributors

Full Changelog: v1.12...v1.13

v1.12: adds ICC-Crawler

04 Aug 19:30
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.11...v1.12

v1.11: adds PetalBot and facebookexternalhit

01 Aug 14:55
6c596a5
Compare
Choose a tag to compare

What's Changed

Full Changelog: v1.10...v1.11