Skip to content

Commit

Permalink
Merge pull request #43 from lxjv/main
Browse files Browse the repository at this point in the history
Update robots.json with Claude respect link
  • Loading branch information
glyn authored Oct 8, 2024
2 parents 6da804e + dc15afe commit 01993b9
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions robots.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
"frequency": "No information. provided.",
"function": "Scrapes data to train Anthropic's AI products.",
"operator": "[Anthropic](https://www.anthropic.com)",
"respect": "Unclear at this time."
"respect": "[Yes](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)"
},
"Diffbot": {
"description": "Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.",
Expand Down Expand Up @@ -279,4 +279,4 @@
"operator": "[Webz.io](https://webz.io/)",
"respect": "[Yes](https://web.archive.org/web/20170704003301/http://omgili.com/Crawler.html)"
}
}
}

0 comments on commit 01993b9

Please sign in to comment.