Skip to content

Commit

Permalink
Update robots.json with Claude respect link
Browse files Browse the repository at this point in the history
  • Loading branch information
lxjv authored Oct 7, 2024
1 parent 6da804e commit dc15afe
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions robots.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
"frequency": "No information. provided.",
"function": "Scrapes data to train Anthropic's AI products.",
"operator": "[Anthropic](https://www.anthropic.com)",
"respect": "Unclear at this time."
"respect": "[Yes](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)"
},
"Diffbot": {
"description": "Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.",
Expand Down Expand Up @@ -279,4 +279,4 @@
"operator": "[Webz.io](https://webz.io/)",
"respect": "[Yes](https://web.archive.org/web/20170704003301/http://omgili.com/Crawler.html)"
}
}
}

0 comments on commit dc15afe

Please sign in to comment.