titanz/NGINX-Configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update robots.txt

Browse Source 
Matches bebffccc0c/robots.txt

Signed-off-by: Tommy <contact@tommytran.io>
This commit is contained in:
Tommy 2025-02-10 04:59:59 -07:00
parent d28691ceed
commit 0f0305614b
No known key found for this signature in database
GPG Key ID: 555C902A34EC968F
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
srv/nginx/robots.txt
View File

@ -32,6 +32,8 @@ User-agent: ChatGPT-User
User-agent: Claude-Web User-agent: Claude-Web
User-agent: ClaudeBot User-agent: ClaudeBot
User-agent: cohere-ai User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Crawlspace
User-agent: Diffbot User-agent: Diffbot
User-agent: DuckAssistBot User-agent: DuckAssistBot
User-agent: FacebookBot User-agent: FacebookBot
@ -56,6 +58,8 @@ User-agent: PanguBot
User-agent: PerplexityBot User-agent: PerplexityBot
User-agent: PetalBot User-agent: PetalBot
User-agent: Scrapy User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Sidetrade indexer bot User-agent: Sidetrade indexer bot
User-agent: Timpibot User-agent: Timpibot
User-agent: VelenPublicWebCrawler User-agent: VelenPublicWebCrawler