robots.txt Generator

Disallow Is a Request, Not a Block

Most accidents involving robots.txt happen because of two common misunderstandings. You need to keep them separate.

First, `Disallow` is a request to "please don't crawl this path," not "please remove this from search results." A crawler won't read the page, but if another site links to that URL, Google can still show it in search results with just the URL and no content. This is what Search Console reports as "Blocked by robots.txt, but indexed."

Second, this is where the real trouble starts. To get a page out of search, you add a `noindex` meta tag to it, and then you also block it with `robots.txt`. The crawler can't read the page, so it can't read the `noindex` tag. As a result, the page is never removed from the index. You should never use `noindex` and `Disallow` together.

Blocking AI Crawlers: What to Block and What to Allow

Blocking all AI bots isn't always the right answer. Bots from the same company often have different roles. The most common strategy is to block data collection for training while allowing bots used for search and citations.

robots.txt Is Not Enforceable

robots.txt is a gentlemen's agreement. It's up to the bot to honor it. While major companies (OpenAI, Anthropic, Google, Naver) have stated their crawlers respect the rules, many bots don't read the file at all, spoof their User-Agent, or change their names frequently. To truly block a bot that keeps showing up in your access logs, you have to block it at the server level.

# nginx — How to actually block bots that ignore robots.txt
# Using 444 (closes connection without response) also saves bandwidth.
map $http_user_agent $bad_bot {
    default 0;
    ~*(GPTBot|CCBot|Bytespider|AhrefsBot|SemrushBot) 1;
}

server {
    # ... snip ...
    if ($bad_bot) { return 444; }
}

# Test config before applying → Reload for zero-downtime update
#   nginx -t && systemctl reload nginx

Where to Upload and How to Verify

Even with perfect syntax, your robots.txt is useless if it's in the wrong place. It must be at the root of your domain, and each subdomain needs its own file.

1. After uploading the downloaded file to your site's root, open https://yourdomain/robots.txt directly in your browser. If you see a 404 error or an HTML page, it's in the wrong place.
2. Check the robots.txt report in Google Search Console to see the latest version Google has read and any errors it found.
3. Use the robots.txt validation tool in Naver Search Advisor to check against Yeti's standards as well.

• https://example.com/blog/robots.txt → Is ignored. Placing it in a subfolder has no effect.
• https://blog.example.com/ → Is not covered by example.com's robots.txt. You must upload a separate file for each subdomain.
• The filename must be exactly `robots.txt` (lowercase). `http` and `https` are also treated as separate sites, so if you serve both, the file should be accessible on both.