Question 1

What is a robots.txt file?

Accepted Answer

A plain text file at the root of your domain that tells search engine crawlers which pages they can and cannot visit. Most sites use it to block admin areas, duplicate content, and API routes from being indexed.

Question 2

Does robots.txt hide pages from Google?

Accepted Answer

No. Disallowing a URL stops Google from crawling it, but the page can still appear in search results if other sites link to it. To completely remove a page from search results, use a noindex meta tag or the URL removal tool in Google Search Console.

Question 3

Where do I put the robots.txt file?

Accepted Answer

It must live at the root of your domain, e.g. https://yourdomain.com/robots.txt. Most website builders and CMS platforms (WordPress, Shopify, Webflow) generate one automatically. For custom sites, upload it directly to your public root directory.

Question 4

What does User-agent: * mean?

Accepted Answer

The asterisk is a wildcard that matches all crawlers. Most robots.txt files start with User-agent: * to apply rules to every bot. You can also target specific crawlers by name, like Googlebot or Bingbot, if you want different rules for different search engines.

Question 5

Should I disallow my /api/ directory?

Accepted Answer

Generally yes. API endpoints rarely contain content worth indexing, and blocking them reduces unnecessary crawl budget usage. The same applies to /admin/, /login/, and any internal tooling routes.

Build a robots.txt file
in seconds

How to add robots.txt to your website

Frequently asked questions

Build a robots.txt filein seconds