Robots.txt is a text file that gives crawlers instructions about which parts of a website they may visit. It is used to manage crawling, not as a security mechanism and not as a reliable way to hide content from search results.
What does robots.txt involve in practice?
In practice, robots.txt should be simple and precise. Incorrect rules can block important pages or waste crawl activity on URLs that should not be prioritised. Content that should not be public must be protected with access control, not only robots.txt. The sitemap address can also be included in the file.
Robots.txt is relevant to technical SEO, migrations and major changes to a website. It should be checked before publishing, especially if the platform has gained new URL patterns. See Google's explanation of robots.txt, Sitemap and Canonical tag.
Let's build something solid together
.avif)
We help ambitious businesses with branding, UI/UX and Webflow solutions that are thoughtful, clear and built to last.
