Crawling

SEO

Robots.txt Basics for SEO and Crawl Control

Understand what robots.txt can do, what it cannot do, and how to prepare clean crawl directives.