Adding URLs to the crawl queue#
In the menu, or in the Administration interface, by clicking 🌐 Crawl a new URL
you can queue an URL that will be crawled when a worker is available.
By default, all links found in the page will be visited and the crawlers will recurse until all pages are discovered. Submitting redirects to the Crawl status page.
How pages are indexed and which pages to recurse into is defined by Crawl Policies.