Adding URLs to the crawl queue#

In the conf_menu_button menu, or in the Administration interface, by clicking 🌐 Crawl a new URL you can queue an URL that will be crawled when a worker is available.

../_images/crawl_new_url.png

By default, all links found in the page will be visited and the crawlers will recurse until all pages are discovered. Submitting redirects to the Crawl status page.

How pages are indexed and which pages to recurse into is defined by Crawl Policies.