🌐 Crawl a new URL

In the conf_menu_button menu, or in the Administration interface, by clicking 🌐 Crawl a new URL you can queue one or multiple URLs to be crawled when a worker is available.

../_images/crawl_new_url.png

By default, only the URLs queued for crawling will be visited. The crawler will not recurse into discovered links unless explicitly configured.

To control how pages are indexed and whether recursion occurs, update the relevant settings in ⚡ Crawl Policies.

After submitting a URL, the next page shows the ✔ Crawl queue.