Administration# Contents: Administration interface Adding URLs to the crawl queue Crawl status Crawl Policies Policy matching Indexing decision Browser HTML snapshot Updates Authentication Recursive crawling No limit recursion Limited recursion Partial limited recursion Archiving Text cache HTML snapshots Page screenshots Browsable home Authentication handling Submitting forms Executing javascript Cookie Edition Documents Status Error Crawl DT Recursion remaining Rejected by robots.txt Too many redirects Show on homepage Domain Settings Browse mode Ignore robots Robots.txt status Robots.txt allow/disallow rules Cookies Excluded URLs External Search Engines User management