Administration

Contents:

• Administration interface
• 🌐 Crawl a new URL
• ✔ Crawl queue
• 🕷 Crawlers
• 📊 Analytics
• ⚡ Crawl Policies
  • Policy matching
  • ⚡ Crawl
  • 🌍 Browser
  • 🔖 Archive
  • 🕑 Recurrence
  • 🔒 Authentication
• Recursive crawling
  • No limit recursion
  • Limited recursion
  • Partial limited recursion
• Atom and RSS feeds
  • Caching for news aggregators
• Documents
  • Status
  • Error
  • Crawl DT
  • Recursion remaining
  • Rejected by robots.txt
  • Too many redirects
  • Show on homepage
  • Hidden
• Domain Settings
  • Browse mode
  • Ignore robots.txt
  • Robots.txt status
  • Robots.txt allow/disallow rules
• Cookies
  • Cookies import
• Excluded URLs
• External Search Engines
• Permissions
  • Crawl Permissions
  • Search Permissions