Recursive crawling#
SOSSE can crawl recursively all pages it finds, or the recursion level can be limited when crawling large websites or public sites.
No limit recursion#
Recursing with no limit is achieved by using a policy with a Recursion set to Crawl all pages (the default).
For example, a full domain can extracted with 2 policies:
A policy for the domain with a
URL regexthat matches the domain, andRecursionset toCrawl all pagesA default policy (with the
URL regexset to.*) with aRecursionset toNever crawl
Limited recursion#
Crawling pages up to a certain level can be simply achieved by setting the Recursion to Depending on depth and setting the Recursion depth when queueing the initial URL.
Partial limited recursion#
A mixed approach is also possible, by setting a Recursion to Depending on depth in one policy, and setting it to Crawl all pages in an other and a positive Recursion depth.
For example, one could crawl all Wikipedia, and crawl external links up to 2 levels with the following policies:
A policy for Wikipedia, with
Recursion depthof 2:
A default policy with a
Depending on depthcondition:
