From Crawl Policies to Collections - Major Change in Sosse 1.14
The upcoming Sosse 1.14 release marks a significant evolution in how web crawling is configured and managed. We’re moving from the complex “Crawl Policy” system to a more intuitive Collections approach that puts user experience first.
⚠️ Important: We strongly recommend backing up your data before upgrading to Sosse 1.14.
Why the Change? The old Crawl Policy system, while powerful, had several limitations that made it challenging for users:
Complex matching logic: URLs were matched against policy patterns with unintuitive recursion rules Single context per URL: Each URL could only belong to one policy, limiting flexibility Unintuitive recursion logic: The depth and recursion rules were hard to understand and configure Performance overhead: Complex pattern matching for every URL discovery Introducing Collections Collections represent a fundamental change in approach.