How to Exclude URLs from Screaming Frog crawl
When crawling a big site with screaming frog sometimes the results are too overwhelming due to the sheer volume. As the devil is in the details and the proof is in the pudding it can be really helpful to exclude some of the unrequired urls. This tutorial will show you how you can exclude urls from the screaming frog crawl so you can return only the most relevant urls related to your goal.
Exclude urls from crawl
go to configuration > exclude
exclude word from the crawl
this regex will exclude urls containing ?_pos
1
.*_pos.*
exclude a paramater
1
2
.*\?filter=.*
exclude directory from crawl
1
https://example.com/exclude/.*
use cases
you can exclude duplicate urls. such as for Ecommerce sites where urls containing query paramaters for size and color etc
This post is licensed under
CC BY 4.0
by the author.