How to Exclude URLs from Screaming Frog crawl

When crawling a big site with screaming frog sometimes the results are too overwhelming due to the sheer volume. As the devil is in the details and the proof is in the pudding it can be really helpful to exclude some of the unrequired urls. This tutorial will show you how you can exclude urls from the screaming frog crawl so you can return only the most relevant urls related to your goal.

Exclude urls from crawl

go to configuration > exclude

exclude word from the crawl

this regex will exclude urls containing ?_pos

.*_pos.*

exclude a paramater

.*\?filter=.*

exclude directory from crawl

https://example.com/exclude/.*

use cases

you can exclude duplicate urls. such as for Ecommerce sites where urls containing query paramaters for size and color etc