Screaming Frog Cheatsheet

** Prerequisites **

Screaming Frog installed on your Computer (there is a free version) Screaming Frog is a desktop application you can download to your computer

Find lorem ipsum text

Screaming frog has a built in feature for this.

• Crawl site
• Go to the content tab
• Filter by lorum ipsum placeholder

Customising a Screaming Frog Crawl

Exclude urls from crawl

When crawling a big site with screaming frog sometimes the results are too overwhelming due to the sheer volume. As the devil is in the details and the proof is in the pudding it can be really helpful to exclude some of the unwanted urls.

This tutorial will show you how you can exclude urls from the screaming frog crawl so you can return only the most relevant urls related to your goal.

Go to configuration > exclude

Exclude a word from the crawl

This regex will exclude urls containing ?_pos

.*_pos.*

Exclude a parameter

.*\?filter=.*

Exclude directory from crawl

https://example.com/exclude/.*

Search for strings

It’s such a useful ability to be able to search an entire website and return all the urls that contain an occurrence of a specific word or string or phrase.

Use Screaming Frog to find all occurences of some HTML or String

1. Go to configuration > custom > search
2. Enter search string (you can search for more then 1 string)
3. In the search crawl results find the custom search column relating to your search

Case sensitivity

By default the search is not case sensitive so if you search for the word lawyers it will also pick up the word Lawyers & LAWYERS

Crawl a Password Protected Site

If the site has a browser password protection that’s no problem for screaming frog. Screaming frog will ask you to enter the username and password of the website before crawling to site.

If you cannot see the text on your website but Screaming Frog can?

If you cannot see the text on your website but Screaming Frog can. The text is hidden. The text could be hidden in a slideshow, accordion or tabs. View html source of a website in your browser and search for the string in the html code. You will see it now.

Fix 404’s with Screaming Frog.

Overall, fixing 404 errors is essential for maintaining a healthy website that provides a positive user experience, preserves SEO performance, and builds trust with visitors and search engines alike.

How to find the 404 errors

This is arguably the hardest part about 404 errors and its the heart of this tutorial.

1. Run the crawl on your site url
2. Filter by status code until 404 descend from the top
3. Select any of the 404’s from the returned results.
4. Look at the footer menu of Screaming Frog for inlinks
5. Right click the url from the from column
6. Hit open from in browser
7. Search for the anchor text of the broken link by doing a “command f” in your browser window

find 404’s with Screaming Frog Crawl

If you can’t replicate the broken link in Screaming Frog

1. If you cant replicate the link it could be that you are logged in (And Screaming frog is not) and therefore cant replicate the error. Test the broken link in a non-logged in state.
2. Try search the html source code if the search for the string on the website frontend returns 0 correct results

Crawl No-index site with Screaming Frog.

What is the no index / no follow site settings

The “noindex” and “nofollow” site settings are directives that can be implemented in the HTML code of a webpage to instruct search engines on how to handle the page and its links:

1. Noindex: This directive tells search engines not to index the content of the page. In other words, the page won’t appear in search engine results pages (SERPs). This is commonly used for pages that are not intended to be publicly visible or for duplicate content that shouldn’t be indexed.
2. Nofollow: This directive tells search engines not to follow the links on the page. It means that search engine crawlers won’t pass authority or PageRank to the linked pages, and those pages won’t benefit from being linked to from the “nofollow” page.

Combining both directives, noindex, nofollow is often used for pages that the website owner doesn’t want to appear in search results and doesn’t want to pass authority to other pages through links on that page.

These directives are typically implemented using meta tags in the HTML code of a webpage. For example:

<meta name="robots" content="noindex, nofollow">

Or they can be set through the robots.txt file or with HTTP response headers. They are useful for controlling how search engines interact with specific pages or sections of a website.

How to bypass the nofollow when crawling a site with Screaming Frog

1. Go to ‘Config > Spider’
2. Scroll down to “Crawl behaviour”
3. Enable Follow Internal Nofollow

How to bypass the noindex

1. Go to Config > Spider > Advanced.
2. Uncheck Ignore Non-Indexable URLs for Issues

Now you can Crawl a site set to noIndex NoFollow

How to bypass robots.txt

Bypass the nofollow When the nofollow has been added to the Robots file.

1. Go to ‘Config > Robots.txt > Settings’
2. Choose ‘Ignore robots.txt’.

Disable crawling javascript enabled pages

To disable JavaScript crawling in Screaming Frog, follow these steps:

Open Screaming Frog SEO Spider: Launch the application.

Go to Configuration:

From the top menu, click on Configuration.

Select Spider Settings:

In the drop-down menu, go to Spider….

Disable JavaScript:

In the Spider Configuration window, uncheck the option ‘Render: JavaScript’ under the Rendering tab. This ensures that Screaming Frog won’t crawl the JavaScript-rendered versions of your pages, and it will focus only on the raw HTML.

Click OK: Save your settings by clicking OK.

By disabling JavaScript crawling, Screaming Frog will now crawl your site in its default HTML-only mode.

Find all images missing alt text

• Run crawl
• Select images tab from the top panel
• Filter by missing alt text