Screaming Frog is a program designed for SEO analysis of websites by SEO specialists. However, part of its functionality significantly helps webmasters and content editors optimize their work with pages. Personally, I have been using this tool since 2020, and in my previous job, I used it daily. Even a small portion of the program’s functionality was sufficient for me to check websites before handing them over to QA.
This tool can speed up page analysis when you first receive them, helping you understand what needs to be worked on. Similarly, it accelerates the analysis of pages you have already finished working on. This allows you to quickly identify and fix errors in your work. Especially when dealing with large sitemaps, manually checking everything can be very challenging, making programmatic checks an optimal solution for saving time and improving the accuracy of the review.
What is Screaming Frog?
As I already mentioned, this is a tool for SEO specialists for technical website analysis. The working principle is as follows: you upload all your data for analysis and click the start button. Data for analysis can be uploaded in various ways:
1) Spider Mode
This is the default mode for the SEO Spider. In this mode, SEO Spider scans the website, collecting links and categorizing URLs into various tabs and filters. Simply enter the URL of your choice and click “Start.”
2) SERP Mode
In this mode, you can upload page titles and meta descriptions directly into SEO Spider to calculate pixel width (and character length!). No crawling is involved in this mode, so the pages don’t need to be on a website.
3) List Mode
In this mode, you can check a predefined list of URLs. This list can come from various sources—by simple copy-pasting or a file like .txt, .xls, .xlsx, .csv, or .xml. Files will be scanned for URLs with the prefix http:// or https://, while all other text will be ignored. If you are migrating content and want to check URLs, I recommend using the “always follow redirects” configuration so SEO Spider identifies the final destination URL. List Mode changes the crawl depth setting to zero, meaning only the uploaded URLs will be checked. If you want to check links from these URLs, set the crawl depth to 1 or more in the Limits tab under Configuration > Spider. This setting is only available in the licensed version, as shown in the screenshot:
In my work, I only use two scanning modes—Spider Mode and List Mode—as they are the most suitable for daily tasks. Screaming Frog is easy to use, making it ideal for non-technical specialists. It allows focusing on important aspects without deep SEO knowledge.
Key Features of Screaming Frog for Content Editors and Webmasters
1) Error Detection:
Identifying 404 errors, redirects, and understanding how these affect users and SEO.
2) Meta Tag Analysis:
Checking titles (Title) and descriptions (Meta Description) for length and presence of keywords.
3) Link Analysis (Internal & External Links):
Identifying broken links and orphan pages (pages without internal links).
4) Regular Expressions (Regex):
Searching for code snippets or text. This simplifies checking content or structural elements. Learn more about Regex in this article.
5) Filters:
Quickly finding the necessary pages by content type or status.
Examples of Using Screaming Frog in Daily Webmaster Routines
No matter what you need to check, the first step is uploading the page URLs that need to be scanned into the program. Screaming Frog scans and analyzes numerous page parameters by default: meta tags, status codes, presence and status codes of links on pages, word count, image sizes, presence of alt text, the number of h1/h2 headings, and much more.
Example 1: Identifying Pages with Embedded YouTube Videos
Custom search is available only in the licensed version
In this case, it’s best to use the Spider mode, as the program needs to scan all pages of the website. Starting from the homepage URL, the spider follows all accessible links and generates a list of site pages:
Without additional settings, the program won’t know that we are looking for something specific on the pages. To filter pages containing embedded YouTube videos, we create a custom search:
1) In the top menu, select Configuration → Custom → Custom search
2) Click the Add button and fill in the parameters:
Fill in the fields for search name, contains/does not contain, search method (text/regex), and what exactly we’re searching for. In our case, we need a unique class or ID of the container that only includes the desired element (present only once across the site). In the last field, choose HTML, as we are searching within HTML tags, not text.
Once all custom search parameters are filled, click OK. Navigate to your list of pages, and in the upper right corner, click the dropdown arrow and select Custom Search. Afterward, a new filter option will appear in the upper left filter panel—the custom one you created earlier. Select your custom search, and only the pages containing YouTube videos will be filtered in the column with URLs.
Example 2: Checking Technical SEO on Specific Pages
This check is available in the free version
This is the most important and also the most common check for pages. It is used by many specialists: SEO experts, webmasters, web developers, content editors, content managers, and even authors and website owners. Learn more about the technical SEO checklist for pages in this article. The good news is that Screaming Frog allows you to perform such technical SEO checks with its free version.
In this case, we can use two scanning modes:
Spider Mode – if you need to analyze all the pages of a website that have internal links.
List Mode – if you have a specific set of pages that need to be analyzed, simply input their URLs to avoid unnecessary information in the scan results table.
Example 3: Important basic parameters to pay attention to in scan results (highlighted in red):
This check is available in the free version
What the highlighted tabs show:
Response Code
The “Response Code” tab displays the HTTP response codes returned by pages during the scan. These can include codes like 200 (OK), 301 (Moved Permanently), 404 (Not Found), 500 (Server Error), etc.
Why this is important: Response codes indicate the availability status of pages. They help identify errors, such as broken links or incorrect redirects, which can negatively impact SEO and user experience.
Page Title
The “Page Title” tab shows the titles of pages, which are the text within the <title> tags.
Why this is important: Page titles are critical for SEO as they appear in search results and influence click-through rates. They should be unique, relevant, and have an optimal length.
Page Description
This tab displays the meta descriptions of pages, found in the <meta name="description"> tags.
Why this is important: Meta descriptions affect how pages are displayed in search results. They can boost click-through rates if they are appealing and relevant to the user’s query.
h1
The “h1” tab displays the content of <h1> tags on pages.
Why this is important: H1 tags signal the main topic of a page to search engines. They should be unique and reflect the primary headline of the content, helping both users and search engines better understand the page.
Content
This tab provides information about the content on pages, including word count, content type, and file size.
Why this is important: Content analysis helps identify pages with insufficient or duplicate content, which can negatively affect SEO. High-quality, relevant content supports better search engine rankings.
Links
The “Links” tab shows all internal and external links found on the site.
Why this is important: Link analysis helps identify broken or redirected links, optimize the internal link structure, and ensure a better user experience as well as efficient crawling by search engine bots.
Images
When working on improving a website’s performance, Screaming Frog can help identify factors like large images, uncompressed files, or unnecessary redirects. Using the “Images” tab, for example, you can detect oversized files and optimize them by reducing their size or converting them to a more efficient format like WebP. To check all important parametres you use the “Images” tab displays all images on the site, along with their attributes such as file size, dimensions, and alt text.
Why this is important: Optimizing images impacts page load speed and SEO. Including alt text improves accessibility and helps search engines understand the content of images.
In conclusion
Screaming Frog is a versatile tool not just for SEO specialists but also for webmasters and content editors. Its range of features allows for thorough website analysis, helping to identify errors, optimize performance, and ensure accessibility. By integrating it into your workflow, you can save time, improve accuracy, and deliver better results in less time.
