The full review
What is Screaming Frog?
Screaming Frog is a UK based search marketing agency that also builds desktop software, and the product most people mean by the name is the Screaming Frog SEO Spider. It installs on Windows, macOS and Linux, then crawls sites like Googlebot, discovering hyperlinks in the HTML using a breadth first algorithm, and it uses a configurable hybrid storage engine that can save data to disk in order to crawl large websites. The vendor calls it an industry leading crawler for technical site audits and reports that it checks over 300 SEO issues.
Screaming Frog runs locally rather than as a hosted dashboard, and above the free tier's 500 URL cap the vendor states that the maximum number of URLs you can crawl is dependent on allocated memory and storage. The company also publishes a separate Log File Analyser.
What a Screaming Frog crawl surfaces
The core audit covers the technical checklist end to end: broken links, server errors, redirect chains and loops, page titles and meta data, near duplicate content, and directives from robots.txt, meta robots and the X-Robots-Tag header. Hreflang auditing, AMP validation, structured data validation and spelling checks across 25 plus languages sit in the same window. The SEO Spider renders JavaScript too, so Angular, React and Vue.js builds are crawled as a browser sees them, with WCAG accessibility auditing, sitemap generation and architecture visualisations included.
Bring your own key AI prompts
Under Config > API Access > AI, Screaming Frog connects to OpenAI, Gemini and Anthropic, to local frameworks including Ollama and LM Studio, and to custom endpoints covering DeepSeek, Grok, Azure OpenAI and OpenRouter. You supply the API key, so inference cost sits with your provider account rather than inside a subscription.
The Prompt Configuration tab supports up to 100 prompts, with an Add from Library shortcut for presets and a tester that previews the model response. Results land in a dedicated AI tab and join the Internal tab beside the rest of the crawl, which makes bulk classification or alt text drafting workable across thousands of URLs.
Semantic similarity and the Content Cluster Diagram
Enabling Config > Content > Embeddings turns that same AI connection into a semantic analysis layer. Screaming Frog captures vector embeddings per page and scores Semantic Similarity from 0 to 1, flagging pages above 0.95 as semantically similar by default, while a Semantic Relevance Score measures how far a page drifts from the average theme of the site. The Content Cluster Diagram, under Visualisations, plots those embeddings so related pages group together and outliers sit at the edges, which helps with cannibalisation reviews and redirect mapping.
Automation, extraction and API integrations
Recurring work runs through scheduled crawls with automatic exports, and through crawl comparison, which diffs one crawl against another or staging against production. Custom extraction using XPath, CSS Path and regex pulls any field off a page, and custom JavaScript snippets extend that to actions such as triggering mouseover events. Google Analytics, Google Search Console and PageSpeed Insights connect natively, and link metrics can be pulled from Majestic, Ahrefs and Moz.
Free Version versus a paid licence
The feature split is unusually generous. The Free Version includes JavaScript rendering, crawl comparison, structured data validation, custom extraction, AI integration and the Google API connections, and caps you at a 500 URL crawl. The Paid Version lifts that ceiling to unlimited crawling, bounded in practice by allocated memory and storage. Licences run for one year, are issued per user, and carry volume bands from 1 to 4 licences up to 20 or more.
Who should choose Screaming Frog
Screaming Frog is the right default for technical SEOs, migration leads and agency teams who need to interrogate a site rather than read a summary of one. If your work involves redirect mapping, structured data QA or bulk content classification, the vendor documents a workflow for each: crawling old and new sites together and adjusting the semantic similarity threshold to build a redirect list, validating structured data against Schema.org specifications, and configuring up to 100 AI prompts against crawl data. It is a poor fit for a generalist wanting an always on hosted dashboard with rank tracking and automated stakeholder reports, since this remains a per user desktop application with a real learning curve.
Strengths
- Best-in-class technical crawler — finds 300+ SEO issues with granular, expert-level control
- Flat £199/year per seat with unlimited crawling; exceptional value versus cloud SaaS suites
- AI integration runs OpenAI, Gemini, Anthropic Claude or local Ollama prompts across every crawled page
- v22 semantic-similarity and embeddings surface near-duplicate and off-theme content, not just exact matches
Watch out for
- Desktop app: crawls run on your own machine and are bounded by local RAM and CPU
- Not a content-optimization tool — AI is bring-your-own-key prompting, not guided on-page editing
- Dense, technical UI with a genuinely steep learning curve
- AI API usage is billed separately by the provider and adds up on large crawls
Key features
| Feature | What it does |
|---|
| Deep technical crawler | Crawls up to millions of URLs, flagging broken links, redirects, duplicate content, metadata issues and 300+ problems. |
| AI prompt integration | Connect OpenAI, Gemini, Anthropic Claude or local Ollama to run custom prompts against page content while crawling. |
| Semantic similarity & embeddings | v22 generates vector embeddings to detect near-duplicate, low-relevance or off-theme pages beyond exact matches. |
| Data-source integrations | Pulls Google Analytics, Search Console and PageSpeed Insights into crawls; renders JavaScript sites via Chromium. |
How Screaming Frog compares
| Alternative | How it differs |
|---|
| Sitebulb | Closest desktop/cloud crawler rival; prioritised audit hints and cleaner reports, but pricier per seat. |
| Surfer SEO | True AI content-optimization tool for on-page scoring — complements rather than replaces Screaming Frog's crawling. |
| Semrush | All-in-one cloud suite with Site Audit plus AI writing; broader coverage but far more expensive. |
Frequently asked questions
Is Screaming Frog free?
Yes — the free edition crawls up to 500 URLs per crawl with core features. Unlimited crawling and advanced features (JavaScript config, scheduling, custom extraction, API and AI access) require the £199/year licence.
Does Screaming Frog use AI?
It has no built-in AI model. Instead it connects to OpenAI, Gemini, Anthropic Claude or local Ollama via your own API key, letting you run custom prompts and semantic analysis against crawl data. Third-party API usage is billed separately by the provider.
Screaming Frog ratings on other platforms
Independent user ratings from third-party review sites, linked here for transparency.
These are not our editorial score, are captured on the date shown, and may have changed since.