Screaming Frog’s advantage over URL Profiler is that, once it reaches the resource limitation, it will ask you if you’d like to save you crawl and then keep going. A desktop crawl is inherently a resource limited crawl this is why it’s prone to crash or run out of memory when it crawls too many pages. If your machine has 4 CPU cores, 8 GB of RAM, you’re running Windows 8, have 50 tabs open in Chrome and have a bunch of TSRs running, the Frog is very likely to actually be screaming in pain while it’s crawling for you. The diagram below from an eBay patent gives a visual representation on of how a cloud-based distributed crawling system typically works.Ĭonversely, desktop crawlers are limited by the specs of your computer and they run in memory.

In summation, cloud-based crawlers can be distributed, faster and more resistant to failure. Finally, cloud-based crawlers have a virtually infinite set of computing resources to pull from to facilitate the crawl. Also cloud-based crawlers are saving their crawl data to a database so memory overhead can be kept very low. Each of these nodes runs a small application managed by another centralized application that makes the crawling fault-tolerant. Typically, cloud-based crawlers operate using a series of nodes that distribute the crawl. Technologically, cloud-based crawlers have a distinct advantage over desktop crawlers. Nonetheless, it’s more important to do what it takes to do work that we’re PROUD of than to use the most sophisticated tool.Īll that said, how many times have you been frustrated by this dialog box? I’m well aware that this is more a reflection of how well I know these products than the shortcomings of the other products. Also one of our team members has built to bring headless browsing features to Screaming Frog, so that is an added incentive for us to make it work. However, despite how awesome the SaaS crawlers are, I always feel like I “know” a website better when I do a Screaming Frog or URLProfiler crawl. sample-based) content audit on millions of pages, we’d typically use DeepCrawl then batches of 50k URLs in URLProfiler.

For example, if we’re doing a population (vs. Although, I’d encourage you to check out Moz’s new content auditing tool as well.įrom what I know of each of these tools is that they all have their own strengths, weaknesses and use cases.

URL Profiler though has planted itself as the go to tool for our content auditing process. Now I’m also getting to know as well and I encourage you to give their free trial a spin. I also love a lot of the other crawler tools that have popped up in its wake like DeepCrawl and URLProfiler. I also very much appreciate Dan Sharp and his team’s continued focus on innovation and improvement with the tool as well. One would be hard-pressed to find a finite number of use cases for the tool. I’ve been a huge fan of Screaming Frog SEO Spider for a number of years now.

Screaming frog database