Can Screaming Frog SEO Spider be used as a Linkbot Pro replacement? SEO Spider is mature desktop PC software meant for SEO-focused webmasters that, in its free form will check 500 Web URLs at a time. Are they broken or moved, and if the latter then where are they now?

Yes, it can be used in this way, and here’s a short tutorial for absolute beginners who wouldn’t know the difference between SEO and a sea-otter.

1. Extract a plain-text list of Web URLs from your source page of bookmarks, or find the URL list you’ve already prepared and cleaned. Sobolsoft’s “Extract Data & Text From…” will extract without fuss from a saved HTML page and give you a one-per-line list, but there are also various bits of Windows freeware to do it.

2. Install Screaming Frog SEO Spider. Turn off: Top menu | Help | ‘Auto Check for Updates’.

3. Now whitelist SEO Spider in your Firewall, so the new software can access the Web…

C:\Program Files (x86)\Screaming Frog SEO Spider\jre\bin\javaw.exe
C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpider.exe
C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe

4. Set Screaming Frog SEO Spider user-agent (i.e. its internal Web browser) to emulate Chrome. Top menu: Configuration | User Agent | Chrome. This is apt as its main internal browser agent is actually Chrome, in the form of a headerless Chrome browser driven by SEO Spider. To a web server, SEO Spider now looks like a normal Chrome browser.

5. Enable Always Follow Redirects in SEO Spider, so we have the required redirect details in the results. Top menu | Configuration | User Agent, as seen here…

6. While you’re in that panel also ensure HSTS is off (otherwise you may get SSL security certificate errors). It should be off by default. Having this off enables the Chrome browser to still grab the status and redirect URL, even if it can’t otherwise access the https:// website due to a squiffy SSL certificate. It’s the only link-checking software I found that can do this.

7. On the top menu bar, find the “Mode” item and use this to switch to List mode. This will enable you to load your list of Web URLs to be checked for viability. By default it is set to a zero depth for following links, except when Always Follow Redirects is on. In that case it makes an exception for redirects.

8. Load and then run the list, by pressing the “Start” button.

9. Let it run until completed. Now filter the results by the Response Codes tab. This removes a whole lot of unwanted columns from the results table.

10. Then adjust the table’s columns so you only see the columns you need. Which in this case would be…

Address | Status Code | Status | Redirect URL

11. Just the one time, then save the current UI configuration as the Default. Top Menu | File Configuration | Save as Default. Now, when you switch through to the Response Codes tab in future, you should see the arrangement of columns you just saved.

12. Nearly done. Now sort the “Status” column by clicking once on it, so all the various status codes are grouped thus…

And save the project file for later reloading, or passing to a colleague for manual checking, as needed.

You can also do things like have a Dark UI (Config | User Interface | Theme | Dark). It appears you can’t make the font a touch bigger, but you can get a .CSV out which you can then do what you want with. That’s done via Top Menu | Reports | Redirects | All Redirects.

So basically SEO Spider is now Linkbot Pro, albeit with the regrettable loss of that fab 1996 Windows UI feel. It has nice new touches such as ignoring SSL errors, and even the ability to check (one at a time) for presence of the URL on Google Search…