What is a link crawler?
A link crawler is a tool that automatically scans your website to find and analyse every link it can reach. It builds a detailed picture of how your site connects together – mapping internal and external links, HTTP status codes, MIME types and crawl depth to help you understand your site's structure and performance.
How does it work?
Enter a starting URL and the crawler will fetch that page, extract all visible links, and then follow them through the site. It records key details for each link – such as the status code, content type and where it was found – before displaying the results in a searchable, filterable table.
What can it be used for?
- Auditing site structure and internal linking
- Finding broken links, redirect loops or missing pages
- Checking canonical and non-canonical URLs
- Identifying external link leakage or dead outbound links
- Analysing crawl depth and discovering orphaned content
- Generating link data for technical SEO or migration work
Key features
- Filter results by URL, hostname, status, type or depth
- Exclude specific MIME types or URL patterns
- Optional external link scanning
- Export results as CSV or XML sitemap files
- Results are saved in your browser, so you can leave and pick up where you left off later
Does it respect robots.txt or nofollow?
The crawler respects rel="nofollow" and meta name="robots". This means that it won't follow any links that have rel=nofollow attributes on them – and it won't follow links on pages that have in the .
It doesn't currently read or enforce robots.txt rules, though this may be added in the future.
Can it handle JavaScript-generated links?
No. The crawler only processes links found in static HTML source code as well as some links found in CSS and JS files. Links created dynamically through JavaScript or frameworks such as React or Vue will not be included.