What is a link crawler?
A link crawler is a tool that automatically scans your website to find and analyse every link it can reach. It builds a detailed picture of how your site connects together – mapping internal and external links, HTTP status codes, MIME types and crawl depth to help you understand your site's structure and performance.
How does it work?
Enter a starting URL and the crawler will fetch that page, extract all visible links, and then follow them through the site. It records key details for each link – such as the status code, content type and where it was found – before displaying the results in a searchable, filterable table.
What can it be used for?
- Auditing site structure and internal linking
- Finding broken links, redirect loops or missing pages
- Checking canonical and non-canonical URLs
- Identifying external link leakage or dead outbound links
- Analysing crawl depth and discovering orphaned content
- Generating link data for technical SEO or migration work
Key features
- Filter results by URL, hostname, status, type or depth
- Exclude specific MIME types or URL patterns
- Optional external link scanning
- Export results as CSV or XML sitemap files
- Results are saved in your browser, so you can leave and pick up where you left off later
Does it respect robots.txt or nofollow?
The crawler can optionally respect rel="nofollow", meta name="robots", and robots.txt – enable whichever you need in the crawl settings. With all three on, it won't follow links that have rel=nofollow attributes on them, won't follow links on pages that have in the , and won't fetch or follow any URL disallowed by the site's robots.txt.
Can it handle JavaScript-generated links?
No. The crawler only processes links found in static HTML source code as well as some links found in CSS and JS files. Links created dynamically through JavaScript or frameworks such as React or Vue will not be included.
Can I dig into a single broken link more deeply?
Yes – once the crawl's finished, use the header checker or redirect checker on any individual URL from the results to see exactly what response it's returning and why.