AI Spider audits your site the way AI models crawl it: extractability, citability, JS dependency, chunk quality, and retrieval readiness. Redirect intelligence, canonical analysis, PageRank, near-duplicate detection, and on-demand AI analysis for N-Grams, Citation Readiness, Section Strength, and Authority. Not just rankings. Actual AI discoverability.
AI Spider goes beyond status codes and titles. It evaluates every page the way an AI language model would encounter it - and scores it accordingly.
Six-dimensional scoring per page: extractability, citability, structure, AI crawlability, chunk quality, and link authority. Aggregated into a single 0-100 score.
Crawls HTML pages AND discovers images, JavaScript, CSS, PDFs, and fonts via HEAD requests. Full site inventory like Screaming Frog - not just HTML.
Watch pages appear in the URL table live during crawl. Scores, signals, and issue counts update the moment each page is processed.
Server-Sent EventsDetects redirect chains (3+ hops), redirect loops, mixed 301/302 chains, and redirects pointing to noindex pages. Full chain visualisation.
Canonical chains, loops, canonical pointing to 404/noindex/redirect pages. Goes beyond "has a canonical" to verify the canonical is actually valid.
Directory-level aggregations: avg AI score per folder, avg crawl depth, issue rate, page count. Understand your site structure at a glance.
Internal PageRank via power iteration plus HITS hub and authority scores. Orphan detection, buried authority pages, leakage analysis, and dead-end flagging.
HITS AlgorithmMinHash Jaccard similarity across all pages post-crawl. Exact duplicates via SHA-256 hash, near-duplicates at 80%+ content overlap. Fully algorithmic. No LLM required..
MinHash · 128 hash functions22+ site-wide and per-page recommendations backed by real data - structured data gaps, AI crawler blocks, citation readiness, section strength, authority leakage, and more.
/pages/recommendationsClick any URL in any tab to open a full-detail drawer: AI scores, bot access, content signals, inlinks/outlinks with anchor analysis, and per-page fixes.
Universal page drawerScores every page on how citable it is - statistics, dates, source attributions, quotes, tables, and lists. Identifies pages AI systems are likely to cite vs ignore.
Post-crawl analysisScores each content section (H2 block) on word count, lists, tables, numbers, and links. Identifies weak sections that undermine AI extractability.
Per-section scoringExtracts the most frequent 2- and 3-word phrases across all crawled content. Reveals your site's key topics and terminology as AI systems actually see them.
Post-crawl NLPFull inventory of JS, CSS, images, fonts, and media. MIME-type aware classification, broken resource detection, and found-on-page counts.
MIME-awareDirectory tree with avg AI score, issue rate, and depth per folder. Orphan pages, weakly-linked pages, buried authority detection, excessive outlinks, and dead-end page analysis.
4 sub-viewsCore analysis runs automatically after every crawl. On-demand tabs (N-Grams, Citation, Section Strength, Authority) run on the URLs you select, so you stay in control of what gets analysed.
Walks every redirect chain, flags loops and long chains with full hop visualisation
Validates every canonical tag destination - not just presence
MinHash Jaccard similarity across all page text. Exact + near-exact duplicates
Power-iteration PageRank and HITS hub/authority scores on the internal link graph
Extracts 2- and 3-word phrases from selected pages to reveal key topics as AI systems see them.
Deterministic scan for statistics, dates, sources, quotes, tables, and lists. Run on selected URLs from the Citation tab.
Hub and authority scores via the HITS algorithm on selected pages. Identifies buried authority, dead ends, and leaking pages.
Per-chunk scoring of word count, structure, and data richness for every H2 section. Run on selected URLs.
Every issue is categorised by severity and linked to the affected page. Issues feed the recommendations engine directly.
Built on a modern full-stack architecture. Runs entirely on your machine - no cloud, no data leaving your infrastructure.
The next frontier of AI Spider. These features go beyond deterministic signals - using large language models to understand how AI systems reason about, retrieve, and reconstruct your content.
Screaming Frog is the gold standard for technical SEO crawling. AI Spider matches it on the fundamentals and goes further on AI-specific signals.
| Feature | Screaming Frog | AI Spider |
|---|---|---|
| HTML page crawling | ✓ | ✓ |
| Resource URL discovery (images, JS, CSS) | ✓ | ✓ |
| AI Retrievability Scoring | ✗ | ✓ |
| GPTBot / ClaudeBot access analysis | ✗ | ✓ |
| Content chunking for AI retrieval | ✗ | ✓ |
| Citation Readiness scoring | ✗ | ✓ |
| Section Strength analysis | ✗ | ✓ |
| HITS hub + authority propagation | ✗ | ✓ |
| N-Gram phrase intelligence | ✗ | ✓ |
| llms.txt detection | ✗ | ✓ |
| Near-duplicate detection (MinHash) | ✗ | ✓ |
| 22+ recommendation engine | ✗ | ✓ |
| Redirect chain detection | ✓ | ✓ |
| Canonical intelligence | ✓ | ✓ |
| Internal PageRank | ✓ | ✓ |
| Structured data extraction | ✓ | ✓ |
| Real-time crawl updates | ✗ | ✓ (SSE live) |
| Architecture Intelligence (4 sub-views) | ✗ | ✓ |
| Page Intelligence Panel | ✗ | ✓ |
| Internal Link Explorer | ✓ | ✓ |
| Resource Explorer (MIME-aware) | ✓ | ✓ |
| MCP server integration | ✓ (v24) | Planned |
| Runs locally (no cloud) | ✓ (desktop app) | ✓ (local server) |
| Pricing | €245/year | Free during beta |
No LLM API key required. Runs locally. Free during private beta.
View on GitHub →