AI Spider audits your site the way AI models crawl it - extractability, citability, JS dependency, chunk quality, and retrieval readiness. Not just rankings. Actual AI discoverability.
AI Spider goes beyond status codes and titles. It evaluates every page the way an AI language model would encounter it - and scores it accordingly.
Six-dimensional scoring per page: extractability, citability, structure, AI crawlability, chunk quality, and link authority. Aggregated into a single 0-100 score.
Crawls HTML pages AND discovers images, JavaScript, CSS, PDFs, and fonts via HEAD requests. Full site inventory like Screaming Frog - not just HTML.
Watch pages appear in the URL table live during crawl. Scores, signals, and issue counts update the moment each page is processed.
Server-Sent EventsDetects redirect chains (3+ hops), redirect loops, mixed 301/302 chains, and redirects pointing to noindex pages. Full chain visualisation.
Canonical chains, loops, canonical pointing to 404/noindex/redirect pages. Goes beyond "has a canonical" to verify the canonical is actually valid.
Directory-level aggregations: avg AI score per folder, avg crawl depth, issue rate, page count. Understand your site structure at a glance.
Internal PageRank via power iteration. Orphan page detection, buried authority pages, dead-end pages, excessive outlink flagging.
TF-IDF cosine similarity across all pages. Detects content cannibalization (>70% overlap) and topic overlap (50-70%). Two separate signals from two engines.
Site-wide and per-page recommendations backed by real data. Orphan pages, buried content, readability, missing citability signals - all prioritised by impact.
/pages/recommendationsNo manual triggers. No separate tools. The moment a crawl completes, AI Spider runs 8 post-crawl analysis engines in sequence.
Walks every redirect chain, flags loops and long chains with full hop visualisation
Validates every canonical tag destination - not just presence
MinHash Jaccard similarity across all page text. Exact + near-exact duplicates
Power-iteration PageRank on the internal link graph. Scores persisted to pages table
TF-IDF cosine similarity. Finds content cannibalization and topic overlap
BCP-47 validation, x-default presence, reciprocal link checking
Every issue is categorised by severity and linked to the affected page. Issues feed the recommendations engine directly.
Built on a modern full-stack architecture. Runs entirely on your machine - no cloud, no data leaving your infrastructure.
Screaming Frog is the gold standard for technical SEO crawling. AI Spider matches it on the fundamentals and goes further on AI-specific signals.
| Feature | Screaming Frog | AI Spider |
|---|---|---|
| HTML page crawling | ✓ | ✓ |
| Resource URL discovery (images, JS, CSS) | ✓ | ✓ |
| AI Retrievability Scoring | ✗ | ✓ |
| GPTBot / ClaudeBot access analysis | ✗ | ✓ |
| Content chunking for AI retrieval | ✗ | ✓ |
| llms.txt detection | ✗ | ✓ |
| Semantic similarity (TF-IDF) | ✗ | ✓ |
| Site-wide recommendations engine | ✗ | ✓ |
| Redirect chain detection | ✓ | ✓ |
| Canonical intelligence | ✓ | ✓ |
| Internal PageRank | ✓ | ✓ |
| Hreflang validation | ✓ | ✓ |
| Structured data extraction | ✓ | ✓ |
| Real-time crawl updates | ✗ | ✓ (SSE live) |
| MCP server integration | ✓ (v24) | Planned |
| Runs locally (no cloud) | ✓ (desktop app) | ✓ (local server) |
| Pricing | €245/year | Free during beta |