Sites & Crawling
Adding sites, crawl schedules, JS rendering, and bulk import.
Adding a site
Dashboard → Add site. You need a URL; a display name is optional. Each site gets a crawl schedule — daily by default — and the first crawl starts immediately.
Crawl schedules
| Frequency | Available on |
|---|---|
| Weekly | All plans |
| Every 3 days | All plans |
| Daily | All plans |
| Every 6 hours | Advanced and Enterprise |
| Hourly | Enterprise |
Change a site's frequency any time under the site's Settings tab. You can also trigger a crawl on demand with Crawl now, or stop a crawl that's in progress.
A crawler with manners
Our crawler (SeoSwiftBot) fetches about one page per second, one page at a time, and backs off politely if a site asks it to slow down. It reads your pages — it never changes anything. Typical sites finish in a few minutes.
PDFs and other files
Documents and images linked from your pages aren't crawled as pages, but we do check their status codes and X-Robots-Tag headers — the only indexing control that exists for non-HTML files. They show up as "assets" in your page counts.JavaScript rendering
If a site builds its metadata with JavaScript (common with React, Vue, and friends), the plain HTML we fetch may look emptier than what Google renders. On Advanced and Enterprise plans you can enable JS rendering per site (site Settings) — we'll crawl it with a real browser instead. The Health tab even warns you when a site looks like it needs this (lots of pages with no H1 and no meta description).
When a crawl fails
If a crawl can't complete, the site shows Failed with the reason — so you can tell a problem on the monitored site apart from one on our end:
- The site is blocking our crawler (HTTP 403): the site's firewall or bot-protection is refusing us. This isn't a SeoSwift error, and JavaScript rendering won't help — the block happens before any page loads. To monitor the site, allow our crawler (user-agent
SeoSwiftBot) through its bot protection. - The site is rate-limiting us (HTTP 429): usually temporary — the next scheduled crawl retries automatically.
- No pages were reachable: the URL may be wrong, or the site may be offline.
Blocked sites aren't re-crawled endlessly
When a site keeps refusing us, we stop firing extra catch-up crawls at it and fall back to its normal schedule — so we never hammer a site that has already turned us away.Bulk import
Migrating a whole client roster? Dashboard → Import accepts a CSV of URLs and names and creates sites in one go, reporting anything it skipped and why (bad URL, duplicate, or plan site-limit reached).
Webhooks
Each site can have a webhook URL (site Settings). When changes are detected we POST a Slack-compatible message — point it at a Slack incoming webhook and your channel gets the news as fast as your inbox does.
Still stuck? Email hello@seoswift.io — a human reads it.