Sites & Crawling

Adding sites, crawl schedules, page caps, crawl speed, and bulk import.

Adding a site

Dashboard → Add site. You need a URL; a display name is optional. Before the crawl starts you tick one checkbox: "I confirm I am authorized to audit this domain — I own or operate it, or I have permission from its owner to crawl and monitor it." That's our anti-abuse guardrail — SeoSwift only crawls sites its customers are entitled to audit. Each site gets a crawl schedule (daily by default) and the first crawl starts immediately.

Crawl schedules

Change a site's frequency any time under the site's Settings tab. You can also trigger a crawl on demand with Crawl now, or stop a crawl that's in progress.

How many pages we crawl

Each crawl covers up to a plan-level number of pages. When a site is bigger than its cap, the crawl still completes normally — it simply stops at the cap, so your changelog and health score describe the first N pages discovered from your homepage outward.

Crawl speed

By default the crawler fetches about one page per second. If a site rate-limits us (or you'd just rather we tread lightly), pick a gentler pace in the site's Settings → Crawl speed — available on every plan:

• Normal — 1 request/second. Fine for almost every site.
• Gentle — 1 request every 3 seconds. For sites that sometimes rate-limit our crawler (HTTP 429).
• Very gentle — 1 request every 6 seconds. For strict rate limits — crawls take noticeably longer but complete.

A crawler with manners

Our crawler, SeoSwiftBot, identifies itself honestly, obeys your robots.txt on every crawl, fetches one page at a time, and backs off politely when a site asks it to slow down. It reads your pages — it never changes anything. The full identity card (exact user-agent, published IPs, how to verify it) lives in Meet SeoSwiftBot.

JavaScript rendering

If a site builds its metadata with JavaScript (common with React, Vue, and friends), the plain HTML we fetch may look emptier than what Google renders. On Advanced and Enterprise plans you can enable JS rendering per site (site Settings) — we'll crawl it with a real browser instead. The Health tab even warns you when a site looks like it needs this (lots of pages with no H1 and no meta description).

When a crawl fails

If a crawl can't complete, the site shows Failed with the reason — so you can tell a problem on the monitored site apart from one on our end:

• The site is blocking our crawler (HTTP 403): the site's firewall or bot-protection is refusing us. This isn't a SeoSwift error, and JavaScript rendering won't help — the block happens before any page loads. To monitor the site, allowlist the SeoSwiftBot user-agent, or allowlist our published crawl IPs from seoswift.io/bot/ips.json.
• The site is rate-limiting us (HTTP 429): the crawler backs off automatically and retries; the next scheduled crawl usually succeeds. If it keeps happening, switch the site to a gentler Crawl speed (above).
• No pages were reachable: the URL may be wrong, or the site may be offline.

Bulk import

Migrating a whole client roster? Dashboard → Import accepts a CSV of URLs and names and creates sites in one go, reporting anything it skipped and why (bad URL, duplicate, or plan site-limit reached). The same authorization confirmation applies — one checkbox covering every domain in the file.

Webhooks

On Advanced and Enterprise plans, each site can have a webhook URL (site Settings). When changes are detected we POST a Slack-compatible message — point it at a Slack or Discord incoming webhook, or any endpoint that accepts one, and your channel gets the news as fast as your inbox does.

Still stuck? Email hello@seoswift.io — a human reads it.