Skip to content
Mizan SEO

Data & integrations

About the Mizan crawler

How your site identifies our crawler, how it respects robots.txt, and how to allow it through a firewall.

Crawler identity

The crawler identifies itself as MizanSEOBot/1.0 in the User-Agent header.

Respecting robots.txt

The crawler reads robots.txt and avoids paths disallowed for * or MizanSEOBot. We also measure whether rules block Googlebot, because that affects your visibility.

Crawl limits

  • A page limit per audit based on your plan
  • A limited number of concurrent requests
  • A timeout per request and an overall time limit per audit

Firewalls and CDNs

If your site sits behind Cloudflare or a firewall that blocks bots, the audit may fail or see challenge pages instead of content. Allow our User-Agent or temporarily relax protection during the audit.