How to Crawl my Website

Did you know that domain crawling is free on the SEO tool? The Content 360 module is designed to provide vital insights into technical performance areas such as meta data, engagement metrics such as Bounce Rate and the backlink profile, all at a page-by-page level.

This kind of detailed information can be available for your site(s) as well as any competitors. To create an automated schedule for your domains, first head over to the Settings part of the platform and click on Crawl Schedule.

Here you will be able to view and edit any already active schedules, as well as setup a new crawl. As an example, let’s say I want to run a fresh crawl each month for every domain that I am monitoring:

Under the ‘New Schedule’ option, I can:

• Configure the sites I want to include

• Choose the interval of the crawl, either weekly or monthly (*)

• Determine specific settings such as where any ignored paths or parameters

• Select how many pages I want to return in the crawl data, up to a maximum of 20,000

(*) Daily interval is available as a special option, just ask your Client Success Manager.

Hitting save will add the schedule to my list and the platform will automatically run these crawls on the first of next month and every month after. The data can then be viewed in the Content 360 module.