How to Read the Crawl Stats Report in Search Console
Short answer
The Crawl Stats report in Search Console shows how many requests Googlebot made to a site, the download size, the server's response time, and host availability problems. The report sits on the Search Console Settings page.
In this article
- Where is the Crawl Stats report in Search Console?
- What does the total crawl requests chart mean?
- What does host status mean in the Crawl Stats report?
- Host problem categories
- How do you read the crawl responses breakdown?
- Crawl purpose, file type, and Googlebot type
- When does the Crawl Stats report belong in a client report?
- Frequently asked questions
The Crawl Stats report in Search Console shows Googlebot's activity while fetching a site's pages: requests, download size, response time, and server problems. Crawling is how Googlebot fetches a page before the page can be indexed.
The crawl stats report is usually opened when new pages are slow to appear in Google, or when clicks drop after a server problem. This guide covers where to find the report, what each chart means, host status, and crawl responses.
Where is the Crawl Stats report in Search Console?
The Crawl Stats report sits on the Search Console Settings page, in the Crawl stats row. The report is only available for root-level properties: a Domain property such as example.com, or a URL-prefix property at the root such as https://example.com.
Google's documentation says the report is aimed at advanced users. Sites with fewer than 1,000 pages usually don't need to monitor it regularly.
What does the total crawl requests chart mean?
Total crawl requests is the number of all Googlebot requests to the site's URLs in the report period, successful or not. Repeated requests to the same URL are counted separately.
| Metric | Meaning | What to check |
|---|---|---|
| Total crawl requests | All requests to the site's URLs, failed ones included | A sharp change against the previous 1–2 weeks |
| Total download size | Bytes downloaded while crawling; cached resources are requested once | A rise without new pages, for example images that grew |
| Average response time | The average response time of all resources fetched | A rise that coincides with fewer crawl requests |
The crawl rate should be relatively stable over 1–2 weeks, according to Google's documentation. A sudden drop can coincide with a robots.txt rule that is too broad, slow server responses, or a rising error rate.
What does host status mean in the Crawl Stats report?
Host status shows whether Googlebot ran into availability problems while accessing the site in the last 90 days. A green status means no significant problems in that period.
Host status has 3 levels:
- No significant availability problems in the last 90 days.
- Problems in the last 90 days, but more than 1 week ago.
- A problem in the last week, which may still recur and should be investigated.
Host problem categories
Host problems fall into robots.txt fetching, DNS resolution, and server connectivity. Each category has a chart with a dotted red threshold line.
- A robots.txt fetch counts as failed when the server answers 429 or 5xx. A 403, 404, or 410 counts as successful, because it means there is no robots.txt file.
- DNS resolution counts as failed when the DNS server doesn't recognize the host name or doesn't answer.
- Server connectivity counts as failed when the server doesn't answer or gives an incomplete response.
The example in Google's documentation: DNS failures above 5% of requests on 1 day count as a problem.
How do you read the crawl responses breakdown?
The crawl responses breakdown splits all server answers by status code, as a percentage of total responses. Most responses should be 200 or another good type, unless the site is being reorganized or moved.
| Group | Response codes | What it means for the site |
|---|---|---|
| Good | 200, 301, 308, 302, 307, 304, meta refresh | The page was fetched or redirected correctly |
| Possibly good | 404 | Normal when the page has really been removed |
| Bad | 5xx, 401, 407, other 4xx, DNS errors, timeouts, redirect errors | Needs checking, starting with the example URLs in the report |
Redirects are counted per hop, so a chain of 3 redirects shows up as 3 requests. The example URLs in each group are a sample, not a full list. Each URL's status can be checked one by one with URL Inspection.
Crawl purpose, file type, and Googlebot type
Crawl purpose splits requests into discovery, for URLs never crawled before, and refresh, for pages Google already knows. New pages Googlebot has never fetched appear under discovery.
Another breakdown splits requests by file type, for example HTML, image, JavaScript, CSS, and PDF. File type percentages count responses, not bytes. The Googlebot type breakdown separates smartphone, desktop, image, video, page resource load, AdsBot, and StoreBot.
When does the Crawl Stats report belong in a client report?
The Crawl Stats report belongs in a client report when host status is red, bad responses rise, or crawl requests drop sharply. A month without problems needs one sentence on host status.
Record 3 numbers every month: total crawl requests, average response time, and the share of bad responses. Last month's numbers make a change easier to explain, for example when organic traffic drops.
Google offers no way to increase the crawl rate. For small sites, Google's documentation points to updating the sitemap and checking for blocked pages.
The Crawl Stats report is not available in the Search Console API. Recalytics reads performance, sitemap, and URL Inspection data through that API, so crawl stats are recorded by hand when they belong in a report.
Frequently asked questions
Can the crawl rate be raised from Search Console?
No. Google offers no setting to increase the crawl rate.
Why do the numbers differ from the server logs?
Google's documentation says the report can differ slightly from server logs. The gap can be much larger when robots.txt was unavailable for a long time.
Does a 404 response mean there is a problem?
Not necessarily. A 404 is normal for a page that has really been removed.
Sources
The Recalytics Team writes guides on SEO and analytics reporting for digital agencies and website owners, based on Google's official documentation and Search Console and GA4 data.