In this post you will learn how to use the Google search console to identify and fix indexing errors.
This is a very important step in the whole SEO process because if there are crawling or indexing problems, Google will not index your website correctly and this can negatively affect your rankings.
The two tools to use available in the Google Search Console are:
The Index coverage report and the
URL inspection tool.
To get access to the tools, the first step is to add and verify your website with the Google Search Console.
If you have already done this, then you can proceed with the next steps.
This post will cover the following topics:
- What is the Index Coverage Report?
- How to Find Errors with the Index Coverage Report
- How to Fix Index Coverage Report Errors
- How to Fix “Submitted URL has crawl issue” Errors
- How to Fix “Submitted URL not found (404)” errors
- How to Fix Server Error (5xx) - url is not on google: indexing errors
- How to Fix Redirect Errors
- How to Fix “Submitted URL seems to be a soft 404” Errors
- How to Fix “Submitted URL marked ‘noindex’” Errors
- How to Fix “Submitted URL blocked by robots.txt” Errors
- How to Fix “Submitted URL returns unauthorized request (401)” Errors
What is the Index Coverage report?
The index coverage report is available in the Google Search Console and shows you which of your pages have been successfully indexed by Google and which pages have not been indexed because of an error.
For each of the pages, you can get more details about the error and you have the option to request Google to re-index your web pages or website as a whole.
The index coverage report has replaced the ‘Crawl Errors’ report previously available in Google Webmaster Tools.
How to Find Errors with the Index Coverage Report
Login to Google Search Console and select your primary domain from the dropdown list (top left corner).
Click COVERAGE under Index to view the Index Coverage Report
You will notice that the top part of the report has 4 tabs:
- Error
- Valid with warmings
- Valid
- Excluded
Since our goal is to investigate any indexing errors, we will use the ERROR tab.
The troubleshooting process has two parts:
- First is to identify the errors
- Second to understand the errors and fix them
Make sure that only the ERROR tab is highlighted and scroll down to the DETAILS section.
You will notice that errors are grouped into categories. The possible values are:
- “Server error (5xx)”
- “Redirect Error”
- “Submitted URL seems to be a soft 404”
- “Submitted URL marked ‘noindex’
- “Submitted URL blocked by robots.txt”
- “Submitted URL returns unauthorized request (401)”
- “Submitted URL has crawl issue”
- “Submitted URL not found (404)”
For each error category, you can see the Validation status, trend and the number of affected pages.
You can click on any row to see more details about the affected pages.
How to Fix Index Coverage Report Errors
Now that you have learned how to find any errors, let’s see how to fix them.
As mentioned above, errors are grouped into 8 categories and depending on the type of error, we can follow a different path to fix it.
How to Fix “Submitted URL has crawl issue” Errors
A crawl issue means that a page has problems and Google cannot index it. You need to find out what exactly is the issue, fix it and resubmit the page to Google.
- The first step is to click the INSPECT URL button.
- Then click the VIEW CRAWLED PAGE and the MORE INFO from the right menu
One of the most common reasons why a page might have crawl issues is that some of the page resources (images, CSS, JavaScript) could not be loaded when Google tried to index the page.
Before investigating further, you should:
- Open a new browser window and visit the page. If it loads ok then most probably the errors were temporary.
- Click the TEST LIVE URL button to force Google to refresh the error report.
- Review the details in MORE INFO again.
- Click the REQUEST INDEXING button to re-submit the page to Google.
Go back to the Index Coverage Report and to the page that has issues and click the VALIDATE FIX button.
Google will notify you via email about the results of your indexing request.
If you still get errors or resources not found after clicking the TEST LIVE URL button, you should first fix the errors by changing your HTML Code and then request indexing and validating the fix.
How to Fix “Submitted URL not found (404)” errors
These kinds of errors are easy to fix. What it means is that a page could be found by the Google bot at the time of indexing.
In most cases, this can be a false alarm. So, the first thing to do is to check that the page is correctly not found.
Click on a page from the list and then the INSPECT URL button.
While waiting to get the data from Google Index, open a new browser window and type the URL.
If the page is found on your website and you want to add it to the Google Index then:
- Click the TEST LIVE URL button.
- Click the REQUEST INDEXING.
- Go Back to the report and click VALIDATE FIX.
If this is a page that indeed returns a 404 code and you don’t want Google to index it, you have two options:
First, to leave it as it is. Google will gradually remove the page from the index. This is normal and expected to happen for pages that are no longer valid or deleted for a valid reason.
Second, to redirect the page using a 301 redirection to a related page on your website.
How to Fix Server Error (5xx)
Pages listed here could not be accessed by the Google bot either because the server was down or not available at the moment.
Normally you should not have any Server Errors reported. If you have a LOT of errors, then this means that your server has issues and you should investigate more to find out why.
If you have a few errors, then most probably the page could not be accessed temporarily and this means that you can request from Google to re-index the page.
Follow this process:
Click on one of the affected pages and you will get a menu with options on the right.
First, click on INSPECT URL. Google will give you more details about the errors from the Google Index.
Url is not on Google: indexing errors
If you get the message "url is not on google: indexing errors" it means that Google has either removed the URL from their index because they could not access it or it's not in their index because when they first tried to crawl it was not available.
Follow these steps:
Open a new browser window and navigate to the URL. If it loads ok then go back to GSC and click the TEST LIVE URL button.
Google will re-fetch the page and give your more details. If it was a temporary error, you can click the REQUEST INDEXING button to re-submit the page to Google
If the page does not load in the browser, you need find out what the problem is and then revisit GSC, click the TEST LIVE ULR and then REQUEST INDEXING.
If you cannot solve the error then consider adding a "noindex" header tag to the page and remove it from your sitemap. This will instruct Google to stop accessing the page and to stop reporting any errors related to this page.
How to Fix Redirect Errors
When you get a ‘redirect error’, it means that the page could not be accessed by the Google bot because it redirects to a page that doesn’t exists or not working.
The procedure for fixing redirect errors is the same as before.
- Click on the INSPECT URL
- Get more details about the errors
- Click the TEST LIVE URL
- Fix the error and REQUEST INDEXING
- Go back and click VALIDATE FIX
If everything is ok and the error is fixed, you will see a PASSED message in the validation column.
How to Fix “Submitted URL seems to be a soft 404” Errors
When you get a soft 404 error it means that the page was not found (because it does not exist) but instead of telling search engines that it should be ignored, it returned a valid code.
You may have pages on your website that cannot be accessed directly but only after a user completes a specific action.
For example, let’s say that your checkout page is only shown to users AFTER they have added an item to their shopping cart.
If you still have the page listed in your sitemap, Google will try to crawl it but it will not find it because no items were added to the cart.
What to do for soft 404 errors?
- You will need to either return a 404 code for pages that are not valid
- Remove them from your sitemap so that Google will not access them
- Redirect them to a valid page
- Do nothing. Sometimes soft 404 errors are normal and expected.
How to Fix “Submitted URL marked ‘noindex’” Errors
This is not actually an error. What this means is that a page was submitted for indexing (through your sitemap) but it has the ‘noindex’ directive which instructs search engines not to add it to their index.
What you should do is review the list of pages that have the ‘noindex’ tag and make sure that you don’t want them in the Google index.
If a page was wrongly tagged as ‘noindex’, remove the page directive from the header and REQUEST INDEXING.
How to Fix “Submitted URL blocked by robots.txt” Errors
A page as submitted for indexing (through your sitemap) but there is a rule in your robots.txt file that instructs search engines not to index it.
You should follow the same procedure as above i.e. check if the pages are supposed to be blocked from Google index.
How to Fix “Submitted URL returns unauthorized request (401)” Errors
A page is included in your sitemap but Google cannot access it because it is password protected.
Since these pages are not publicly available, you should:
- Remove them from your sitemap
- Add a ‘noindex’ directive in the page’s header
- Block the directory (or protected areas) in your robots.txt file
How to Use the URL Inspection Tool
The URL Inspection tool allows you to check the index status of any page of your website and troubleshoot errors or request Google to re-index your website or a particular page.
The URL inspection tool replaced two tools that were previously available in Google Webmaster tools: The FECTH AS GOOGLE and BLOCKED RESOURCES report.
To use the URL inspection tool just type any URL in the INSPECT ANY URL dropdown found at the top of the page.
You can either enter your website domain or a specific URL.
The URL INSPECTION TOOL can be used to troubleshoot errors reported in the INDEX COVERAGE REPORT (as explained above) or to:
- Re-submit your website to Google
- Re-submit a particular page to Google.
How to resubmit your website to Google
When you make a number of changes to your website and you want to speed up the indexing process, you can request Google to re-index your website.
Type your domain in the URL INSPECTION TOOL.
Click REQUEST INDEXING.
How to resubmit a web page to Google
If a page content has changed dramatically and you want to inform Google about the changes, you can use the URL INSPECTION Tool and the REQUEST INDEXING button to speed up the process.
The process is the same as re-submitting your website (explained above).
When to use the REQUEST INDEXING feature?
Google is very good at picking up changes made to a website or webpage so in the majority of cases you don’t need to use the ‘Request Indexing’ feature.
Some valid uses are:
- When you migrate domains
- When you migrate from http to https
- When you make a website redesign
- When you publish time-sensitive news and you want to inform Google about it.
Conclusion
Don’t panic when you see errors in your Google Search Console account. In many cases errors are valid and expected.
Your priority is to solve any CRAW ISSUES or NOT FOUND (404) errors since these are the errors directly related with your rankings.
