Google’s John Mueller lately answered a query about phantom noindex errors reported in Google Search Console. Mueller asserted that these experiences could also be actual.
Noindex In Google Search Console
A noindex robots directive is without doubt one of the few instructions that Google should obey, one of many few ways in which a web site proprietor can train management over Googlebot, Google’s indexer.
And but it’s not completely unusual for search console to report being unable to index a web page due to a noindex directive that seemingly doesn’t have a noindex directive on it, not less than none that’s seen within the HTML code.
When Google Search Console (GSC) experiences “Submitted URL marked ‘noindex’,” it’s reporting a seemingly contradictory state of affairs:
- The location requested Google to index the web page by way of an entry in a Sitemap.
- The web page despatched Google a sign to not index it (by way of a noindex directive).
It’s a complicated message from Search Console {that a} web page is stopping Google from indexing it when that’s not one thing the writer or website positioning can observe is occurring on the code degree.
The individual asking the query posted on Bluesky:
“For the previous 4 months, the web site has been experiencing a noindex error (in ‘robots’ meta tag) that refuses to vanish from Search Console. There isn’t any noindex wherever on the web site nor robots.txt. We’ve already seemed into this… What might be inflicting this error?”
Noindex Exhibits Solely For Google
Google’s John Mueller answered the query, sharing that there have been all the time a noindex exhibiting to Google on the pages he’s examined the place this sort of factor was occurring.
Mueller responded:
“The circumstances I’ve seen previously had been the place there was really a noindex, simply generally solely proven to Google (which may nonetheless be very onerous to debug). That mentioned, be happy to DM me some instance URLs.”
Whereas Mueller didn’t elaborate on what may be occurring, there are methods to troubleshoot this concern to search out out what’s occurring.
How To Troubleshoot Phantom Noindex Errors
It’s doable that there’s a code someplace that’s inflicting a noindex to point out only for Google. For instance, it might have occurred {that a} web page at one time had a noindex on it and a server-side cache (like a caching plugin) or a CDN (like Cloudflare) has cached the HTTP headers from that point, which in flip would trigger the previous noindex header to be proven to Googlebot (as a result of it often visits the positioning) whereas serving a contemporary model to the positioning proprietor.
Checking the HTTP Header is simple, there are numerous HTTP header checkers like this one at KeyCDN or this one at SecurityHeaders.com.
A 520 server header response code is one which’s despatched by Cloudflare when it’s blocking a consumer agent.
Screenshot: 520 Cloudflare Response Code
Under is a screenshot of a 200 server response code generated by cloudflare:
Screenshot: 200 Server Response Code
I checked the identical URL utilizing two totally different header checkers, with one header checker returning a a 520 (blocked) server response code and the opposite header checker sending a 200 (OK) response code. That exhibits how in another way Cloudflare can reply to one thing like a header checker. Ideally, strive checking with a number of header checkers to see if there’s a constant 520 response from Cloudflare.
Within the state of affairs the place an online web page is exhibiting one thing solely to Google that’s in any other case not seen to somebody trying on the code, what you’ll want to do is to get Google to have a look at the web page for you utilizing an precise Google crawler and from a Google IP tackle. The best way to do that is by dropping the URL into Google’s Wealthy Outcomes Check. Google will dispatch a crawler from a Google IP tackle and if there’s one thing on the server (or a CDN) that’s exhibiting a noindex, this can catch it. Along with the structured knowledge, the Wealthy Outcomes take a look at can even present the HTTP response and a snapshot of the online web page exhibiting precisely what the server exhibits to Google.
Whenever you run a URL via the Google Wealthy Outcomes Check, the request:
- Originates from Google’s Information Facilities: The bot makes use of an precise Google IP tackle.
- Passes Reverse DNS Checks: If the server, safety plugin, or CDN checks the IP, it’s going to resolve again to googlebot.com or google.com.
If the web page is blocked by noindex, the device will likely be unable to offer any structured knowledge outcomes. It ought to present a standing saying “Web page not eligible” or “Crawl failed”. If you happen to see that, click on a hyperlink for “View Particulars” or develop the error part. It ought to present one thing like “Robots meta tag: noindex” or ‘noindex’ detected in ‘robots’ meta tag”.
This strategy doesn’t ship the GoogleBot consumer agent, it makes use of the Google-InspectionTool/1.0 consumer agent string. Which means if the server block is by IP tackle then this technique will catch it.
One other angle to verify is for the state of affairs the place a rogue noindex tag is particularly written to dam GoogleBot, you possibly can nonetheless spoof (mimic) the GoogleBot consumer agent string with Google’s personal Person Agent Switcher extension for Chrome or configure an app like Screaming Frog set to determine itself with the GoogleBot consumer agent and that ought to catch it.
Screenshot: Chrome Person Agent Switcher
Phantom Noindex Errors In Search Console
These sorts of errors can really feel like a ache to diagnose however earlier than you throw your fingers up within the air take a while to see if any of the steps outlined right here will assist determine the hidden cause that’s accountable for this concern.
Featured Picture by Shutterstock/AYO Manufacturing