.Google.com's John Mueller addressed a concern regarding why Google.com indexes webpages that are actually refused from crawling through robots.txt as well as why the it is actually secure to overlook the relevant Search Console reports concerning those crawls.Bot Web Traffic To Query Guideline URLs.The individual talking to the inquiry documented that robots were producing links to non-existent query guideline Links (? q= xyz) to pages with noindex meta tags that are actually additionally obstructed in robots.txt. What urged the concern is that Google is crawling the hyperlinks to those pages, getting obstructed through robots.txt (without seeing a noindex robots meta tag) then obtaining reported in Google Search Console as "Indexed, though blocked out by robots.txt.".The individual inquired the adhering to inquiry:." Yet right here's the large concern: why would Google index webpages when they can't also view the web content? What is actually the perk in that?".Google.com's John Mueller verified that if they can not crawl the page they can not find the noindex meta tag. He likewise creates an intriguing acknowledgment of the internet site: hunt operator, encouraging to dismiss the outcomes since the "typical" individuals will not find those results.He composed:." Yes, you are actually appropriate: if our company can not crawl the webpage, our team can't view the noindex. That stated, if we can not crawl the web pages, after that there is actually not a great deal for us to mark. So while you might see some of those pages with a targeted internet site:- query, the average consumer won't observe all of them, so I wouldn't bother it. Noindex is additionally great (without robots.txt disallow), it only implies the Links will end up being crept (and end up in the Explore Console record for crawled/not recorded-- neither of these standings trigger problems to the rest of the web site). The vital part is that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the limitations in using the Site: search accelerated search operator for analysis explanations. One of those reasons is given that it is actually certainly not attached to the frequent hunt mark, it's a different point entirely.Google.com's John Mueller talked about the internet site hunt operator in 2021:." The quick answer is actually that a site: query is actually certainly not suggested to become total, nor made use of for diagnostics functions.A website inquiry is a details sort of hunt that confines the results to a certain website. It's essentially just the word website, a colon, and then the website's domain name.This concern confines the outcomes to a specific website. It is actually certainly not suggested to be a complete assortment of all the pages from that web site.".2. Noindex tag without making use of a robots.txt is actually alright for these sort of circumstances where a crawler is connecting to non-existent webpages that are actually receiving found by Googlebot.3. Links with the noindex tag will generate a "crawled/not recorded" entry in Browse Console and also those won't have a negative effect on the remainder of the web site.Go through the concern and answer on LinkedIn:.Why will Google.com index pages when they can not even find the material?Included Graphic through Shutterstock/Krakenimages. com.