HTTP header checker — see every response header

Quick answer: paste a URL and read the full response headers as Googlebot receives them, with the ones that can silently deindex or mis-cache a page called out first.

The header that deindexes a page invisibly

X-Robots-Tag carries the same directives as the robots meta tag — noindex, nofollow, noarchive — but it travels in the response rather than the HTML. View source shows nothing. A page can be perfect in every other respect and still be excluded by a line no one has looked at, set months ago at a CDN or a reverse proxy for a path that has since grown.

It is the first row in the result for that reason. An absent X-Robots-Tag is good news; a present one deserves reading carefully, because it may be doing exactly what someone intended and exactly what you did not expect.

Where a header beats a meta tag

A meta tag only exists inside HTML. If you need to keep a PDF, an image or a feed out of the index, the header is the only instrument that reaches it. The same applies to applying a directive across a whole path at once rather than editing every template.

Caching and Vary

Cache-Control decides how long a response can be reused, and on a hosted platform it is usually not yours to set. What is worth noticing is a page that says no-store while its content changes rarely, because every visitor and every crawl then pays the full cost for identical bytes.

Vary is the companion nobody checks. It names the request headers that change the response, and a page that serves different HTML by language or user agent without declaring it will be cached once and handed to everyone. That is a quiet way for a crawler to receive content no human sees.

What this does not tell you

Headers describe the transport, not the page. Nothing here says whether the content is any good, whether it is indexable once the HTML is parsed, or how fast it renders. For the HTML side, run the same URL through the SEO report generator, and for the crawl-control side the robots.txt tester.