robots.txt Checker — Server Diagnostic Tool

Network

robots.txt Checker is a server-based utility that checks whether a site has robots.txt and previews key lines from an entered url.

robots.txt Checker

Server lookupSSRF protected

Utillica checks only public URLs or domains from the server. Localhost, private IPs, internal networks, and metadata hosts are blocked, and raw API errors are hidden.

Site URL

What is it

robots.txt Checker fetches HTTP, DNS, robots.txt, sitemap, and Open Graph signals that are hard to inspect directly from a browser and summarizes them in a table. Localhost, private IPs, and internal networks are blocked for security, and results are a point-in-time reference.

Input URL or domain → server lookup → response status, headers, and metadata summary

How to use

1Enter the url.
2Press lookup.
3Review status codes, records, headers, or metadata.

Reference

robots.txt Checker — Server Diagnostic Tool Reference
Input	Result
URL	checks whether a site has robots.txt and previews key lines

Sources & standards

HTTP Semantics - IETF
The robots.txt file format - IETF
Sitemaps XML protocol - Sitemaps.org
The Open Graph protocol - Open Graph protocol
WHOIS Protocol Specification - IETF
The Transport Layer Security (TLS) Protocol Version 1.3 - IETF

FAQ

Can robots.txt Checker check localhost?

No. Localhost, private IPs, and internal network addresses are blocked to prevent server-side request forgery.

Are robots.txt Checker results live?

They are fetched by the server at request time. DNS, caches, redirects, and target server policies can change the result.