How the check works—and where it stops.

A provider can publish different crawler names for search discovery, AI model use or training, and user-requested retrieval. WebVisible checks each named crawler separately and shows its purpose before the rule result.

A WebVisible request is always a WebVisible request. Changing a User-Agent would not supply the official crawler's IP range, reverse-DNS identity, signed headers, cookies, account state, or private provider behavior, so WebVisible does not present simulated User-Agent fetches as official bot visits.

The parser retains evidence rather than returning only a color:

• byte-order mark, comments, LF/CRLF, and multiple groups are handled;
• the most specific matching product-token group is selected, with wildcard fallback;
• matching groups for the same token are combined;
• the longest matching path rule wins;
• an Allow rule wins an equal-length tie;
• *, end-anchor $, query strings, and percent encoding are evaluated;
• the exact matched rule and line number are returned where available.

HTTP 404/410 for /robots.txt is reported as no published policy file. A denied, rate-limited, oversized, timed-out, or server-error response is reported as indeterminate—not automatically allowed.

WebVisible combines generic and bot-specific robots meta directives with X-Robots-Tag evidence. It reports noindex and contradictions explicitly. “Eligible by observed directives” only means that the checked response did not explicitly exclude the page; it is not an index lookup.

HTML analysis is bounded. The scanner extracts structural facts and a short text sample, then discards the response body after the request. It does not execute page markup in the WebVisible interface.

The optional browser check waits for the page network to settle, then reviews the rendered page, readable text, and accessibility structure. WebVisible compares text, headings, links, JSON-LD, canonical, and robots metadata before and after JavaScript.

The result can identify important content that appears only after JavaScript, content removed during page startup, metadata conflicts, or a possible challenge overlay. It still describes one WebVisible browser check—not every crawler's rendering system or schedule.

• Blocker: deterministic evidence prevents the tested access or directive outcome.
• Warning: a verified conflict or material risk that does not prove exclusion.
• Pass: one narrow check passed; it is never generalized beyond that check.
• Information: useful context without a negative judgment.
• Unknown: evidence is unavailable, ambiguous, provider-specific, heuristic, or outside WebVisible's authority.

A pass in one layer never cancels a blocker in another.

A fix is generated only when the scan has exact supporting evidence. It targets the smallest line, tag, or delivery behavior rather than rewriting an entire robots file, framework configuration, or schema graph.

WebVisible will not automatically recommend opening account, login, checkout, admin, API, private, or similarly sensitive paths. Every fix includes a re-scan path and a statement of what the change still cannot prove.

• public HTTP/HTTPS hostnames and standard ports only;
• IP literals, numeric host forms, localhost, internal suffixes, and common wildcard-DNS pivots are rejected;
• every redirect destination is revalidated;
• five redirects maximum, strict request deadlines, and bounded decompressed body reads;
• no custom headers, cookies, credentials, methods, proxies, or browser scripts;
• separate rate limits for Quick Scan, cache bypass, and rendered comparison.

These application controls reduce server-side request-forgery risk. They do not constitute cryptographic proof of the destination IP selected by the hosting platform at every moment.

WebVisible cannot determine from a public technical scan:

• whether a platform has actually crawled the URL;
• whether a private index contains or selected the URL;
• whether a verified official crawler can pass a target's network-level bot checks;
• whether the page will rank, be recommended, be summarized, or be cited;
• whether a missing optional llms.txt file has any negative effect.