Crawl, render, index, and serve reliably

Four different gates

Crawling is fetching a URL. Rendering is producing usable content from its HTML and required resources. Indexing is a system deciding to store and make that page eligible. Serving is selecting it for a particular request. Passing one gate does not guarantee the next.

Start with the URL users should reach. Confirm a successful response, useful server-rendered HTML, one stable canonical, indexability, and links that a crawler can follow. Test without relying only on a browser's final visual state: a beautiful client-rendered shell can send thin initial HTML or fail when scripts are unavailable.

Audit in dependency order

1. Resolve DNS and TLS reliably.
2. Return the intended status code without redirect loops.
3. Serve the main title, headings, copy, and links in meaningful HTML.
4. Check robots.txt access and page-level robots directives separately.
5. Confirm canonical and language signals match the intended URL.
6. Make the page reachable through crawlable internal links.
7. Inspect indexing tools and logs after the foundations are correct.

A robots.txt disallow controls crawling; it is not a reliable way to remove an already known URL from search. A noindex directive must be fetched to be seen. Authentication, accidental 404s, soft 404s, server errors, infinite parameters, and duplicate routes each need different remedies.

SaaS failure patterns

Marketing migrations commonly leave chains from old slugs, canonical tags pointing at staging, orphaned feature pages, or JavaScript-only copy. Documentation and app routes can also leak into the public index when host or robots rules are too broad. Keep app/API paths private while preserving access to genuinely public product, tool, course, and blog pages.

Use a bounded audit rather than assuming a crawler saw everything. Record tested URLs, exclusions, timeouts, and coverage limits. A “clean” audit over 50 pages is not proof that a 5,000-page site is clean.