# Search engines. # Private / transactional routes are deliberately absent from sitemap.xml AND # disallowed here: /auth, the per-user /account dashboard, the post-checkout # pages (/success, /product-success, /payment-failed), the owner-only dashboards # (/analytics, /errors, /health-check, /scan-metrics), the Agent Pass receipt # (/agents/pass — signed-in only, reads the buyer's own row) and the OAuth # consent route (/oauth/consent — exists only for an agent host's authorization # request), the market-pulse confirmation page (/market-pulse/confirm -- # opened only from a single-use link in a confirmation mail) and the fix-plan # confirmation page (/fix-plan/confirm -- opened only from the button in a # report mail) and the unsubscribe page (/email/unsubscribe -- opened only from # the link in one of our mails) have zero search value and must never be indexed. (Lovable's # route-vs-sitemap checker flags these as "missing" — that is expected; do NOT # add them to the sitemap.) # Disallow is repeated per named group on purpose: a crawler obeys only its own # most-specific User-agent block and never inherits rules from "*". The list is # the PRIVATE_ROUTES allowlist in src/test/sitemap-prerender-parity.test.ts, # and src/test/the-board-hands-a-job-and-a-search-to-the-agent.test.tsx fails # when any of those routes is missing from any block here. User-agent: Googlebot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: Bingbot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe # AI / answer engines — explicitly welcome. Our pages are prerendered static # HTML with cited sources; being read and cited by these is the point. The # private routes above are private to these readers too: the receipt and # consent pages serve the app shell with nothing to read. # See also: /llms.txt (overview) and /llms-full.txt (full guide text). User-agent: GPTBot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: OAI-SearchBot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: ChatGPT-User Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: ClaudeBot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: Claude-Web Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: anthropic-ai Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: PerplexityBot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: Google-Extended Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: Applebot-Extended Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: CCBot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe # Social preview fetchers User-agent: Twitterbot Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: facebookexternalhit Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe # Meta's AI training crawler. On 2026-10-03 it rendered /jobs as an app from # 57.141.0.0/16 and read the job board about 2,400 times an hour, about 85% of # all board traffic, while sending no visitors. The board has refused its # network since job-board .87; this stops it loading the pages as well. Meta # documents that this agent obeys robots.txt. Link previews on Facebook, # Instagram and WhatsApp come from facebookexternalhit, above, and are # unaffected. The private routes are listed again because every block must # name them (src/test/the-board-hands-a-job-and-a-search-to-the-agent.test.tsx). User-agent: meta-externalagent Disallow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe User-agent: * Allow: / Disallow: /auth Disallow: /account Disallow: /success Disallow: /product-success Disallow: /payment-failed Disallow: /analytics Disallow: /errors Disallow: /health-check Disallow: /scan-metrics Disallow: /agents/pass Disallow: /oauth/consent Disallow: /market-pulse/confirm Disallow: /fix-plan/confirm Disallow: /email/unsubscribe Sitemap: https://resumebooster.work/sitemap.xml # ONE SITEMAP, AND EVERY URL IN IT SERVES ITS OWN PAGE. # # A second Sitemap line used to point at an index the board function serves # live. Measured 2026-09-23: it listed 767,391 URLs across 30 pages, every one # of the form /jobs?job= — and every one of those served, byte for byte, # the same file as /jobs. Same MD5, the board's own title and description, a # canonical naming /jobs, and no job markup in the served bytes; an id that # does not exist returned the identical bytes too. So we were asking crawlers # to index three quarters of a million copies of one page, and telling them on # each copy that the page was really /jobs. # # Individual openings now have real pages (/jobs/posting/...), a bounded set of # them is prerendered by scripts/prerender-seo.mjs, and exactly those are in # sitemap.xml above. The rule and the number are printed on /jobs itself. # Do not restore a sitemap of URLs that have no file behind them.