# AI training crawlers: disallowed. These bots harvest training data and # do not drive search traffic, so blocking them costs nothing. User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: GPTBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: SERankingBacklinksBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: CCBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: TikTokSpider Disallow: / # Everyone else (includes Googlebot, Bingbot, DuckDuckBot, OAI-SearchBot) User-agent: * Allow: / Crawl-delay: 5 # Admin and API endpoints Disallow: /admin/ Disallow: /api/ # Internal pages Disallow: /staff/ Disallow: /monitoring/ Disallow: /design/ # The Mod Center. Belt and braces: every route under it already redirects an # anonymous visitor to login, so a crawler can never render one. Safe as a prefix # because nothing else on the site lives under /mod/ (the recap is /monthly-recap/, # which this does not match -- the trailing slash is what keeps it exact). Disallow: /mod/ # Guides (temporarily hidden during redesign) Disallow: /guides/ Disallow: /my-guides/ # Game Lists (2026-09). The browse index and the public detail pages ARE crawlable and are in the # sitemap -- these are the parts of the feature that have nothing to offer a crawler: # # /my-lists/ personal and login-only, so a crawler gets a redirect to login, every time # .../search/ the adder's typeahead: bare JSON, no , and an unbounded query space # the write paths create/update/reorder/like/follow/add/remove are POST-only, so a GET is a 405 # # The write paths are listed individually rather than as a `/community/lists/*` wildcard, because # that pattern would also match the DETAIL pages -- `*` matches zero or more characters in robots # syntax, and there is no way to say "exactly one more segment". That is the same mistake the # /games/*/* rules made in 2026-08, when three sitemaps advertised pages this file was blocking. Disallow: /my-lists/ Disallow: /community/lists/create/ Disallow: /community/lists/*/search/ Disallow: /community/lists/*/update/ Disallow: /community/lists/*/reorder/ Disallow: /community/lists/*/like/ Disallow: /community/lists/*/follow/ Disallow: /community/lists/*/add/ Disallow: /community/lists/*/items/ # Quick-add: a per-concept read and a POST-only create. Login-gated and useless to a crawler. Disallow: /community/lists/for-game/ Disallow: /community/lists/new-with-game/ # Sections (2026-09), same reasoning as the rest of this block: POST-only, so a GET is a 405, and # nothing here is a page. Enumerated rather than wildcarded for the same reason too -- a # `/community/lists/*/sections*` rule would also match nothing useful while risking the detail # pages, which `*` matching zero characters makes easy to get wrong. Disallow: /community/lists/*/sections/ # `/edit/` is an address the rebuilt app deliberately does not have: it 302s to the homepage for any # id, so it is an infinite soft-404 space over an unbounded integer. Every other id-keyed sub-path is # blocked and the detail pages are in a sitemap, which makes the shape guessable. Disallow: /community/lists/*/edit/ # Challenges (2026-10). The browse page, the Hall of Fame and the run pages ARE crawlable, and finished runs are # in a sitemap. These two are not pages: # # /my-challenges/ personal and login-only, so a crawler gets a redirect to login; the picker, share and # write doors under it answer JSON, a PNG or POST-only # .../day/ a Calendar day square: a bare HTML fragment fetched into the run page's sheet # # `*/day/` cannot match a run page: `/community/challenges//` has no `/day/` segment after the id. Disallow: /my-challenges/ Disallow: /community/challenges/*/day/ # Profile-scoped variants of canonical pages (/games/// etc.) are NOT # handled here any more (2026-08-23, SEO Lane 0). The old wildcard rules # (/games/*/*) ALSO matched the canonical pages themselves: in robots syntax `*` # matches zero or more characters, so /games// matched /games/ * / * -- # every game, badge and jobs detail page on the site was blocked while three # sitemaps advertised them. There is no robots pattern for "exactly two path # segments." The variants are defended instead by the bot canonical 301 # (BotCanonicalRedirectMiddleware) and the anonymous-visitor gate, which both # survive URL renames the way this file cannot. # Bare HTML fragment endpoints (no , no title, no canonical) -- these are # fetched into tabs/modals by JS and must not be indexed as pages. Disallow: /jobs/*/ranks/ Disallow: /jobs/*/contracts/ Disallow: /badge-ranks/ Disallow: /leaderboards/rows/ Disallow: /group-badge-peek/ Disallow: /group-badge-progress-peek/ Disallow: /career/contracts/ # The per-game leaderboard URL serves a headless HTML fragment (the HTMX panel) -- no head, no # canonical, and the most expensive query on the site. ~35k URLs a crawler has no use for. Disallow: /games/*/leaderboard/ # Filtered / paginated / tabbed variants of a profile. The canonical # /profiles// page stays crawlable -- profiles are the free floor of # the product and should keep their search and share value -- but the # ?tab= / ?page= / ?sort= permutations are duplicates of it on axes that multiply # out into an unbounded crawl space, and each one costs a real query. # # The `/` before `?` is load-bearing: `*` matches the empty string, so the simpler # /profiles/*?* would also match /profiles/?page=2 and block the profile INDEX's # pagination -- the very thing a crawler should walk to discover profiles. # Requiring a path segment first confines this to profile detail URLs. # # ALL THREE spellings: the section moved twice in 2026-08, /community/profiles/ -> # /profiles/ -> /hunters/. Like the Cloudflare origin guard, this file holds hardcoded # paths that do NOT follow a rename the way {% url %} does -- leaving only an old line # would silently un-throttle crawling of the exact pages whose crawl cost started the # 2026-08 outage. The superseded lines stay while their 301s do, since a crawler # re-walking them still passes through the old path first. Disallow: /hunters/*/?* Disallow: /profiles/*/?* Disallow: /community/profiles/*/?* # Below the group on purpose: Sitemap is a non-group directive, and mid-group placement is the # one structure in this file that leans on parser generosity rather than being unambiguous. Sitemap: https://platpursuit.com/sitemap.xml