# robots.txt — harcourtvaluations.ai
#
# Policy: search and citation YES, model training NO.
#
# Harcourt Valuations wants to be indexed, quoted and cited — by search engines
# and by AI assistants answering user questions. It does not want its NAV models
# absorbed into training corpora. Those are different crawlers, and each vendor
# publishes separate user-agent tokens for them, so the distinction is enforceable
# rather than aspirational.
#
# Maintained by hand. Cloudflare's Managed robots.txt must stay OFF: it PREPENDS
# its own directives to this file and adds new bots to its blocklist over time,
# which is how ClaudeBot and GPTBot came to be disallowed here without a decision
# being made. If those Disallow lines reappear at the top of the served file,
# managed mode has been re-enabled.
#
# Content Signals Policy (Cloudflare): search=yes, ai-train=no, use=reference
#   search      = may be indexed and linked in search results
#   ai-train    = may NOT be used to train generative models
#   use=reference = may be retrieved and cited to answer a user's question
#
# Last reviewed: 2026-08-29

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

# ── AI TRAINING CRAWLERS — disallowed ────────────────────────────────────────
# Each of these is documented by its vendor as collecting content for training
# generative models. Blocking them does not affect search ranking or citation.

# OpenAI — training only. Their search and on-demand agents are allowed below.
User-agent: GPTBot
Disallow: /

# Anthropic — training only. Their search and on-demand agents are allowed below.
User-agent: ClaudeBot
Disallow: /

# Apple — Applebot-Extended is the training control. Applebot itself (Siri and
# Spotlight search) is deliberately NOT blocked.
User-agent: Applebot-Extended
Disallow: /

# Common Crawl — its archive is a primary source for third-party training sets.
User-agent: CCBot
Disallow: /

# Meta — training.
User-agent: meta-externalagent
Disallow: /

# ByteDance — training, and does not reliably identify a citation path.
User-agent: Bytespider
Disallow: /

# ── SEARCH AND CITATION AGENTS — explicitly allowed ──────────────────────────
# These are listed even though "User-agent: *  Allow: /" already permits them.
# The explicit group is the point: it records that allowing them is a decision,
# so a future blanket AI block does not sweep them up by accident.

# Google — Google-Extended is a SINGLE token covering BOTH Gemini training and
# grounding in the Gemini apps and the Vertex AI Gemini API. Unlike OpenAI and
# Anthropic, Google publishes no separate search agent, so the search-yes /
# train-no policy at the top of this file CANNOT be expressed here. It is one
# switch: disallowing it forfeits Gemini grounding entirely.
#
# Measured 2026-08-29: with this disallowed, Gemini could not answer "what is
# Vizsla Silver worth" from this site at all, and answered correctly the moment
# the user typed "Harcourt" — naming the site moves Gemini off grounding and
# onto a direct fetch, which this token does not govern. The pages themselves
# were never the problem: 18,088 bytes of server-rendered HTML to a Googlebot
# user-agent, NAV and P/NAV present as plain text, Googlebot allowed, sitemap 200.
#
# DECISION 2026-08-29 (Mumford): ALLOW. Grounding in Gemini is the distribution
# channel. Everything a crawler can reach is the teaser layer — name, headline
# NAV, P/NAV, price deck, model date; the models, PDFs and provenance sit behind
# R2 and the Qatom paywall regardless. ai-train=no was protecting the numbers we
# most want propagated while the assets worth protecting are protected by
# architecture. This is a deliberate exception to the Content-Signal policy
# above, not an oversight. Note Google has not committed to honouring
# Content-Signal, so this is a real concession, not a paper one.
User-agent: Google-Extended
Allow: /

# OpenAI — OAI-SearchBot builds the ChatGPT search index; ChatGPT-User fetches a
# page on demand when a user asks a question.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic — Claude-SearchBot indexes for search quality; Claude-User fetches on
# demand for a user's question.
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Perplexity — search and citation. Already referring traffic.
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# ── CONVENTIONAL SEARCH ENGINES — never block ────────────────────────────────

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

User-agent: DuckDuckBot
Allow: /

Sitemap: https://harcourtvaluations.ai/sitemap.xml

# Machine-readable description of this site for AI agents and LLMs.
# Note: "LLMs:" is not a recognised robots.txt directive and no crawler acts on
# it — carried over from the previous file because it is harmless and documents
# intent. Discovery of llms.txt happens by convention at the well-known path,
# and it is listed in sitemap.xml, which is what actually gets it crawled.
LLMs: /llms.txt