Making the Statamic Docs Agent-Ready

People aren't the only ones reading documentation anymore. Very possibly not even reading them much at all anymore. But that's okay, we've updated the Statamic docs so coding agents and answer engines can discover, understand, and consume them more efficiently without wasting tokens.

When someone asks an AI agent how to build something with Statamic, there's a good chance it will visit our documentation before answering. Unfortunately, documentation designed exclusively for browsers can be surprisingly expensive and awkward for an agent to consume. Navigation, scripts, styles, and other interface elements all compete for space in the agent's context window.

Cloudflare calls this type of optimization agent readiness. It complements Answer Engine Optimization (AEO), which focuses on whether answer engines understand, trust, and recommend your content. Two sides of the same coin. And since both of these things are particularly relevant for documentation, we gave statamic.dev the full robot treatment. Here's what we did, in case you want to consider doing something similar for your own projects.

Markdown all the things

HTML is excellent for humans, but agents prefer markdown. It contains the same useful structure without the layout and interface markup surrounding it. Cloudflare has measured token reductions of up to 80% when serving plain markdown instead of busy HTML. Valuemaxxing > tokenmaxxing.

Every Statamic documentation page is now available by appending .md to its URL:

https://statamic.dev/control-panel/users
https://statamic.dev/control-panel/users.md

The markdown response only contains the important content without the surrounding navigation or presentation markup. Internal links in these responses also point to their markdown equivalents. Once an agent enters the markdown version of the docs, it can continue navigating without unexpectedly falling back to HTML.

Content negotiation for agents that know how to ask

While explicit .md URLs are useful, agents can also request markdown using standard HTTP content negotiation:

curl https://statamic.dev/control-panel/users \
    -H "Accept: text/markdown"

The same URL then returns actual markdown even without the explicit .md at the end:

Content-Type: text/markdown; charset=UTF-8
Vary: Accept

Browsers (and the people looking at them, 🫵) continue to receive HTML, wildcard requests continue behaving normally, but clients explicitly preferring markdown will receive markdown. The Vary: Accept header is important. It tells browsers and CDNs that HTML and markdown are separate representations of the same URL. Without it, a cache could serve markdown to a browser or an entire HTML page to an agent. Neither is great.

Showing agents where everything lives

Every HTML documentation page now includes an alternate link in its <head>:

<link rel="alternate" type="text/markdown" href="https://statamic.dev/control-panel/users.md">

We also publish the same discovery information through HTTP Link response headers:

Link: <https://statamic.dev/control-panel/users.md>; rel="alternate"; type="text/markdown",
      <https://statamic.dev/llms.txt>; rel="describedby"; type="text/plain"

A map of the docs in `llms.txt`

We added an llms.txt file as a machine-readable map of the documentation. It introduces the site and lists the main documentation tree for easy reference.

We deliberately didn't add a llms-full.txt containing the entire documentation site. Statamic's docs are quite large, and shoving everything into one enormous file would gobble up an agent's context window before it even begins answering the question. A compact index lets the agent identify and fetch only the relevant pages.

Making `robots.txt` speak agent

Our robots.txt now does more than greet traditional search crawlers with a wave.

It now:

Advertises the sitemap
Points agents toward llms.txt
Explains the .md URL convention
Defines rules for common AI crawlers
Declares our Content Signals preferences for search, AI input, and training

Measuring the changes

We put these changes to the test with Cloudflare's Is Your Site Agent-Ready?, which checks discoverability, content accessibility, and bot access control. Before we started, statamic.dev scored just 13 points, earning Level 0 ("Not ready"). With these changes, it now scores a perfect 100 points and reaches Level 5 (Agent-native).

isitagentready.com results for the Statamic docs at statamic.dev before the changes: a score of 13 and Level 0. — Sad robot.

isitagentready.com results for the Statamic docs at statamic.dev after the changes: a score of 100 and Level 5. — Happy robot!

This is using the "Content Site" test, which is recommended for blogs, docs, and marketing sites since our docs don't provide an application API or require bot authentication or credentials.

Try it

Ask your agent to take a look at statamic.dev/llms.txt, statamic.dev/robots.txt, or any docs page and ask it something hard.

Remember: The docs are open source at github.com/statamic/docs. If you spot a tpyo or want to improve them, feel free to open a PR!

So what are you waiting for? There's never been a better time to put your clankers to work and have them build something awesome with Statamic. Now faster and with fewer tokens than ever.