Markdown Pages for AI: Do AI Crawlers Read the .md Version of Your Page?

While building Radar, my AI visibility tool, I asked Claude to research how other tools define AI citation rate.

Its web search listed geotoolbox.ai twice: once as a Markdown file, once as the normal blog post. Claude opened the Markdown one to read the definition.

A .md address is a plain-text copy of a web page. It was sitting in the search results as its own entry, and I clicked through to confirm it.

That left me with three questions. What is that file? Do AI crawlers really read it? Does my own site have one?

In this guide, you'll understand what a Markdown page is, what the logs show AI bots doing with it, whether it helps you get recommended, how to set it up with code, and what I already run on Exibran.com.

What Is a Markdown (.md) Page for AI Crawlers?

A Markdown page is the content of a web page as clean text. It has the title, headings, paragraphs, tables and links, and none of the menus, scripts or pop-ups around them.

There are three ways to offer it, and people often mix them up:

Keep them apart. A bot can fetch /page.md without ever sending an Accept header, and the logs below show that is what happens.

What Did I Actually See Claude Do?

I saw two things, and neither was a crawl.

The search result. The title of the .md entry began with a #, which is how a Markdown heading looks as raw text. So the search had found the Markdown copy and listed it as its own result, beside the normal page.

When I opened it, it appeared this way:

The fetch. A crawler is a bot that visits sites on its own to collect pages. This was one page, opened because I had asked Claude a question. It is the same kind of request as ChatGPT-User or Claude-User, the bots that fetch a page when a person asks.

Claude read the quattr.com page as a normal web page. Not every page has a Markdown copy.

I do not know why Claude chose the Markdown copy over the normal post. Clean text with no menus or pop-ups is a likely reason, but I cannot confirm it.

This is one example, not a study. It does show that a .md page can be found through search, not only by bots following links.

Do AI Crawlers Really Fetch .md Pages?

Yes, some do, on sites that have them. The best numbers I found come from Dries Buytaert's server logs (opens in a new tab).

He added Markdown copies to his site and counted 58,279 AI crawler requests over a month.

What the logs show:

• Training and search bots asked for .md pages. About 8.5% of all requests were for Markdown files. GPTBot and OAI-SearchBot led.
• Bots that fetch for a user were mixed. ChatGPT-User sent 0.1% of its requests to .md pages. Claude-User sent 7.4%, but it made only 94 requests in total, so read that number with care.
• No bot used the Accept header. Zero of the 58,279 requests asked for text/markdown on the normal address. Bots found the Markdown through the .md address and the discovery link.
• Traffic did not drop. Bots fetched both versions, so crawler traffic rose about 7%.
• Fetching is not citing. He measured 1,241 pages fetched by AI crawlers for every citation sent back to his site.

This is one website, not a measure of the internet. Google's John Mueller reported something different on his own test sites, where only SEO tools asked for Markdown, according to Search Engine Journal (opens in a new tab). Sites differ, so your own logs are the real test.

What About llms.txt, Does It Do More?

No. It does less. llms.txt is one index file at the root of your site, and it is read far less often than individual .md pages.

• Ahrefs checked 137,210 domains. About 28% had an llms.txt file, and 97% of those got zero requests in the month. AI search bots made only about 1% of the requests the files did get. See the Ahrefs study (opens in a new tab).
• Dries Buytaert got 52 requests for his llms.txt in a month. All came from SEO audit tools, none from AI systems.

If you only have time for one, the logs favor .md pages over llms.txt. Neither is proven to help you get recommended.

Will Markdown Pages Raise Your AI Pick Rate?

There is no evidence of it. The studies count requests. They do not measure whether a brand was recommended afterward.

Google says its Search does not use special files, including Markdown and llms.txt, and that they neither help nor harm your rankings. Read that in Google's guide to generative AI features (opens in a new tab). I found no public promise from other AI apps either.

A bot fetching your page and an AI recommending you are different events. Recommending depends on what AI reads about you across the web.

In my 405-answer test, AI cited community threads, video, and review and comparison sites beside the tools' own websites.

Even when an answer cited a tool's own site, AI picked that tool only 13% to 34% of the time. The full numbers are in my guide to AI Pick Rate.

So treat Markdown pages as delivery, not as a fix. If AI picks a competitor over you, a .md file will not change that.

What Are the Risks of Serving Markdown Pages?

Google and Bing staff have both pushed back on separate Markdown pages for AI, mostly over crawl load and duplicate content. The concerns are real, and each has a fix:

How Do You Set Up Markdown Pages for AI Crawlers?

Start with the first two steps. They work on any site and are what the logs show bots using.

How Do You Add a .md URL for Each Page?

Generate the Markdown from the same source as your page, publish it at the page address plus .md, and return the right content type.

For example, my site runs on Astro, and one file creates the .md copy of every post:

// src/pages/blog/[slug].md.ts
export async function GET({ props }) {
  const markdown = await makeMarkdown(props.post); // same source as the page
  return new Response(markdown, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  });
}

Put the title, a one-line summary, the author, the dates and the original page address at the top of each file. Without the address, a bot that cites the .md file has nothing to link back to.

On WordPress, a plugin does this for you. Serve Markdown (opens in a new tab) and Markdown for Agents and Statistics (opens in a new tab) both add .md addresses.

How Do You Tell Bots Where the .md Page Is?

Add this line to the <head> of the normal page:

<link rel="alternate"
      type="text/markdown"
      href="https://yoursite.com/blog/page.md" />

The tag only points to the file. You still have to create the file.

How Do You Serve Markdown From the Same URL?

This is the Accept header method. The normal address returns Markdown when the request asks for it. It is optional. No AI crawler used it in the logs above, though coding agents such as Claude Code do.

On Cloudflare: turn on Markdown for Agents in the AI Crawl Control section of your dashboard.

Cloudflare converts your page to Markdown for any request that asks for it, and you do not create any files. Cloudflare lists it for paid plans, so check yours first. See the Cloudflare documentation (opens in a new tab).

On Next.js: add a rewrite that matches the Accept header, then a route that returns Markdown. This follows the pattern in Vercel's guide (opens in a new tab):

// next.config.js
module.exports = {
  async rewrites() {
    return {
      beforeFiles: [
        {
          source: '/blog/:slug*',
          has: [
            { type: 'header', key: 'accept', value: '(.*)text/markdown(.*)' },
          ],
          destination: '/blog/md/:slug*',
        },
      ],
    };
  },
};

On Astro: a static build cannot read request headers. You can publish .md files and the discovery tag as static files. For the Accept header, you need request-time code, such as a Cloudflare Worker.

With this method, send these headers on the Markdown response:

Content-Type: text/markdown; charset=utf-8
Vary: Accept

How Do You Stop Search Engines Treating the .md File as a Duplicate?

You can send two headers on every .md file. A noindex header asks search engines to keep the file out of their results. A canonical Link header says the normal page is the original.

On Cloudflare, add a _headers file to your site's public folder. This is what I use:

/blog/*.md
  Content-Type: text/markdown; charset=utf-8
  X-Robots-Tag: noindex
  Link: <https://yoursite.com/blog/:splat>; rel="canonical"

The :splat part fills in the page name, so one rule covers every post. On other hosts, find where your host sets response headers and send the same two.

Noindex is an instruction for search engines. I have not found out whether AI bots follow it, so I treat it as a search-results fix only.

How Do You Test That It Works?

Run these in a terminal and swap in your own address:

# 1. The .md address returns Markdown with the duplicate-content headers
curl -I https://yoursite.com/blog/page.md

# 2. The normal page still returns HTML
curl -I https://yoursite.com/blog/page

# 3. The discovery tag is in the page head
curl -s https://yoursite.com/blog/page | grep -i 'text/markdown'

# 4. Same-URL Markdown works (only if you set it up)
curl -I -H "Accept: text/markdown" https://yoursite.com/blog/page

The first command should show 200, text/markdown, x-robots-tag: noindex and a link header with rel="canonical". If it shows text/html or 404, your route is falling back to the normal page.

How Do You Check Whether AI Bots Fetch Your .md Pages?

Only your own logs can tell you. Do this after 30 days:

1. Open your access logs. Use your host's log view, or the analytics in your CDN.
2. Filter for addresses ending in .md.
3. Group the requests by user agent. Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and PerplexityBot.
4. Compare with the same bots' requests to the normal pages. That gives you each bot's .md share, like the chart above.
5. Treat user agents as claims. Anyone can fake a name. Where a bot operator publishes its IP ranges, check against them.

Then measure your AI Pick Rate before you publish the .md pages, and again six weeks later with the same questions. That is the only way to know whether it moved anything for you.

Does Exibran Have Markdown Pages Too?

Yes. Every post on Exibran.com has one.

• A .md copy of every post. Add .md to any article address. This one is at exibran.com/blog/markdown-pages-for-ai.md. Tables stay as tables, and images are described in words.
• A discovery tag. Each page head points to its copy, as shown above.
• A link back. Every file starts with the original page address, and sends a canonical link and a noindex header.
• A site index. /llms.txt lists the pages, and /llms-full.txt holds the full text.

I do not use the Accept header method, because the logs show no AI crawler sending it.

I also do not claim this makes AI recommend me. It costs almost nothing, so I treat it as an experiment and judge it by results.

What Should You Do First?

Do the cheap thing, then spend your time where AI picks a winner. Match your problem to the fix:

That last row is where I start with clients. My service covers the real answers AI gives your buyers, a gap report of the sources AI trusts, directory listings written and live, and comparison, alternatives and use-case pages with structured data.

After 60 days, I run the same questions again so you can see what changed. See the services page.

Let me ask you: Do you have an .md file on your website? Have you ever seen an AI search engine serve one?