# Markdown Pages for AI: Do AI Crawlers Read the .md Version of Your Page?
> I watched Claude open the .md version of a web page. Here is what a Markdown page is, which AI bots fetch it, what the logs show, and how to set it up.
By Michael Zacs · Getting Recommended · Published Nov 4, 2026 · Last updated Nov 4, 2026
Source: https://exibran.com/blog/markdown-pages-for-ai
## Quick answer
Some AI crawlers do fetch the .md (Markdown) version of a page when one exists. In one site's logs, GPTBot sent 35% of its requests to .md pages and OAI-SearchBot 23%. No bot used the Accept: text/markdown header. No study I found shows Markdown raises how often AI recommends you, so treat it as a cheap experiment and measure it with your AI Pick Rate.
## Key takeaways
- A Markdown page is a plain-text copy of a web page, usually at the same address with .md on the end.
- A .md copy can appear in search results as its own entry. I watched Claude find one and open it to read a definition.
- In one site's logs, GPTBot and OAI-SearchBot fetched .md pages often. ChatGPT-User and PerplexityBot almost never did, and no bot sent the Accept: text/markdown header.
- llms.txt is read far less than .md pages. Ahrefs found that 97% of the llms.txt files in a 137,210-domain study got zero requests in a month.
- Being fetched is not being recommended. Google says its Search does not use Markdown or llms.txt files, and no study shows a lift in AI citations.
- Set it up in an hour or less, keep it generated from the same source as the page, and check your logs after 30 days.
While building Radar, my AI visibility tool, I asked Claude to research how other tools define **AI citation rate**.
Its web search listed geotoolbox.ai twice: once as a Markdown file, once as the normal blog post. Claude opened the Markdown one to read the definition.
A `.md` address is a plain-text copy of a web page. It was sitting in the search results as its own entry, and I clicked through to confirm it.
That left me with three questions. What is that file? Do AI crawlers really read it? Does my own site have one?
In this guide, you'll understand what a Markdown page is, what the logs show AI bots doing with it, whether it helps you get recommended, how to set it up with code, and what I already run on Exibran.com.
## What Is a Markdown (.md) Page for AI Crawlers?
A Markdown page is the content of a web page as clean text. It has the title, headings, paragraphs, tables and links, and none of the menus, scripts or pop-ups around them.
There are three ways to offer it, and people often mix them up:
| Method | What It Is | Example |
| --- | --- | --- |
| **.md URL** | A separate Markdown copy at its own address | `yoursite.com/blog/page.md` |
| **Discovery tag** | A line in the page's HTML head that points to that copy | `` |
| **Accept header** | The same address returns Markdown when the visitor asks for it | `Accept: text/markdown` |
Keep them apart. A bot can fetch `/page.md` without ever sending an Accept header, and the logs below show that is what happens.
## What Did I Actually See Claude Do?
I saw two things, and neither was a crawl.
**The search result.** The title of the `.md` entry began with a `#`, which is how a Markdown heading looks as raw text. So the search had found the Markdown copy and listed it as its own result, beside the normal page.
When I opened it, it appeared this way:
**The fetch.** A crawler is a bot that visits sites on its own to collect pages. This was one page, opened because I had asked Claude a question. It is the same kind of request as ChatGPT-User or Claude-User, the bots that fetch a page when a person asks.
Claude read the quattr.com page as a normal web page. Not every page has a Markdown copy.
I do not know why Claude chose the Markdown copy over the normal post. Clean text with no menus or pop-ups is a likely reason, but I cannot confirm it.
This is one example, not a study. It does show that a `.md` page can be found through search, not only by bots following links.
> **I Will Keep This Updated**
>
> I will update this article whenever I notice or observe anything related, from any AI search. The updated date at the top shows the last change.
## Do AI Crawlers Really Fetch .md Pages?
Yes, some do, on sites that have them. The best numbers I found come from [Dries Buytaert's server logs](https://dri.es/node/6121).
He added Markdown copies to his site and counted 58,279 AI crawler requests over a month.
*Image: Bar chart of the share of each AI bot's requests that went to .md pages: GPTBot 34.8%, OAI-SearchBot 22.7%, Amazonbot 10.9%, Claude-User 7.4%, Meta AI 5.4%, ClaudeBot 2.1%, PerplexityBot 0.2%, ChatGPT-User 0.1%, Bytespider 0%, CCBot 0%*
*Share of each bot's requests that went to a Markdown page. One website, one month.*
What the logs show:
- **Training and search bots asked for .md pages.** About 8.5% of all requests were for Markdown files. GPTBot and OAI-SearchBot led.
- **Bots that fetch for a user were mixed.** ChatGPT-User sent 0.1% of its requests to `.md` pages. Claude-User sent 7.4%, but it made only 94 requests in total, so read that number with care.
- **No bot used the Accept header.** Zero of the 58,279 requests asked for `text/markdown` on the normal address. Bots found the Markdown through the `.md` address and the discovery link.
- **Traffic did not drop.** Bots fetched both versions, so crawler traffic rose about 7%.
- **Fetching is not citing.** He measured 1,241 pages fetched by AI crawlers for every citation sent back to his site.
This is one website, not a measure of the internet. Google's John Mueller reported something different on his own test sites, where only SEO tools asked for Markdown, according to [Search Engine Journal](https://www.searchenginejournal.com/googles-mueller-shares-their-experience-with-markdown-for-ai-seo/587671/). Sites differ, so your own logs are the real test.
## What About llms.txt, Does It Do More?
No. It does less. llms.txt is one index file at the root of your site, and it is read far less often than individual `.md` pages.
- **Ahrefs** checked 137,210 domains. About 28% had an llms.txt file, and 97% of those got zero requests in the month. AI search bots made only about 1% of the requests the files did get. See [the Ahrefs study](https://ahrefs.com/blog/llmstxt-study/).
- **Dries Buytaert** got 52 requests for his llms.txt in a month. All came from SEO audit tools, none from AI systems.
If you only have time for one, the logs favor `.md` pages over llms.txt. Neither is proven to help you get recommended.
## Will Markdown Pages Raise Your AI Pick Rate?
There is no evidence of it. The studies count requests. They do not measure whether a brand was recommended afterward.
Google says its Search does not use special files, including Markdown and llms.txt, and that they neither help nor harm your rankings. Read that in [Google's guide to generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide). I found no public promise from other AI apps either.
A bot fetching your page and an AI recommending you are different events. Recommending depends on what AI reads about you across the web.
In my 405-answer test, AI cited community threads, video, and review and comparison sites beside the tools' own websites.
Even when an answer cited a tool's own site, AI picked that tool only 13% to 34% of the time. The full numbers are in my guide to [AI Pick Rate](/blog/ai-pick-rate).
So treat Markdown pages as delivery, not as a fix. If AI picks a competitor over you, a `.md` file will not change that.
## What Are the Risks of Serving Markdown Pages?
Google and Bing staff have both pushed back on separate Markdown pages for AI, mostly over crawl load and duplicate content. The concerns are real, and each has a fix:
| Concern | What Can Happen | What to Do |
| --- | --- | --- |
| **Duplicate content** | A search engine finds the `.md` file and lists it as a separate page. I saw this in Claude's search results. Some site owners report `.md` files showing as crawled but not indexed | Link back to the original page inside the file, and send a noindex header on `.md` files |
| **Cloaking** | Serving bots something different from what people see. Google defines cloaking as doing this to [manipulate rankings](https://developers.google.com/search/docs/essentials/spam-policies#cloaking) | Generate the `.md` copy from the same source as the page, so both always say the same thing |
| **Crawl load** | Bots fetch both versions, so total requests can go up | Cache the files. It only matters if your server is already under strain |
| **Stale copies** | The `.md` file keeps old pricing after you update the page | Rebuild the `.md` file every time the page changes. Never edit it by hand |
| **Cache mix-ups** | With the Accept header, a cache serves Markdown to a browser, or the reverse | Send `Vary: Accept`, and test both versions after each change |
## How Do You Set Up Markdown Pages for AI Crawlers?
Start with the first two steps. They work on any site and are what the logs show bots using.
### How Do You Add a .md URL for Each Page?
Generate the Markdown from the same source as your page, publish it at the page address plus `.md`, and return the right content type.
For example, my site runs on Astro, and one file creates the `.md` copy of every post:
```ts
// src/pages/blog/[slug].md.ts
export async function GET({ props }) {
const markdown = await makeMarkdown(props.post); // same source as the page
return new Response(markdown, {
headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
});
}
```
Put the title, a one-line summary, the author, the dates and the original page address at the top of each file. Without the address, a bot that cites the `.md` file has nothing to link back to.
On WordPress, a plugin does this for you. [Serve Markdown](https://wordpress.org/plugins/serve-md/) and [Markdown for Agents and Statistics](https://wordpress.org/plugins/markdown-for-agents-and-statistics/) both add `.md` addresses.
### How Do You Tell Bots Where the .md Page Is?
Add this line to the `` of the normal page:
```html
```
The tag only points to the file. You still have to create the file.
### How Do You Serve Markdown From the Same URL?
This is the Accept header method. The normal address returns Markdown when the request asks for it. It is optional. No AI crawler used it in the logs above, though coding agents such as Claude Code do.
**On Cloudflare:** turn on Markdown for Agents in the AI Crawl Control section of your dashboard.
Cloudflare converts your page to Markdown for any request that asks for it, and you do not create any files. Cloudflare lists it for paid plans, so check yours first. See the [Cloudflare documentation](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/).
**On Next.js:** add a rewrite that matches the Accept header, then a route that returns Markdown. This follows the pattern in [Vercel's guide](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation):
```js
// next.config.js
module.exports = {
async rewrites() {
return {
beforeFiles: [
{
source: '/blog/:slug*',
has: [
{ type: 'header', key: 'accept', value: '(.*)text/markdown(.*)' },
],
destination: '/blog/md/:slug*',
},
],
};
},
};
```
**On Astro:** a static build cannot read request headers. You can publish `.md` files and the discovery tag as static files. For the Accept header, you need request-time code, such as a Cloudflare Worker.
With this method, send these headers on the Markdown response:
```
Content-Type: text/markdown; charset=utf-8
Vary: Accept
```
### How Do You Stop Search Engines Treating the .md File as a Duplicate?
You can send two headers on every `.md` file. A `noindex` header asks search engines to keep the file out of their results. A canonical `Link` header says the normal page is the original.
On Cloudflare, add a `_headers` file to your site's public folder. This is what I use:
```
/blog/*.md
Content-Type: text/markdown; charset=utf-8
X-Robots-Tag: noindex
Link: ; rel="canonical"
```
The `:splat` part fills in the page name, so one rule covers every post. On other hosts, find where your host sets response headers and send the same two.
Noindex is an instruction for search engines. I have not found out whether AI bots follow it, so I treat it as a search-results fix only.
### How Do You Test That It Works?
Run these in a terminal and swap in your own address:
```
# 1. The .md address returns Markdown with the duplicate-content headers
curl -I https://yoursite.com/blog/page.md
# 2. The normal page still returns HTML
curl -I https://yoursite.com/blog/page
# 3. The discovery tag is in the page head
curl -s https://yoursite.com/blog/page | grep -i 'text/markdown'
# 4. Same-URL Markdown works (only if you set it up)
curl -I -H "Accept: text/markdown" https://yoursite.com/blog/page
```
The first command should show `200`, `text/markdown`, `x-robots-tag: noindex` and a `link` header with `rel="canonical"`. If it shows `text/html` or `404`, your route is falling back to the normal page.
## How Do You Check Whether AI Bots Fetch Your .md Pages?
Only your own logs can tell you. Do this after 30 days:
1. **Open your access logs.** Use your host's log view, or the analytics in your CDN.
1. **Filter for addresses ending in .md.**
1. **Group the requests by user agent.** Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and PerplexityBot.
1. **Compare with the same bots' requests to the normal pages.** That gives you each bot's `.md` share, like the chart above.
1. **Treat user agents as claims.** Anyone can fake a name. Where a bot operator publishes its IP ranges, check against them.
Then measure your [AI Pick Rate](/blog/ai-pick-rate) before you publish the `.md` pages, and again six weeks later with the same questions. That is the only way to know whether it moved anything for you.
## Does Exibran Have Markdown Pages Too?
Yes. Every post on Exibran.com has one.
- **A `.md` copy of every post.** Add `.md` to any article address. This one is at `exibran.com/blog/markdown-pages-for-ai.md`. Tables stay as tables, and images are described in words.
- **A discovery tag.** Each page head points to its copy, as shown above.
- **A link back.** Every file starts with the original page address, and sends a canonical link and a noindex header.
- **A site index.** `/llms.txt` lists the pages, and `/llms-full.txt` holds the full text.
I do not use the Accept header method, because the logs show no AI crawler sending it.
I also do not claim this makes AI recommend me. It costs almost nothing, so I treat it as an experiment and judge it by results.
## What Should You Do First?
Do the cheap thing, then spend your time where AI picks a winner. Match your problem to the fix:
| What You See | Do This First |
| --- | --- |
| AI cannot or does not read your pages | Check that bots are allowed in, then add `.md` pages. The steps are in my [AI Pick Rate](/blog/ai-pick-rate) guide |
| AI reads your pages but picks a competitor | Find the review sites, directories and comparison pages it cites for your category, and get listed there |
| AI mentions you with a warning attached | Trace the objection to its source and fix it there |
| You do not know which of these is true | Measure your AI Pick Rate first |
That last row is where I start with clients. My service covers the real answers AI gives your buyers, a gap report of the sources AI trusts, directory listings written and live, and comparison, alternatives and use-case pages with structured data.
After 60 days, I run the same questions again so you can see what changed. See the [services page](/services).
Let me ask you: Do you have an `.md` file on your website? Have you ever seen an AI search engine serve one?
## Frequently asked questions
### Do AI crawlers read .md pages?
Some do. In one site's logs, GPTBot sent about 35% of its requests to .md pages and OAI-SearchBot about 23%, while ChatGPT-User and PerplexityBot sent almost none. It is one site over one month, so check your own logs.
### Is llms.txt the same as a Markdown page?
No. llms.txt is one index file at the root of your site. A Markdown page is the full content of one page in plain text. Logs show bots fetch .md pages far more often than llms.txt.
### Do I need Accept text/markdown, or is a .md URL enough?
A .md URL is enough to start. In the log study I found, no AI crawler sent the Accept: text/markdown header, and the .md pages were found through the .md address and a discovery link in the page head. Coding agents such as Claude Code do use the header.
### Is serving Markdown to AI bots cloaking?
Not if both versions say the same thing. Google defines cloaking as showing different content to search engines and people in order to manipulate rankings. Generate the .md copy from the same source as the page so the two never disagree.
### Will .md pages cause duplicate content problems?
They can. Some site owners report .md pages showing as crawled but not indexed in Google Search Console. Link back to the original page inside the .md file, and where your host allows it, send a noindex header on .md files.
### Will serving Markdown get me recommended by ChatGPT?
There is no evidence of that yet. The studies measure which bots request the files, not whether a brand is recommended. Measure your AI Pick Rate before and after so you know whether it changed anything for you.
## Sources
1. [Markdown, llms.txt and AI crawlers](https://dri.es/node/6121), Dries Buytaert
2. [We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read](https://ahrefs.com/blog/llmstxt-study/), Ahrefs
3. [Guide to optimizing for generative AI features in Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), Google Search Central
4. [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), Cloudflare
5. [Google's Mueller shares their experience with Markdown for AI SEO](https://www.searchenginejournal.com/googles-mueller-shares-their-experience-with-markdown-for-ai-seo/587671/), Search Engine Journal
6. [Google and Bing don't recommend separate Markdown pages for LLMs](https://searchengineland.com/google-bing-dont-recommend-seperate-markdown-pages-for-llms-468365), Search Engine Land
7. [Making agent-friendly pages with content negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation), Vercel