Serve Markdown to Agents with Content Negotiation

Introduction

I wanted one blog URL to work well for two kinds of readers:

• Humans should receive HTML.
• AI agents should receive Markdown.

The important part is that both readers should use the same canonical URL. I did not want separate routes such as /blog/html and /blog/md. I wanted the web server to return the right representation based on the request.

That is exactly what HTTP content negotiation is for.

---

Why Markdown Helps Agents

Most websites are designed for browsers and humans. That is fine, but HTML often includes a lot of extra structure that agents do not need:

• navigation
• layout wrappers
• CSS classes
• scripts
• metadata duplicated across the page

Agents usually need the content itself. Markdown is cleaner, easier to parse, and much cheaper to fit into a context window.

For one of my posts, the HTML version had far more tokens than the Markdown source because every tag, attribute, and wrapper becomes part of the input. The Markdown version was much smaller and easier to process.

You can test this yourself by copying the HTML and Markdown versions into AI Tokenizer.

---

The Basic Idea

The URL stays the same:

https://sivothayan.com/blogs/i/8

The response changes based on the Accept header:

curl -H "Accept: text/markdown" https://sivothayan.com/blogs/i/8

An agent can request Markdown. A browser can request HTML. The resource is the same, but the representation is different.

---

Markdown for the Blog Index

The blog index also supports Markdown.

When an agent requests /blogs with Accept: text/markdown, it receives a compact index of posts, metadata, tags, descriptions, and links to the Markdown and HTML representations.

That turns the blog list into a simple content index without building a separate API.

The generated Markdown index follows this shape:

export function blogsToMarkdownIndex(blogs: BlogPost[]) {
  const linkMarker = '\u{1F449}';
  const header = `# Sivothayan's Blogs

This is a collection of all the blogs that I have written. You can find the latest blogs at:

${linkMarker} [Read as Markdown (Accept: text/markdown)](https://sivothayan.com/blogs)
${linkMarker} [Read as HTML (Accept: text/html)](https://sivothayan.com/blogs)

This resource supports HTTP content negotiation.

- For Markdown: send header \`Accept: text/markdown\`
- For HTML: send header \`Accept: text/html\`

Example:
\`curl -H "Accept: text/markdown" https://sivothayan.com/blogs\`

---
`;

  const content = blogs
    .filter((blog) => blog.isPublished)
    .map((blog) => {
      const tags = blog.tags.map((tag) => `\`${tag}\``).join(', ');

      return `## ${blog.title}

- **Date:** ${blog.date}
- **Read Time:** ${blog.readTime} min
- **Language:** ${blog.language}
- **Tags:** ${tags}

${blog.description}

${linkMarker} [Read as Markdown (Accept: text/markdown)](https://sivothayan.com/blogs/i/${blog.id})
${linkMarker} [Read as HTML (Accept: text/html)](https://sivothayan.com/blogs/i/${blog.id})`;
    })
    .join('\n\n---\n\n');

  return `${header}\n${content}\n`;
}

---

Handling Markdown Requests

In my Next.js app, the negotiation happens before the page renders.

For the blog index:

if (pathname === '/blogs' && chosen === 'text/markdown') {
  const res = NextResponse.rewrite(new URL(serverEnvConfig.BLOGS_INDEX_MD_URL));

  res.headers.set('Content-Type', 'text/markdown; charset=utf-8');
  res.headers.set('Vary', 'Accept');

  return res;
}

For individual blog posts:

const blogIdMatch = pathname.match(/^\/blogs\/i\/(\d+)$/);

if (blogIdMatch && chosen === 'text/markdown') {
  const id = Number(blogIdMatch[1]);

  const data = await fetch(serverEnvConfig.BLOGS_INDEX_JSON_URL).then((res) =>
    res.json()
  );
  const blogEntry = data.find((entry) => entry.id === id);

  if (!blogEntry?.mdUrl) {
    return NextResponse.rewrite(new URL('/404', req.url), { status: 404 });
  }

  const res = NextResponse.rewrite(new URL(blogEntry.mdUrl));

  res.headers.set('Content-Type', 'text/markdown; charset=utf-8');
  res.headers.set('Vary', 'Accept');
  res.headers.set('Cache-Control', 's-maxage=60, stale-while-revalidate=86400');
  res.headers.set('Content-Signal', 'ai-train=no, search=yes, ai-input=no');

  return res;
}

The Vary: Accept header is important because caches need to know that the response can change based on the Accept request header.

---

A Note on Rewrite Headers

NextResponse.rewrite() can expose the target URL through an x-middleware-rewrite response header.

That may be fine if the target Markdown files are public, as they are in my setup. If you rewrite to private storage or internal APIs, check your deployment stack and make sure you are not leaking private URLs through headers.

For my reverse proxy setup, I hide the rewrite-related header with nginx:

location {
  proxy_hide_header x-middleware-path;
}

Related discussion:

Can I remove x-middleware-rewrite http header from the response headers when using NextResponse.rewrite?

---

Advertising Markdown Support in Metadata

The page metadata should also tell clients that Markdown is available.

For an individual blog page:

import type { Metadata } from 'next';

export const metadata: Metadata = {
  alternates: {
    canonical: `${site}/blogs/i/${id}`,
    types: {
      'text/markdown': `${site}/blogs/i/${id}`,
      'application/rss+xml': `${site}/rss.xml`,
      'application/atom+xml': `${site}/feed.xml`
    }
  }
};

For the blog index:

import type { Metadata } from 'next';

export const metadata: Metadata = {
  alternates: {
    canonical: `${site}/blogs`,
    types: {
      'text/markdown': `${site}/blogs`,
      'application/rss+xml': `${site}/rss.xml`,
      'application/atom+xml': `${site}/feed.xml`
    }
  }
};

This makes the Markdown representation easier to discover.

---

Content Signals

Since this flow is designed for agents, I also wanted the usage policy to be explicit.

For Markdown responses, I send:

Content-Signal: ai-train=no, search=yes, ai-input=no

That says the content may be used for search indexing, but not for AI training or AI input.

I also keep a robots.txt version of the same policy:

# As a condition of accessing this website, you agree to abide
# by the following content signals:

# (a)  If a content-signal = yes, you may collect content for
# the corresponding use.
# (b)  If a content-signal = no, you may not collect content for
# the corresponding use.
# (c)  If the website operator does not include a content signal
# for a corresponding use, the website operator neither grants
# nor restricts permission via content signal with respect to
# the corresponding use.

# The content signals and their meanings are:

# search: building a search index and providing search results
# (e.g., returning hyperlinks and short excerpts from your
# website's contents). Search does not include providing
# AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g.,
# retrieval augmented generation, grounding, or other real-time
# use for generative AI answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS
# RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION
# DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE
# DIGITAL SINGLE MARKET.

User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

This does not replace legal judgment, but it makes the site preference machine-readable.

---

Token Count Header

I also expose an estimated Markdown token count with x-markdown-tokens.

That helps agents decide whether they can safely process a page before loading it into a context window.

The token map is generated ahead of time:

import { readdir, readFile, writeFile, stat } from 'fs/promises';
import path from 'path';

import Tokenizer, { ModelName, models } from 'ai-tokenizer';
import * as encoding from 'ai-tokenizer/encoding';

const PUBLIC_DIR = path.join(process.cwd(), 'public');
const TOKEN_MAP_JSON_PATH = path.join(PUBLIC_DIR, 'token-map.json');

const MODEL_NAMES: ModelName[] = [
  'openai/gpt-4o',
  'openai/gpt-4o-mini',
  'anthropic/claude-3.5-sonnet',
  'anthropic/claude-3-haiku',
  'google/gemini-2.5-pro',
  'google/gemini-2.5-flash'
];

const tokenizers = Object.fromEntries(
  MODEL_NAMES.map((modelName) => {
    const model = models[modelName];
    return [modelName, new Tokenizer(encoding[model.encoding])];
  })
);

async function walk(dir: string): Promise<string[]> {
  const entries = await readdir(dir);
  const results: string[] = [];

  for (const entry of entries) {
    const fullPath = path.join(dir, entry);
    const s = await stat(fullPath);

    if (s.isDirectory()) {
      results.push(...(await walk(fullPath)));
    } else if (entry.endsWith('.md')) {
      results.push(fullPath);
    }
  }

  return results;
}

async function main() {
  const files = await walk(PUBLIC_DIR);
  const result: Record<string, Record<string, number>> = {};

  for (const file of files) {
    const content = await readFile(file, 'utf-8');
    const relativePath = path.relative(PUBLIC_DIR, file).replace(/\\/g, '/');

    result[relativePath] = {};

    for (const [modelName, tokenizer] of Object.entries(tokenizers)) {
      result[relativePath][modelName] = tokenizer.encode(content).length;
    }
  }

  await writeFile(
    TOKEN_MAP_JSON_PATH,
    JSON.stringify(result, null, 2),
    'utf-8'
  );
}

main();

Then the Worker can attach the largest estimate:

const contentType = headers.get('Content-Type') || '';

if (contentType.startsWith('text/markdown')) {
  const tokenMap = await getTokenMap(env, request);
  const key = pathname.replace(/^\/+/, '');
  const tokens = tokenMap[key];
  const tokenCount = tokens ? Math.max(...Object.values(tokens)) : 0;

  headers.set('x-markdown-tokens', tokenCount.toString());
}

return new Response(assetResponse.body, {
  status: assetResponse.status,
  headers
});

I use the maximum token count across supported tokenizers as a conservative estimate.

---

Testing

I test the Markdown response:

curl -H "Accept: text/markdown" https://sivothayan.com/blogs/i/8

I test the normal HTML response:

curl https://sivothayan.com/blogs/i/8

And I check headers:

curl -I -H "Accept: text/markdown" https://sivothayan.com/blogs/i/8

Expected headers include:

Content-Type: text/markdown; charset=utf-8
Vary: Accept
Cache-Control: s-maxage=60, stale-while-revalidate=86400
Content-Signal: ai-train=no, search=yes, ai-input=no

---

What Improved

This setup made the blog friendlier to both browsers and agents:

• Browsers still get normal HTML.
• Agents can request clean Markdown.
• The blog index works like a compact content map.
• Caches know that Accept changes the response.
• Token estimates help agents plan before reading.
• Usage preferences are visible through content signals.

The result feels less like a separate API and more like using HTTP properly.

---

Conclusion

Serving Markdown to agents does not require a new content system. If the source content is already Markdown, content negotiation is a practical way to expose it without duplicating routes.

Same URL. Different representation. Cleaner content for agents. Normal pages for humans.

That is the kind of web feature I like: simple, boring, and surprisingly useful.

---

References

• Markdown for Agents - Cloudflare Blog
• Markdown for Agents - Cloudflare Developers
• Content Signals
• Serve Markdown to agents with content negotiation
• AI Tokenizer
• Next.js GitHub Discussion: Can I remove x-middleware-rewrite?
• Next.js GitHub Discussion: Add support for robots.txt Content-Signals