Serve Markdown to Agents with Content Negotiation

Introduction

I wanted a simple thing:

Serve the same blog URL to both humans and AI agents - but give them what they actually need.

• Humans → HTML (nice, styled, readable)
• Agents → Markdown (clean, lightweight, no nonsense)

No duplicate routes. No overengineering. Just one URL doing the right thing.

---

The Problem

AI agents are everywhere now. But here’s the issue:

Most websites are still built only for humans.

HTML is: - noisy

• bloated
• full of layout junk

Agents don’t care about your navbar, animations, or CSS.

They just want:

clean, structured, token-efficient content

Markdown fits that perfectly.

---

Why even bother?

Short answer: tokens + clarity

Long answer:

• HTML wastes tokens (a lot)
• Markdown is predictable and easy to parse
• Agents perform better with less noise

So instead of building separate endpoints like:

/blog/html
/blog/md

Why not just:

same URL → different response based on Accept header

---

The Approach

At some point it hit me:

“I already write in Markdown… why am I converting it away just to serve it?”

So I stopped doing that.

• Agents → raw Markdown
• Humans → rendered HTML

---

Implementation (Next.js)

This is handled in proxy.ts (edge layer) - not inside page components.

Markdown for /blogs

if (pathname === '/blogs' && chosen === 'text/markdown') {
  const res = NextResponse.rewrite(new URL(serverEnvConfig.BLOGS_INDEX_MD_URL));

  res.headers.set('Content-Type', 'text/markdown; charset=utf-8');
  res.headers.set('Vary', 'Accept');

  return res;
}

Markdown for individual blogs

const blogIdMatch = pathname.match(/^\/blogs\/i\/(\d+)$/);

if (blogIdMatch && chosen === 'text/markdown') {
  const id = Number(blogIdMatch[1]);

  const data = await fetch(serverEnvConfig.BLOGS_INDEX_JSON_URL).then((res) =>
    res.json()
  );
  const blogEntry = data.find((entry) => entry.id === id);

  if (!blogEntry?.mdUrl) {
    return NextResponse.rewrite(new URL('/404', req.url), { status: 404 });
  }

  const res = NextResponse.rewrite(new URL(blogEntry.mdUrl));

  res.headers.set('Content-Type', 'text/markdown; charset=utf-8');
  res.headers.set('Vary', 'Accept');

  return res;
}

---

Markdown Index (/blogs)

This is where things get interesting.

/blogs itself also supports Markdown.

So instead of scraping pages, agents can fetch everything in one request.

Generation

export function jsonToFlatMarkdown(blogs: BlogPost[]): string {
  const header = `# Sivothayan's Blogs

This is a collection of all the blogs that I have written. You can find the latest blogs at:

👉 [Read as Markdown (Accept: text/markdown)](https://sivothayan.com/blogs)  
👉 [Read as HTML (Accept: text/html)](https://sivothayan.com/blogs)

This resource supports HTTP content negotiation.

- For Markdown: send header \`Accept: text/markdown\`
- For HTML: send header \`Accept: text/html\`

Example:
\`curl -H "Accept: text/markdown" https://sivothayan.com/blogs\`

---
`;

  const content = blogs
    .filter((b) => b.isPublished)
    .map((blog) => {
      const tags = blog.tags.map((t) => `\`${t}\``).join(', ');

      return `## ${blog.title}

- **Date:** ${blog.date}
- **Read Time:** ${blog.readTime} min
- **Language:** ${blog.language}
- **Tags:** ${tags}

${blog.description}

👉 [Read as Markdown (Accept: text/markdown)](https://sivothayan.com/blogs/i/${blog.id})  
👉 [Read as HTML (Accept: text/html)](https://sivothayan.com/blogs/i/${blog.id})`;
    })
    .join('\n\n---\n\n');

  return `${header}\n${content}\n`;
}

---

Metadata (Next.js)

You also need to advertise Markdown support.

alternates: {
  canonical: `${site}/blogs/i/${id}`,
  types: {
    'text/markdown': `${site}/blogs/i/${id}`,
    'application/rss+xml': `${site}/rss.xml`,
    'application/atom+xml': `${site}/feed.xml`,
  },
}

Without this, agents won’t know Markdown is available.

---

Testing

If you don’t test this, you’re just guessing.

Markdown

curl -H "Accept: text/markdown" https://sivothayan.com/blogs/i/1

HTML

curl https://sivothayan.com/blogs/i/1

Headers

curl -I -H "Accept: text/markdown" https://sivothayan.com/blogs/i/1

You should see:

Content-Type: text/markdown; charset=utf-8
Vary: Accept

---

Content Signals

Since this setup is explicitly agent-friendly, it also makes sense to be explicit about what is allowed and what is not.

Instead of relying on assumptions, I expose content usage rules directly.

# As a condition of accessing this website, you agree to abide
# by the following content signals:

# (a)  If a content-signal = yes, you may collect content for
# the corresponding use.
# (b)  If a content-signal = no, you may not collect content for
# the corresponding use.
# (c)  If the website operator does not include a content signal
# for a corresponding use, the website operator neither grants
# nor restricts permission via content signal with respect to
# the corresponding use.

# The content signals and their meanings are:

# search: building a search index and providing search results
# (e.g., returning hyperlinks and short excerpts from your
# website's contents). Search does not include providing
# AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g.,
# retrieval augmented generation, grounding, or other real-time
# use for generative AI answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS
# RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION
# DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE
# DIGITAL SINGLE MARKET.

User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

---

What actually improved?

• Agents skip HTML parsing entirely
• Less data → fewer tokens → cheaper + faster
• /blogs acts like a content index
• Same URL works for everything

At this point, /blogs is basically a read-only content API - without building one.

---

Conclusion

This isn’t some big architecture.

It’s just using the web properly.

Same resource. Different representations.

We’ve had this for years.

We just ignored it.

---

What I learned

• Accept header is massively underused
• Middleware > duplicating routes
• If your source is Markdown… just serve it
• Simpler systems age better

---

References

https://blog.cloudflare.com/markdown-for-agents/

https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/

https://contentsignals.org/

https://acceptmarkdown.com/