Skip to main content

Documentation Index

Fetch the complete documentation index at: https://motherfuckingsideproject.com/docs/llms.txt

Use this file to discover all available pages before exploring further.

​
Troubleshooting

Common issues when setting up AgentXP and how to fix them.

​
I get HTML instead of markdown

Your request is not being detected as an AI agent. Check two things:
  1. You’re sending the right header. Use Accept: text/markdown in your request:
curl -H "Accept: text/markdown" http://localhost:3000/
  1. Your middleware is running. Verify middleware.ts exists in the correct location — either your project root or src/middleware.ts if you use a src/ directory. Next.js only loads middleware from these two paths.
If you’re testing with a custom bot, add it to customAgents in your config. See Custom agents.

​
Markdown output is empty or minimal

This happens when your page content is rendered client-side with JavaScript. AgentXP fetches the page HTML from the server, so content loaded via useEffect, useState, or client-side API calls won’t appear in the output. Fix: Make sure your pages use Server Components (the default in Next.js App Router) or static generation. Content must be present in the server-rendered HTML. You can verify what the server returns by viewing your page’s HTML source: 
curl http://localhost:3000/your-page | head -100
If the <main> section is mostly empty <div> tags with no text content, the page is client-rendered and AgentXP can’t extract content from it. AgentXP strips common boilerplate elements (nav, header, footer, aside, script), but third-party widgets injected inside your <main> element may survive extraction. Fix: Add their CSS selectors to excludeSelectors:
middleware.ts
export default withAgentXP({
  transform: {
    excludeSelectors: [
      '.cookie-consent',
      '#intercom-container',
      '.crisp-client',
      '#drift-widget',
      '#hubspot-messages-iframe-container',
      '.chat-widget',
    ],
  },
})
See Common widget selectors for a full reference.

​
Content is cut off / truncated

AgentXP limits output to 8,000 tokens by default (roughly 32,000 characters). For long pages like documentation or legal content, increase the limit:
middleware.ts
export default withAgentXP({
  transform: {
    maxTokens: 16000,
  },
})
Truncated output ends with a note indicating the content was cut. Increase maxTokens or use contentSelector to target only the main content area, which reduces output size by skipping sidebar and footer content.

​
Middleware causes an infinite loop or timeout

This happened in versions before 0.1.1 when the middleware’s internal fetch to your origin triggered itself recursively. Fix: Update to the latest version. The middleware now sends an x-agentxp-internal: 1 header on internal fetches and skips processing when it sees that header. If you’re on the latest version and still seeing loops, check that no other middleware or proxy is stripping the x-agentxp-internal header from internal requests.

​
/llms.txt returns 404

The /llms.txt route is handled by the middleware, not by a file in your public/ directory. If it returns 404, the middleware is not running for that path. Fix: Check your middleware matcher. If you have a custom config.matcher in middleware.ts, make sure it includes /llms.txt:
middleware.ts
export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
}
The default matcher (when you don’t export one) catches all routes, including /llms.txt.

​
Existing middleware stops working after adding AgentXP

Pass your existing middleware as the second argument to withAgentXP():
middleware.ts
import { withAgentXP } from '@reaganhsu/agentxp-next'
import { clerkMiddleware } from '@clerk/nextjs/server'

export default withAgentXP({}, clerkMiddleware())
AgentXP intercepts agent requests only. For regular browser requests, it calls your middleware unchanged. If you need to run your middleware first (for example, to set up auth context that AgentXP needs), reverse the order and call AgentXP from within your middleware instead.

​
Wrong content is extracted

If AgentXP is extracting sidebar content, navigation text, or other non-main content, set contentSelector to target your main content area:
middleware.ts
export default withAgentXP({
  transform: {
    contentSelector: 'main',
  },
})
Common selectors: main, article, #content, .content, [role="main"]. Pick the one that wraps your page’s actual content in your HTML.

​
Response is slow (> 2 seconds)

AgentXP fetches your page HTML internally before transforming it. If the response is slow:
  1. Your page itself is slow. The internal fetch has a 5-second timeout. If your page takes 3 seconds to render, the agent response takes at least 3 seconds plus transformation time.
  2. The page is very large. Pages with 50,000+ characters of HTML take longer to transform. Set contentSelector to reduce the amount of HTML processed.
Transformation alone takes under 50ms for typical pages. The bottleneck is almost always the origin page load time.