Open-source · Apache 2.0 · Vendor-neutral
Detect AI crawlers. Verify Web Bot Auth signatures. Apply RSL 1.0 policy. Issue HTTP 402 with a structured payment offer. All in vendor-neutral code you can audit — one package, every Node framework, every edge runtime.
The AI-crawler-monetization space consolidated around four standards in 2025–2026. CrawlerToll is the vendor-neutral OSS implementation that sits underneath all of them.
RSL Technical Steering Committee
Really Simple Licensing — robots.txt vocabulary for AI-licensing terms. Backed by Reddit, Yahoo, People Inc., Medium, Quora, O'Reilly, Stack Overflow, and Cloudflare.
IETF (draft-meunier-web-bot-auth-architecture-05)
HTTP Message Signatures over Ed25519 keys served from /.well-known/http-message-signatures-directory. Implemented by Cloudflare, AWS WAF, OpenAI, and Google.
Cloudflare pay-per-crawl + x402
Payment Required response shape — Cloudflare ships >1 billion HTTP 402s per day. Structured Crawler-Price / Crawler-Price-Rail / Link headers.
x402 Foundation (Linux Foundation)
Coinbase + LF agent-payment rail. Founding coalition: AWS, Anthropic, Circle, Cloudflare, Google, Microsoft, Stripe, Visa, Mastercard, NEAR, Chainlink.
Same decision engine across every adapter. Pick your framework — the wiring is one line.
import express from "express";
import { crawlertoll } from "@crawlertoll/express";
const app = express();
app.use(crawlertoll({
offer: { rail: "x402", priceMicros: 5000, currency: "USD" },
contextLicenseUrl: "https://example.com/.well-known/context-license.json",
}));
app.get("/", (req, res) => res.send("hello"));
app.listen(3000);The core decides; the adapters translate. Same RSL parser, same Web Bot Auth verifier, same 402 builder — across every Node framework, every edge runtime.
Framework-agnostic. Bot catalogue, Web Bot Auth, HTTP 402, RSL 1.0 parser.
Idiomatic Fastify plugin. Decorates request.crawlertoll. Fastify 4 + 5.
One package, four runtimes: Cloudflare Workers, Bun, Deno, Vercel Edge.
Drop into middleware.ts. Edge runtime. Next 14 + 15.
Settlement-rail adapter for x402. Quote builder, header parser, facilitator verify.
Publisher SDK + CLI. npx crawlertoll init scaffolds context-license.json in 60s.
JSON Schema validator + TypeScript types for context-license.json.
Fork-and-deploy templates pre-wired with the right adapter, a sample RSL 1.0 policy, and the discovery files crawlers look for.
Hono + @crawlertoll/hono. No CF Enterprise required. wrangler deploy and you're live.
Next.js 15 + @crawlertoll/next. One click to clone-and-deploy to vercel.app.
30+ AI crawler catalogue, RSL 1.0 robots.txt, settings page. PHP 7.4+. wp.org-submission-ready.
GitHub repo →Anonymised, opt-in telemetry across CrawlerToll installs. Privacy-first. CF Worker + static dashboard.
See insights →CrawlerToll does not try to be a marketplace, a payment processor, or a metadata-vocabulary author. It is the vendor-neutral implementation that sits underneath — and ships adapters TO Cloudflare Pay Per Crawl, TollBit, Skyfire, x402, and Stripe ACP.
Apache-2.0 OSS. Pick whatever settlement rail your business already uses. Audit every line. Fork freely.
The optional insights pipeline accepts six fields only: operator, bot name, action, category, verified flag, and a clamped path segment. No IPs, no full UAs, no full URLs, no headers, no PII.
Read the privacy posture →Eight published npm packages, four deploy templates, ~140 vitest tests, all Apache 2.0. Adoptable by any publisher, audited by anyone, controlled by no single vendor.
Start in 60 seconds →