VisibilitySearch
The actual playbook

How to get your site listed by AI.

There's no "Submit to Common Crawl" button. Common Crawl, OpenAI, Anthropic and Google don't accept manual submissions the way Google Search does. But there is a clear, well-understood playbook for getting included. Follow these six steps and your site will appear in AI training data and live retrieval within 60-90 days.

The honest reality: No company will let you "submit" a URL to AI directly. Anyone selling a paid "AI submission service" is reselling work you can do yourself in an afternoon. These six steps are the work that actually moves the needle.

Step 1 — Allow AI crawlers in your robots.txt

This is the single biggest fix. If your robots.txt blocks the bots, nothing else matters. Open your site's /robots.txt file (or create one in your site root) and confirm the following blocks are not blocked, ideally explicitly allowed:

User-agent: CCBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

If your robots.txt has Disallow: / under any of these user-agents, that bot cannot read your site. Remove those blocks. Save. Wait up to 48 hours for AI crawlers to re-fetch.

Step 2 — Make sure your homepage is crawlable

Common Crawl's bot (and most AI crawlers) are basic HTML fetchers. They do not execute JavaScript. If your homepage is a React/Vue/Next single-page app that renders content after JS executes, the crawler sees an empty shell.

Test: right-click your homepage, View Source. Search the source for your main heading text. If it's there, you're fine. If you see only <div id="root"></div> or similar, you have a problem. Either switch to server-side rendering (Next.js SSR, Nuxt, Astro) or pre-render static HTML.

Step 3 — Add schema.org structured data

Schema.org markup gives AI models a clean, machine-readable description of what your site is, who runs it, and what you do. The four schemas that matter most:

Add them as JSON-LD in the <head> of each page. Validate at Google's Rich Results Test. See our complete schema guide for copy-paste examples.

Step 4 — Get linked from already-indexed sites

Common Crawl seeds from high-authority domains and follows links outward. If you're not linked from anywhere already in their index, you'll never be discovered.

The minimum viable link-building plan:

  1. Add your site to relevant industry directories
  2. Submit a guest post to two industry blogs or publications
  3. Get listed on Wikidata (no notability required, unlike Wikipedia)
  4. Create profiles on LinkedIn, Crunchbase, AngelList, GitHub (whichever apply) - all are crawled heavily
  5. If you qualify for Wikipedia notability, work towards a Wikipedia page (huge AI multiplier)

Step 5 — Submit your sitemap to search engines

Common Crawl pulls hints from public web search indexes. Sites well-indexed in Google and Bing tend to be well-indexed by Common Crawl too. There's no direct "submit to Common Crawl" form, but submitting to search engines indirectly helps.

Step 6 — Wait, then re-check

Common Crawl publishes roughly monthly. New crawl data takes 4-8 weeks to be released, plus another few weeks before AI companies process it into training updates. Realistic timeline: 60-90 days from doing the work to seeing your domain in our AI Visibility Check at a higher score.

Re-run the AI Visibility Check quarterly. Track your score over time. The trend matters more than the absolute number.

Advertisement

Optional: directly contacting Common Crawl

Common Crawl is a small non-profit. They do not have a submission system, but they do read email at info@commoncrawl.org. If your site is genuinely high-value content that's somehow missed their seed list, a polite email noting your domain and the value of including it has occasionally led to additions. Do not spam them - they are a tiny team. This route is for exceptional cases only.

Want personalised guidance?

Drop your domain and email below. We'll run the check, review your specific results, and send you a one-page tailored action plan within 48 hours. No charge, no upsell.