Google doesn't know you exist (yet):crawling, sitemaps and getting indexed

It has never been easier to build a website. You describe what you want to an AI on Sunday afternoon, and by Sunday evening there's a real site with your name on it, a contact form and a colour palette you'd never have picked yourself. You share the link, feel brilliant, and go to bed.

A week later you type your business name into Google. Nothing. Not page one, not page five. It's as if the site doesn't exist, and as far as Google is concerned, it doesn't. Not because it's bad, but because nobody has told Google it's there.

That's what this series is for. AI tools like Lovable, Bolt, v0, Framer, Wix, ChatGPT and Claude are brilliant at building what you ask for. They're less good at building what you didn't know to ask for. Each part of So you're creating a website with AI covers one of those unasked questions, in plain English, with the exact steps to fix it.

We're starting at the very beginning, because nothing else in this series matters until Google knows where you live.

How Google actually finds a website

Picture Google as an enormous library with a very fast, very literal team of robot assistants. The main one is called Googlebot, and its whole job is wandering the internet reading pages. Getting from "my site exists" to "my site appears in results" happens in three stages:

1. Crawling. The robot visits your page and reads it. Think of it as the librarian picking your book up off the doormat.
2. Indexing. Google decides the page is worth keeping and files it in its catalogue. Your book is now on a shelf.
3. Ranking. When someone searches, Google picks which books from the shelves to hand them first.

Most advice online jumps straight to stage three. But you can't rank if you're not indexed, and you can't be indexed if you've never been crawled. So how does the robot find your door in the first place? Two ways: it follows links from pages it already knows, and it reads sitemaps that site owners hand it.

A brand-new website has no links pointing at it. Nobody has written about it, nothing connects to it, and the robot has no reason to wander past. It's a lovely shop on an island with no bridge. The sitemap is you rowing over to the librarian with a map.

You can't rank if you're not indexed, and you can't be indexed if Google has never been round.

What a sitemap actually is

Despite the name, a sitemap isn't a picture. It's a plain text file, usually called sitemap.xml, that lists every page on your site you'd like search engines to know about. It lives at the root of your domain, so for a site at example.com you'd find it at example.com/sitemap.xml. Here's a small one:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-09-25</lastmod>
  </url>
  <url>
    <loc>https://example.com/services</loc>
    <lastmod>2026-09-18</lastmod>
  </url>
  <url>
    <loc>https://example.com/contact</loc>
    <lastmod>2026-09-01</lastmod>
  </url>
</urlset>

It looks technical, but there are only two things in each entry worth caring about:

• <loc> is the full address of the page, exactly as it appears in the browser, including the https://.
• <lastmod> is the date the page last meaningfully changed. Google uses this to decide what's worth re-reading, as long as you keep it honest.

What belongs in it, and what doesn't

Your sitemap should be a list of pages you'd be happy for a stranger to land on from Google. That means pages that actually load, live on your real domain, and are the one "official" version of that content. Leave out:

• Admin, login, checkout and "thanks for your enquiry" pages.
• Old addresses that now redirect somewhere else. List the destination instead.
• Test pages, drafts and anything you've hidden from search on purpose.
• Broken pages. If it shows a 404, it doesn't belong on the map.

One sitemap can hold up to 50,000 addresses, so unless you're running a very large shop, a single file will do.

Does your site already have one?

Find out in ten seconds: type your domain followed by /sitemap.xml into your browser. If you see a list of your pages, you're off to a good start. If you get a "page not found", keep reading.

If you built on a hosted platform, there's a good chance it's already done. Wix, Squarespace, Shopify, Webflow and Framer all generate a sitemap automatically, usually at /sitemap.xml. WordPress has one built in at /wp-sitemap.xml, or /sitemap_index.xml if you use the Yoast plugin.

If your AI wrote the code for you, in tools like Lovable, Bolt, v0, Replit, Cursor or Claude Code, the odds flip. Many of these projects ship with no sitemap at all, or one the AI wrote on day one and never touched again. The good news is that the same AI can fix it. Paste this in:

Create a sitemap.xml for my website at https://www.yourdomain.com.

- Include every public page that should appear in Google, using the full https address exactly as it appears in the browser on my live domain (not a preview or localhost address).
- Leave out admin, login, checkout, thank-you and test pages, and any page that redirects.
- Give each page an accurate lastmod date.
- Put the file where it will be served at https://www.yourdomain.com/sitemap.xml.
- Make it update automatically whenever I add or remove a page.

Then create a robots.txt at https://www.yourdomain.com/robots.txt that allows all search engines and includes a line pointing to the sitemap.

robots.txt: the sign on your front door

Your sitemap says "here's everything worth seeing". Its partner file, robots.txt, says "here's where you're allowed to go". It also lives at the root of your domain, and a healthy one for a small site is almost boringly short:

User-agent: *
Allow: /

Sitemap: https://www.yourdomain.com/sitemap.xml

That Sitemap: line matters. It means any search engine that checks your front door gets handed the map straight away, whether or not you've submitted it anywhere.

The one line that hides your whole website

Here's the horror story. While a site is being built, it's common to block search engines so a half-finished version doesn't get indexed. The block looks like this:

User-agent: *
Disallow: /

That forward slash means "everything". Launch with it still in place and you've politely asked every search engine on earth to ignore you. There are two cousins to watch for too:

• A noindex tag hidden in your page code, which looks like <meta name="robots" content="noindex">. It tells Google "you can read this, but don't file it". Ask your AI to search the project for "noindex" and remove it from any page you want found.
• A platform setting. WordPress has a tickbox under Settings → Reading called "Discourage search engines from indexing this site". Other builders have a similar "hide from search engines" switch in their SEO settings. Make sure it's off.

A quick word about JavaScript

Lots of AI-built sites are what developers call single-page apps. The server sends an almost empty page, and JavaScript builds everything you see once it reaches the browser. To you, it looks identical. To a crawler, it can look like a blank sheet of paper.

Try this: on your homepage, right-click and choose View page source (or press Ctrl+U, Cmd+Option+U on a Mac). If you can find your headline and paragraphs in that code, great. If you mostly see an empty <div id="root"></div> and a pile of scripts, your words only exist after JavaScript runs.

Google can run JavaScript, but it does so in a slower second pass, and many of the crawlers feeding AI assistants don't run it at all. If your source looks empty, ask your AI to set up pre-rendering or static site generation so every page arrives with its words already in it. And if your page addresses contain a hash, like yoursite.com/#/about, ask for real paths like yoursite.com/about. Google treats everything after the # as the same page, so your whole site can look like one address.

Hand the map to Google: Search Console, step by step

Google Search Console is a free tool that shows you exactly how Google sees your site. It's where you submit your sitemap, and it'll become your best friend over the rest of this series. Setting it up takes about fifteen minutes.

1. Go to search.google.com/search-console and sign in with a Google account. Use one your business will keep, not a personal one you might lose.
2. Click Add property. You'll see two options. Domain is the better choice, because it covers every version of your address (with and without www, http and https) in one go. URL prefix is the easier one if you don't have access to your domain settings.
3. Prove you own the site. For a Domain property, Google gives you a line of text (a TXT record) to add in the DNS settings wherever you bought your domain, such as GoDaddy, Namecheap, 123 Reg or Cloudflare. For a URL prefix, you can instead add a small tag or file to your site, which your AI builder can do for you. DNS changes can take anything from a few minutes to a few hours to be recognised, so don't panic if verification fails first time.
4. Once verified, click Sitemaps in the left-hand menu. Under "Add a new sitemap", type sitemap.xml and press Submit.
5. Refresh after a little while. You're looking for a status of Success and a count of discovered pages that roughly matches how many pages you have.

While you're in the mood, do the same at Bing Webmaster Tools. It lets you import your Search Console setup in a couple of clicks, and Bing's index sits behind more than just Bing, including other search engines and several AI assistants. Two minutes of work for a second front door.

Give your key pages a nudge

Submitting a sitemap says "here's everything". Sometimes you want to say "look at this, now". That's what the URL Inspection tool is for.

Paste any page address into the search bar at the very top of Search Console. Google will tell you either URL is on Google (lovely) or URL is not on Google, usually with a reason. For a page that isn't indexed yet, click Request indexing. It takes a minute while Google runs a live check, then pops your page into a priority queue.

Use this for your homepage and most important pages at launch, and whenever you publish something new or make a big update. Don't use it on every page every day. There's a daily limit, and asking twice doesn't get you there faster. It's a doorbell, not a buzzer you can lean on.

Reading the Page indexing report without panicking

After a few days, go to Indexing → Pages. You'll see two numbers: indexed and not indexed. New site owners tend to see a big grey "not indexed" number and assume the worst. Most of the time, it's fine. Here's what the common reasons actually mean:

• Page with redirect. An old address that now forwards to a new one. Completely normal. Just don't list these in your sitemap.
• Alternate page with proper canonical tag. Google found a duplicate version and correctly filed the original instead. Working as intended.
• Excluded by 'noindex' tag. Fine for a thank-you page. A five-alarm fire if it's your homepage. See the horror story above.
• Blocked by robots.txt. Your front door sign is saying no. Check that file.
• Not found (404). A link points at a page that doesn't exist. Fix the link, or if the page is gone on purpose, make sure it's out of your sitemap.
• Discovered - currently not indexed. Google knows the page exists but hasn't visited yet. Very common on new sites. Patience, plus links between your own pages, usually does it.
• Crawled - currently not indexed. The one to take seriously. Google read the page and decided it wasn't worth filing yet. That's usually about the content, not the code: thin pages, near-duplicates, or copy that says what a thousand other sites already say.

That last one is where AI-built sites get caught out most. If every page reads like a polite, generic brochure, Google has little reason to keep it. We'll dig into that properly in part three.

How long does it take?

For a brand-new site, anything from a few days to a few weeks. There's no switch that makes it instant, but you can make yourself much easier to find:

• Build a few bridges. Link to your site from places Google already visits: your Google Business Profile, your social profiles, industry directories, a supplier or friend's website.
• Link your own pages together. Every important page should be reachable from your navigation or from another page, not just listed in the sitemap.
• Keep the map current. New page? Make sure it's in the sitemap with an honest date.

For a rough health check, search Google for site:yourdomain.com. It shows a sample of the pages Google has filed. It isn't an exact count, but if it returns nothing at all, you know where to start.

And if all of this feels like something only a beginner would miss, it isn't. During a tidy-up of this very website in September, we found Search Console had no sitemap on record and seven of our pages Google hadn't met yet. Nothing was broken. We just hadn't handed over the map.

Your five-minute checklist

Screenshot this, pin it, tattoo it somewhere discreet:

• yourdomain.com/sitemap.xml loads and lists your real pages on your real domain.
• yourdomain.com/robots.txt has no Disallow: / and includes a Sitemap line.
• No noindex on pages you want found, and any "hide from search engines" setting is off.
• View page source shows your actual words, not an empty shell.
• Search Console is verified and your sitemap shows Success.
• Your homepage and key pages have been through URL Inspection.
• Bing Webmaster Tools is set up too.
• You've linked to your site from your Google Business Profile and social profiles.

Building with AI is a genuinely brilliant way to get a business online. It just can't know what it wasn't asked. If you've built a site and want a second pair of eyes before, or after, it goes live, that's exactly the kind of thing we love helping with.