Skip to main content
How-to Search & discovery (SEO, GEO, AEO)

Technical SEO for Google and Bing

Great content does not rank if a crawler cannot reach the page, two URLs quietly split the signal that should belong to one page, or nobody ever told Google and Bing the page exists. This is the concrete, do-it-now technical checklist: crawlability, canonical URLs, sitemaps, Search Console and Bing Webmaster Tools, Core Web Vitals, and the URL and linking structure that lets a crawler find everything else.

6 steps13 min readLast verified August 2026

What you’ll learn

  • Tell robots.txt Disallow rules apart from noindex directives, and use the correct one for the job
  • Set one canonical URL per page and link internally to that version everywhere
  • Publish and submit an XML sitemap to both Search Console and Bing Webmaster Tools
  • Read the Page indexing report to diagnose why a specific page is not indexed
  • Hit Core Web Vitals thresholds, LCP, INP, and CLS, checked on mobile
  • Build clean, stable URLs and confirm every real page has an internal link pointing to it

Technical SEO is the plumbing underneath the content and structured-data work covered elsewhere in this Academy. The best content in a category still will not rank if a crawler cannot reach it, if two near-identical URLs split the ranking signal that should belong to one page, or if nobody ever submitted a sitemap and the page sits undiscovered for weeks. How SEO, GEO, and AEO actually work covers what happens once a page is found. This guide covers getting it found: six checks, run in order, that you can do yourself in an afternoon with your site's files and a free Google and Bing account. Most of it is confirming a default has not gone accidentally wrong, not building anything new.

1. Make sure pages can actually be crawled

New to this? Crawling is an automated program, Googlebot or Bingbot, visiting your pages and following the links it finds, the same way a person clicking around your site would. Indexing is the separate step after that: deciding whether a crawled page is worth storing and eligible to show up in search results. As the next paragraph explains, a page can be crawled without being indexed, and it can even be indexed without ever being crawled, from a link alone.

Before a page can rank, get selected for an answer box, or show up in an AI-generated answer, a crawler has to reach it and read it. Two separate mechanisms control that, and mixing them up is a common technical SEO mistake: robots.txt and the noindex directive are not interchangeable, and reaching for the wrong one produces the opposite of what you wanted.

A Disallow rule in robots.txt controls crawling, not indexing. It tells a well-behaved crawler not to fetch a URL, it does not tell Google to keep that URL out of search results. Google is explicit: robots.txt "is not a mechanism for keeping a web page out of Google."1 If another site links to a page you blocked with robots.txt, Google can still index that URL from the link alone, without ever crawling the page, and it can appear in results with no description.1

A robots.txt that blocks admin/cart but leaves everything else crawlable
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://example.com/sitemap.xml
Check: open yoursite.com/robots.txt right now. Confirm there is no Disallow: / line left over from a staging deploy, that one line quietly blocks an entire production site.

noindex is the actual tool for keeping a page out of results, either a <meta name="robots" content="noindex"> tag in the <head>, or an X-Robots-Tag header for non-HTML files like PDFs. But it only works if the crawler can see it: the page must not be blocked by robots.txt. Google states this directly: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler."4 Block a page in robots.txt and add noindex too, and the noindex is never read. That page can still end up indexed by link alone, the opposite of what both directives were trying to do.

Check: search site:yoursite.com in Google for pages you expect to be public, then confirm a page you deliberately want excluded has a reachable noindex tag, not one hidden behind a robots.txt block.

2. Give every page one canonical URL

Most real sites generate more than one URL for the same content without meaning to: http and https, www and the bare domain, a trailing slash or not, tracking parameters like ?utm_source=... on every campaign link. A search engine treats each variant as a distinct URL. Left alone, that splits links, authority, and ranking signal across several URLs instead of consolidating it behind one.

rel="canonical" tells a search engine which version should represent all of them. Google ranks canonicalization signals by strength: a redirect is "a strong signal that the target of the redirect should become canonical," a rel="canonical" annotation is also "a strong signal," and listing a URL in a sitemap is only "a weak signal that helps the URLs that are included in a sitemap become canonical."2 Redirect true duplicates outright, and reach for rel=canonical on pages that legitimately need to stay separately reachable, like a print view or a URL carrying a required tracking parameter.

Canonical tag in <head> (HTML pages)
<link rel="canonical" href="https://example.com/shoes/red-sneakers" />
Canonical via HTTP header (non-HTML files, like PDFs)
Link: <https://example.com/downloads/whitepaper.pdf>; rel="canonical"
Check: view source on any page reachable through a query parameter or an alternate URL, and confirm exactly one <link rel="canonical"> pointing at the version you want indexed, not the URL currently sitting in the address bar if that URL is a duplicate.

Internal links should point straight at the canonical URL, not at a duplicate. Google says this directly: "When linking within your site, link to the canonical URL rather than a duplicate URL."2 A site that canonicalizes correctly but still links internally to the tracked, non-canonical version is sending a mixed signal on every click.

3. Publish and submit an XML sitemap

A sitemap lists the URLs on your site you want crawled and indexed, in a standard XML format. Google describes XML sitemaps as "the most versatile of the sitemaps formats," extensible enough to carry extra data about images, video, and news content.3 A sitemap does not force indexing, nothing does, but it is the most direct way to tell a search engine what exists and when it last meaningfully changed, instead of waiting to be discovered by chance.

A minimal XML sitemap
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-08-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-07-15</lastmod>
  </url>
</urlset>

Use absolute, fully-qualified URLs, https://example.com/page, not relative paths, since Google will "attempt to crawl your URLs exactly as listed."3 Include a <lastmod> date that reflects a real, meaningful change, not a timestamp that updates on every deploy. Google ignores <priority> and <changefreq> entirely, so do not spend time tuning them.3

Submit the sitemap in both places that matter. In Search Console, submit it under the Sitemaps report, which shows when Googlebot last fetched it and any processing errors.3 In Bing Webmaster Tools, submit the same file from the Sitemaps section of the dashboard.8 Either way, add a pointer to it in robots.txt too, so any well-behaved crawler can find it without a manual submission: Sitemap: https://example.com/sitemap.xml.3

Check: fetch yoursite.com/sitemap.xml directly. It should load as valid XML with real, live URLs, none of them 404s, none of them pages you have deliberately noindexed.
Going further: a single sitemap file caps out at 50,000 URLs. Past that, split it into several sitemap files referenced from one sitemap index file. On a large site, also check your server logs directly to see which pages Googlebot is actually spending its crawl budget on, rather than assuming a scanner's guess is the full picture.

4. Verify the site in Search Console and Bing Webmaster Tools, and read the reports

Submitting a sitemap tells a search engine what exists. Verifying ownership in Search Console and Bing Webmaster Tools is what shows you what happened next, and why. Two Search Console tools matter most. The Page indexing report "shows which pages have errors, warnings, or are excluded from Search,"5 broken out by reason: a noindex tag, a robots.txt block, a duplicate with a different canonical, a server error, or a page not crawled yet. URL Inspection works one page at a time, it "provides the current index status of website pages and options to test a live URL, to ask Google to crawl a specific page, and to view detailed information about the page's loaded resources."5 Bing Webmaster Tools mirrors this with its own verification and reporting.

  • Page expected to be indexed but is not: check URL Inspection first, then robots.txt and the noindex tag, before assuming a ranking problem.
  • Page indexed under the wrong canonical: check the Page indexing report's duplicate-content category, it names the URL Google actually picked.
  • Sudden coverage drop after a deploy: check the sitemap's last-read date and diff robots.txt against the previous version.
Check: open the Page indexing report right now and look at the Excluded tab. Every category listed there has a specific, fixable reason attached to it, not a mystery.

5. Make it fast and mobile-friendly

Once a page is crawlable, canonical, and in the index, speed and usability decide how well it competes. Google measures this with three Core Web Vitals metrics, each with a published "good" threshold, at the 75th percentile of real page loads across mobile and desktop:7

  • Largest Contentful Paint (LCP): the main content should finish rendering within 2.5 seconds of the page starting to load.7
  • Interaction to Next Paint (INP): the page should respond to a click, tap, or keypress in 200 milliseconds or less.7
  • Cumulative Layout Shift (CLS): unexpected layout movement should stay at 0.1 or less.7

Mobile is not a secondary check. Most real traffic is a phone on a mid-range connection, and a page that performs well only on a desktop with fast office Wi-Fi is not representative of what Google or Bing actually measures. A responsive layout, a correct viewport meta tag, tap targets sized for a thumb, and images compressed and sized for the container cover most of what breaks this in practice.

Check: run the real, live URL, not localhost, through a Core Web Vitals report and confirm LCP, INP, and CLS each land in the Good band, not Needs improvement.

6. Use clean, stable URLs and real internal links

Crawlers discover new pages almost entirely by following links: from your own site, from other sites, and from your sitemap. An orphan page, one with zero internal links pointing to it, can sit in the sitemap indefinitely and still take far longer to get crawled than a page linked from a homepage or category page.

URL structure matters too. Google recommends "readable words rather than long ID numbers" in URLs, structured "logically and in a manner that is most intelligible to humans," hyphens rather than underscores to separate words, and trimming parameters that do not change the content.6 Google also treats casing as entirely separate URLs, /Shoes and /shoes are two distinct pages,6 so pick one case and stay consistent.

  • Every real page reachable by at least one link from another indexed page, not sitemap-only.
  • URLs stay stable once published, a URL that changes on every re-publish forces a fresh crawl and redirect each time.
  • Redirects point straight at the final destination, one hop, not a chain stacked from years of small changes.
  • Navigation links use real <a href> tags, not a JavaScript-only handler a crawler cannot follow.
Check: pick five pages at random from your sitemap and confirm each one is reachable by clicking through your own navigation, not just by pasting the URL directly into the address bar.

Common technical SEO issues

Most of what breaks indexing on a real site is one of a short, repeatable list.

IssueSymptomFix
A stray <code>Disallow: /</code> in robots.txt, often left over from stagingTraffic drops sharply; Search Console flags pages as blocked by robots.txtDelete the blanket rule, confirm with URL Inspection, then resubmit the sitemap
A noindex tag left over from a staging buildPages vanish from results with no robots.txt block and no manual action takenSearch the CMS for <code>content="noindex"</code> and remove it from public pages
Duplicate URLs with no canonical (http/https, www, trailing slash, tracking params)Page indexing report shows Google chose a different canonical than expectedRedirect true duplicates, add rel=canonical on variants that must stay reachable
Sitemap never resubmitted after new pages went liveNew pages take weeks to crawl; sitemap report shows an old last-read dateRegenerate on every publish, resubmit to Search Console and Bing
Orphan pages with no internal links pointing to themPage sits in the sitemap indefinitely but never shows as indexedAdd at least one contextual link from an already-indexed page
Unoptimized hero images or render-blocking scriptsCore Web Vitals report flags LCP as Needs improvement or PoorCompress and size images, preload the hero image, defer non-critical scripts
Redirect chains of three or more hopsSlower loads, wasted crawl budget, a diluted canonical signalPoint every redirect straight at the final URL, one hop
Issue, symptom, and fix

Do this

  1. Confirm robots.txt is not blocking pages you want indexed, and any noindex tag sits on a page the crawler can reach.
  2. Pick one canonical URL per page, add rel=canonical, and link internally to that version everywhere.
  3. Publish an XML sitemap with absolute URLs and real lastmod dates, referenced in robots.txt.
  4. Submit and verify the sitemap in Search Console and Bing Webmaster Tools, then read the exclusion reasons.
  5. Get LCP under 2.5 seconds, INP under 200 milliseconds, and CLS under 0.1, checked on mobile.
  6. Use clean, stable URLs, and make sure every real page is reachable by an internal link, not sitemap-only.
Key idea
Crawlability, one canonical URL per page, a submitted sitemap, verified reporting in Search Console and Bing Webmaster Tools, Core Web Vitals, and real internal links. robots.txt controls crawling, not indexing; noindex controls indexing, but only works if the crawler can reach the page to read it. Get this plumbing right and the content-level work elsewhere in this Academy has something to actually rank.
Read next: How SEO, GEO, and AEO actually work covers the surfaces this plumbing feeds into, Structured data and how engines use it covers the schema layer on top of a crawlable site, and the Shipping web apps course covers getting a real site live in the first place.

Sources

Verified against primary sources: August 2026.

  1. Robots.txt Introduction and Guide. Google Search Central. https://developers.google.com/search/docs/crawling-indexing/robots/intro
  2. Consolidate Duplicate URLs. Google Search Central. https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
  3. Build and Submit a Sitemap. Google Search Central. https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
  4. Block Search Indexing with 'noindex'. Google Search Central. https://developers.google.com/search/docs/crawling-indexing/block-indexing
  5. How to Use Search Console. Google Search Central. https://developers.google.com/search/docs/monitor-debug/search-console-start
  6. URL Structure. Google Search Central. https://developers.google.com/search/docs/crawling-indexing/url-structure
  7. Web Vitals. web.dev, Google. https://web.dev/articles/vitals
  8. Sitemaps. Bing Webmaster Tools Help. https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed
Read nextHow SEO, GEO, and AEO actually work
All How-tos