Skip to content
FOCUS POINT Agency
All articles
SEO··9 min

Technical SEO Audit: Crawl Budget and Indexation at Scale

A diagnose-prioritize-act audit for large marketplaces and retail sites: fix crawl budget waste, indexation gaps and site architecture before they cost you revenue.

SB

Sami Belkacem

Head of SEO

ShareLinkedInXMail

TL;DR

On large marketplaces and retail sites, technical SEO is a revenue lever: if Google cannot efficiently crawl and index your important pages, they cannot rank. This audit walks through diagnosing crawl budget waste, closing indexation gaps, and fixing site architecture—then prioritizing fixes by impact so you act on what moves rankings first.

Key takeaways

  • Crawl budget is finite: every wasted crawl on junk URLs is a crawl your best products don't get.
  • Diagnose with log files and Search Console before touching anything—data first, opinions second.
  • Faceted navigation and parameters are the number-one crawl-budget sink on marketplaces—control them deliberately.
  • A shallow, logical architecture keeps money pages within a few clicks and helps Google prioritize them.
  • Prioritize fixes by revenue impact and effort—not by how satisfying they are to complete.

On a ten-page brochure site, technical SEO barely matters. On a marketplace or retail site with hundreds of thousands of URLs, it decides whether you have a business online. Google allocates a finite crawl budget to your domain; if that budget is burned on infinite faceted-filter combinations, expired products, session parameters and redirect chains, your best category and product pages get crawled rarely, indexed late, and updated slowly. The result is invisible lost revenue: pages that could rank never get the chance. This audit follows a strict order—diagnose, prioritize, act—so a Dutch retailer or marketplace can find where crawl and indexation break down and fix the issues that actually move revenue, in the right sequence.

Step 1 — Diagnose: read the data, not your assumptions

  • Server log analysis: see exactly which URLs Googlebot crawls, how often, and how much budget is wasted on non-indexable or low-value URLs.
  • Search Console coverage: catalog why URLs are excluded—crawled-not-indexed, discovered-not-indexed, duplicate, soft 404—each points to a different fix.
  • Crawl simulation: run a full crawl to map depth, orphan pages, redirect chains, broken links and the ratio of indexable to non-indexable URLs.
  • Index bloat check: compare the number of URLs Google has indexed against the number you actually want indexed—the gap is your problem size.

The single biggest culprit on marketplaces is faceted navigation. Every filter combination—color, size, brand, price range, sorting order—can generate a unique URL, and multiplied together they produce millions of low-value permutations that crawlers happily wander through. Add session IDs, tracking parameters, paginated archives and printer-friendly versions, and a catalog of 50,000 real products can expose Googlebot to tens of millions of URLs. Your job in the audit is not to eliminate facets—shoppers need them—but to decide precisely which combinations deserve to be crawled and indexed, and to make every other combination invisible to search engines without breaking the user experience.

WARNING

Do not fix crawl bloat by blocking URLs in robots.txt if those URLs are already indexed—Google can't recrawl them to see a noindex tag if it's disallowed from fetching them. Sequence matters: noindex or canonicalize first, let Google process it, then disallow. Getting the order wrong freezes junk in the index.

Work with us

Is Google wasting its crawl budget on junk URLs while your best products go unseen? FOCUS POINT runs log-based technical SEO audits for marketplaces and large retail sites, then prioritizes and ships the fixes that recover revenue. Let's diagnose your site.

Book a technical SEO audit

Steps 2 & 3 — Prioritize by impact, then act

  • Score every issue on a simple grid: revenue impact × number of affected pages × implementation effort. Fix high-impact, low-effort items first.
  • Protect crawl budget: canonicalize or noindex low-value facets, consolidate parameters, and reserve indexation for combinations with real search demand.
  • Flatten architecture: ensure top categories and best-sellers sit within three clicks of the home page, with clean internal links and HTML sitemaps.
  • Clean up signals: fix redirect chains, resolve soft 404s on out-of-stock pages, and keep XML sitemaps limited to canonical, indexable, 200-status URLs.
  • Handle out-of-stock and expired listings with a clear policy: keep, redirect, or return 410—so budget stops flowing to dead inventory.

DATA

When we cut index bloat on large retail sites, the pattern is consistent: as the ratio of indexable-to-crawled URLs improves, Googlebot reallocates crawl frequency toward money pages, new products get indexed in days instead of weeks, and category rankings climb without a single new link. Efficiency, not more content, is the win.

Technical SEO at scale is never finished—it is a discipline. Once the audit fixes are shipped, keep a monitoring loop: watch log-file crawl distribution, index coverage trends and Core Web Vitals monthly, and re-run a lighter audit each quarter as your catalog, filters and platform evolve. The marketplaces that win in search are rarely those with the most pages; they are the ones whose crawlable, indexable architecture makes every important page easy for Google to find, understand and refresh. That is where technical work quietly turns into revenue.

Ready to put this to work?

Let's start a project together.

Tell us about your brand. We come back with a strategic read within 48h.

48h response21 creative hubsBespoke project support

Next step

Ready to make noise?

Describe your project in 3 minutes. Our team will come back with a first strategic read within 48h.

Start a ProjectTalk to us
Response within 48h