AI search hubAI SEO servicesGEO servicesAEO servicesEntity SEOAI visibility tracking
Performance marketingSocial media marketingInfluencer marketingAffiliate marketing
Web developmentWeb designApp developmentCROLanding page optimisation
Content marketingContent writingDigital PRSearch everywhere optimisation
Email marketingMarketing automationCustomer lifecycleCustom agentic AICustom LLM skills
Ecommerce marketingSaaS marketingProduct marketingApp marketingASO
Ecommerce / D2CMarketplaceClothing & FashionElectronicsLocal Business
Finance & BankingInsuranceWealth & Portfolio MgmtLaw FirmsRecruitment & Staffing
Real EstateHospitalityTravel & TourismConstruction & Home ImprovementHome Services
White label servicesHire an SEO consultantHire a marketing consultantLead generation
Outsourcing overviewOutsource SEOOutsource PPCOutsource CROOutsource ASOOutsource marketing automationOutsource landing pages
From the UKFrom the USAFrom AustraliaFrom CanadaFrom the UAEFrom Singapore
Crawl Budget
Crawl budget is the number of pages a search engine is willing and able to crawl on a website within a given period, shaped by how fast the server responds and how much Google wants to crawl the site.
When it matters
For most small and medium sites, crawl budget is not a problem. It matters for large sites with many thousands of URLs, such as ecommerce catalogues, marketplaces and publishers, and for sites whose filters or parameters generate endless URL variations.
What wastes crawl budget
- Faceted navigation creating thousands of filter combinations
- Duplicate pages and URL parameters
- Long redirect chains
- Soft 404s and thin pages
- Slow server responses
How to manage it
Block or canonicalise low-value URL variations, keep XML sitemaps clean, fix redirect chains and improve server speed. Google’s spam policies on scaled low-value content (Google Search spam policies) are another reason to keep thin pages out of the index. See marketplace SEO and ecommerce SEO.