A stage rental company published 1,900 pages last year: every truss size, every generator, every city inside a forty-minute drive. Nine months on, roughly 700 of them had never been fetched by a search engine at all. They existed. Nobody could find them.
Publishing and being found are two different events, separated by machinery most teams never look at. In a county built on suppliers with deep inventories, that gap is not a technical footnote — it is the difference between a catalog that sells and a catalog that sits there.
Sites in this county grow sideways
Look at what a supplier business actually has to publish. An equipment list with hundreds of line items. Stage and facility specifications with dimensions, power, clearance and load ratings. Fabric and material catalogs with weights, widths, mill numbers and color runs. Lane and dock information. And then service-area pages, because Los Angeles County holds eighty-eight incorporated cities and somebody decided each deserved a page.
On top of that sits the portfolio. Every job finishes, somebody writes it up, and the site gains another page. Ten a month is not unusual for a busy shop. None of those pages are deleted, because deleting them feels like erasing the work.
Five years of that and a twelve-person company owns a website the size of a mid-market retailer, assembled by nobody in particular, with no policy about what should exist. The crawler that arrives has to decide, unaided, which of those thousands of addresses is worth its time.
Found, kept, ranked
Ranking gets all the attention, and it is the third stage of three. A page has to be discovered before it can be evaluated, and evaluated before it can compete. Skip a stage and the work upstream is invisible.
- Discovery. A crawler learns the address exists — from a link, from a sitemap, from a submission. Until that happens, nothing you wrote on the page has any effect whatsoever.
- Crawling. The address is fetched. This costs the search engine money and costs your server time, which is exactly why it is rationed.
- Indexing. The content is judged worth storing. This is a decision, not an automatic consequence of a successful fetch, and it can go against you.
- Serving. The stored page is considered for queries. Everything written about keywords and content quality applies only from here on.
Most reports of a page "not ranking" turn out, on inspection, to be a failure two stages earlier. The remedy for a page that was never fetched is not a better headline.
Never discovered
The address is in no sitemap, nothing on the site links to it, and no submission was ever made. It is published in the sense that a file exists on a server.
- Typical of pages reachable only through a filter
- Fixable, and the fix is mechanical
Discovered and refused
The crawler arrived, read the page and declined to store it. Almost always because six other pages on the same site say the same thing.
- Typical of templated city and project pages
- Fixable only by editing or merging
Crawl budget and where suppliers spend it
Crawl budget is the working shorthand for how much fetching a search engine is willing to do on your domain in a given period. It is set by two things: how much load your server can absorb without slowing down, and how much the engine judges your site is worth. Neither is purchasable. Both are influenced by what you do.
The critical property is that it is spent, not allocated. Every fetch draws from the same account, including all the fetches you did not intend.
| What consumes it | How it shows up on a supplier site | What it costs you |
|---|---|---|
| Filter and sort parameters | An equipment list with filters for type, capacity, availability and price | One inventory page becomes thousands of fetchable addresses |
| Near-duplicate location pages | Eighty-eight city pages with one word changed | Repeated fetching that teaches the crawler your pages are interchangeable |
| Thin portfolio entries | Four hundred project pages with three sentences and a photo | Budget drained before the crawler reaches the specification pages |
| Redirect chains | Three site rebuilds, each layered on the last | Two or three fetches spent to arrive at one page |
| Slow responses | An uncached catalog query behind a filter | The engine reduces its rate to protect your server |
| Large uncontrolled files | Spec sheets and drawings published as PDFs | Fetches spent on documents nobody searches for by name |
The pattern is consistent: budget is rarely lost to the pages you care about. It is lost to combinations, duplicates and leftovers that no one on your team knows exist, and it is lost quietly.
Which of your pages a buyer would ever search
Before optimizing discovery, decide what deserves to be discovered. Most oversized supplier sites do not have a crawling problem so much as a publishing problem with technical symptoms.
Pages a buyer types toward
Specification pages that name a capability, a size, a material or a standard. These are what a coordinator searches for at four in the afternoon with a slot to fill.
- Equipment and capability pages with real numbers
- Facility specs: dimensions, power, load, clearance
- Material and fabric detail with weights and widths
- Service areas where your answer genuinely differs
Pages that exist because of a template
Addresses generated by a system or a habit rather than by demand. They compete with each other, dilute the signal and consume fetching.
- City pages differing by one place name
- Portfolio entries with a photo and two lines
- Filter combinations nobody would link to
- Tag and archive pages the CMS created on its own
Portfolio pages deserve a specific ruling, because every supplier here has hundreds. A project write-up that names the problem, the specification and the constraint is a genuine capability page and often outperforms the service page above it. A project write-up that says "another great build for a great client" alongside four photographs is a scrapbook entry. Keep the first kind as pages. Put the second kind in a gallery on one page and stop generating addresses for them.
Using sitemaps as a discovery instrument
Most sites treat the sitemap as a formality: one file, generated automatically, containing everything, submitted once and forgotten. On a site of two hundred pages that is harmless. On a supplier site of four thousand it throws away the most useful diagnostic available.
Sitemaps can nest, and the Indexing Hub parses them recursively up to three levels deep, handling as many as 1,000 sitemaps in a single job — supplied either as an uploaded file or as a URL. Three levels is enough to model the way an inventory business is actually organized.
- Level one: the index. A single entry point listing the sitemaps below it, which is the only address you ever have to hand over.
- Level two: page type. Capability pages, facility specs, material catalog, service areas, projects, editorial. One sitemap per type.
- Level three: the divisions inside a type. Catalog split by category, projects split by year, service areas split by region of the county.
- Why bother. Because now coverage can be read per group. "Sixty percent of the material catalog is unfetched" is an instruction. "Some pages are missing" is not.
Two sitemap jobs run at once and up to twenty wait in the queue, so a full restructuring can be submitted in one sitting and processed in order rather than by hand over a fortnight. That queue matters most right after a rebuild, when several thousand addresses change at the same time.
The daily budget and the IndexNow route
Alongside sitemaps sits direct submission, and it operates under two published limits. The URL tracker carries a daily budget of 1,000 URLs per account, and bulk submission accepts up to 10,000 URLs in one batch. The batch is the container; the daily figure is the rate at which it drains.
Both numbers are generous for a normal week and finite for a rebuild. A shop that relaunches with 4,200 changed addresses is looking at a queue that clears over several days, which forces the useful question: which addresses go first?
Submissions travel through the IndexNow API, which notifies participating crawlers — GoogleBot and BingBot among them — that an address is new or changed, rather than waiting for them to rediscover it in their own time. It is a notification protocol. It moves the moment of discovery earlier; it does not move the decision that follows. That is the substance behind a submission pipeline built on IndexNow rather than a promise of placement.
| Situation | Sitemaps | Direct submission |
|---|---|---|
| Ongoing publishing, a few pages a week | Sufficient on its own | Optional |
| Site rebuild, thousands of new addresses | Required, structured by type | Yes, in priority order over several days |
| One updated specification page | Too slow to be the plan | The right instrument, immediately |
| Seasonal catalog refresh | Refresh the affected sitemap | Only the addresses that actually changed |
| New service-area pages | Their own sitemap, for measurement | Only after checking the pages differ |
What the log tells you and what it does not
Each URL carries its own record: bot visits with timestamps, current status, and error details when a fetch failed. Live counters run above the batch for submitted, found and failed. Read together, three counters and one timestamp column answer most questions people otherwise guess at.
- Submitted but never visited. Discovery has not happened yet, or the address was judged not worth the trip. Wait first; investigate if a week passes with nothing.
- Visited, then nothing. The crawler came and the page did not earn storage. That is a content and duplication verdict, and it is the most useful message the log produces.
- Visited repeatedly, still absent. Usually a duplicate signal: several addresses saying the same thing, with the engine picking one and dropping the rest.
- Failed with an error. Server errors, timeouts and redirect loops. These are the only entries you can fix with certainty, and they should be cleared before anything else is attempted.
The failure column is worth a separate hour. On sites that have survived two or three rebuilds — normal for a company that has been trading in this county since the nineties — a surprising share of the catalog answers slowly or resolves through a chain of redirects, and every one of those is budget spent on nothing.
Four thousand two hundred addresses, one thousand a day
Take an equipment rental company in Sun Valley rebuilding its site. The new structure produces 4,200 addresses: 260 capability and equipment pages, 1,900 catalog detail pages, 88 service-area pages, 1,750 project entries, and around 200 editorial and general pages.
The daily budget is 1,000 URLs. Submitted indiscriminately, the pages that pay the bills arrive somewhere in the middle of five days of traffic, and the 1,750 project entries — most of them a photo and a caption — get the same treatment as the specification pages.
Ordered by what a buyer types, the same week looks different. Day one: 260 capability pages plus the 200 general pages plus the first slice of the catalog. Day two and three: the rest of the catalog detail. Day four: the 88 service-area pages, but only the ones that survived the honest review — usually closer to twenty-five. Day five onward: the project entries that were rewritten as capability evidence, while the rest move into a gallery and never become addresses at all.
The technical limits did not change. The order of operations did, and the pages that produce quotes were discovered on day one instead of day four. That is the whole benefit, and it is available to anyone willing to rank their own pages honestly before a crawler does it for them.
AutoSEO — discovery inside a running campaign
For a supplier whose catalog changes faster than anyone can hand-manage.
- Indexing sits beside the campaign. Submission, sitemaps and the URL log live in the same panel as the analytics that will later tell you whether any of it produced traffic.
- Keyword work runs automatically. Candidates are found and prioritized from Search Console history, the live results page and your own seed terms.
- On-site suggestions come from the same data. Recommendations reference the pages the crawler is actually reaching, not a generic checklist.
FullSEO — when a person has to approve the structure
For catalogs where terminology is contractual and a wrong edit costs a quote.
- Terms are chosen by hand. Manual keyword selection with automatic fallback, so trade vocabulary is not overwritten by something more popular and wrong.
- Placements target an authority level. Links are placed manually against a Domain Authority goal rather than accepted as they arrive.
- Review mode before publication. Specialists, developers and writers check the output; nothing reaches the catalog unseen.
Stream takes URL lists by the batch, which is the practical reason it appears in a piece about indexing: a rebuild arrives as a spreadsheet, not as a series of typed addresses. The feed keeps submissions, reports, new links and open to-dos in one chronological place per project, filterable and searchable. Related write-ups sit on our blog, and the ways we work with catalog sites are listed under services.
Five questions from catalog owners
We submitted 3,000 URLs a month ago and half are still not indexed. Is something broken?
Probably not. Submission requests attention; storage is decided separately. Open the log and split the half into three groups: never visited, visited and dropped, failed with an error. Only the third group is a fault you can repair, and the second group is telling you those pages are too similar to each other to be worth keeping.
Should we build a page for each of the eighty-eight cities?
Only where the page can say something that is true and different — a yard you deliver from, response times that actually vary, permits or restrictions specific to that jurisdiction. Where the only difference is a place name, one well-built regional page beats forty thin ones, and it stops your own pages competing against each other for the same result.
Do we delete a decade of project pages?
Not delete — reclassify. Write-ups that name a specification, a constraint and a solution are capability evidence and should stay as pages. The rest belong in a gallery on a single page: still visible to a client, no longer a thousand separate addresses asking to be fetched.
Does faster indexing improve our rankings?
No. It moves a page into contention sooner, which matters when your catalog changes weekly, but competition is judged on entirely separate grounds. Fast discovery of a weak page produces a weak page in the index a little earlier than it would have appeared otherwise.
How often should we resubmit the sitemap?
When something changes, not on a calendar. Resubmitting an unchanged file weekly achieves nothing. If a catalog section is rebuilt, refresh that section's sitemap; that is precisely why splitting by page type is worth the setup effort.
The work here is unglamorous and it is finite. Inventory what exists. Decide what deserves an address. Structure the sitemaps so coverage can be read by page type. Submit in the order a buyer would care about. Then read the log instead of guessing, and fix the errors it names.
The instruments make that possible rather than doing it for you. Background workers keep the counters current, site tags let one operator hold several domains in one view, and a single property can be released to an outside email address when a developer needs access to one site. The indexing views and the analytics views sit in the same panel, which is what allows the question to be closed properly: the page was discovered on the eleventh, first appeared in the query report on the twenty-fourth, and produced its first click a week after that. Where discovery stops being the constraint, generative research into an unfamiliar category becomes the more useful place to spend an afternoon.
Start with the inventory you already own. Open the Indexing Hub and submit the sitemap you think is complete — the first honest number most catalog owners see is how many of their own addresses were never in it. That number, not the ranking report, usually explains the quiet quarter.
