Skip to content
TrafficBlog TrafficBlog Est. 2019
Vol. 7 · Issue 41 — Tuesday, 9 AM ET refresh Lisbon · Austin · Berlin · ISSN 2789-0144
SEO与数字营销

Sitemap Submitted but Pages Still Not Indexed? Separate Crawling From Indexing

TTrafficBlogAuthor

Brand promotion: this article introduces Guangsuan's Google indexing service (GSI). It is a vendor-introduction piece, not an independent review or a commissioned evaluation.

You submitted your sitemap, Search Console reports it as processed, and weeks later the page you care about still does not show up. The next thought is often that you need an indexing service. Before spending anything, it helps to split the single word indexed into two questions: has Google found and fetched the URL, and has Google decided to put it in the index? A sitemap submission only speaks to the first of those.

The short answer

A sitemap tells Google that addresses exist, and search engines read it to crawl a site more efficiently. Google's documentation is explicit about the limit: a sitemap helps search engines discover URLs on your site, but it does not guarantee that all the items in the sitemap will be crawled and indexed. So when pages stay out of search after submission, the useful next step is to read the per-URL status in Search Console's Page indexing report, and use the URL Inspection tool for a single address, to see where the chain stopped.

One condition belongs with that answer. For a new page or site, Google notes that it can take about a week before crawling and indexing begin, so a quiet first week is not by itself evidence of a fault. Google's guidance is also that you should not expect every URL on a site to be indexed; the goal is the canonical version of every important page.

What a sitemap does, and what it does not

Google describes a sitemap as a file that provides information about the pages, videos and other files on a site and the relationships between them. Its job is discovery. It is not a page, not the HTML sitemap built for visitors, and not a request that a page be indexed.

Three steps behind one missing page

  • Discovery. Google has to learn that the URL exists, through a link from a page it already knows, through a sitemap, or through an indexing request.
  • Crawling. Google has to be allowed and able to fetch the page. A robots.txt rule, a login requirement, a server error, or a page that never finishes rendering can each stop a fetch.
  • Indexing. Google decides whether the page belongs in the index. A successful crawl does not settle that decision, and an indexed page is not guaranteed to appear in every relevant search.

Only the first two are directions a crawl and indexing service can work on. The third is Google's decision.

Two Search Console statuses that look similar and point in different directions

These two page-level labels are easy to confuse, and they call for different work.

StatusWhat it meansWhere to look first
Discovered, currently not indexedGoogle found the URL but has not crawled it yet, and the last crawl date is empty.Internal links, the sitemap, server availability and how many pages the site has.
Crawled, currently not indexedGoogle fetched the page but did not put it in the index.Content value, duplicate pages, canonical choice and indexability.

Treat either label as a starting point rather than a verdict. Crawled, currently not indexed does not by itself prove that the content is poor, and Discovered, currently not indexed does not by itself prove that paid crawl help is required.

Check these on your own site first

Before buying anything, confirm the following for the specific URLs you care about. Several of these reasons belong to the site rather than to Google.

  • Robots.txt. Google lists a robots.txt rule that blocks Google from crawling large sections of a site among the reasons large groups of URLs are not indexed. Test a specific path and crawler rather than assuming a rule is harmless.
  • Noindex. A noindex directive keeps a reachable page out of the index. One way this happens is a noindex left over from a staging or preview setup.
  • Canonical and duplicates. If several URLs serve the same or very similar content, Google may index one canonical version and treat the others as alternates. That is intended behaviour rather than an error to fight.
  • What the sitemap entries point at. An entry should return the page itself rather than a redirect, an error or a login screen.
  • Whether the page is reachable at all. A page behind a login, returning a 404, or behaving like a soft 404 is not indexed as an ordinary page.
  • Server behaviour. Repeated server errors during a crawl can interrupt fetching, which is a hosting or application problem to fix before adding crawl work.

Not every non-indexed URL needs fixing. Google states that it is fine for a URL not to be indexed for a good reason, such as an expected robots.txt rule, a noindex tag, a duplicate URL, or a 404 for a page that was removed and has no replacement.

When a crawl-discovery service is relevant

If a URL is reachable, not blocked by robots.txt, not marked noindex and not an unresolved duplicate, and Search Console reports it as discovered but not crawled, then a service that promotes discovery and crawling has something concrete to work on. That is where Guangsuan's Google indexing service, GSI, is positioned.

Guangsuan states that GSI, short for Google Speed Indexing, is its own service name, and that it is not a Google product, an official partnership or a priority-indexing permission. It describes the work as promoting crawl discovery and indexing around the URLs you submit, using its self-built GPC crawler pool and related link structures. The same page states the boundary plainly: the technical work addresses the opportunity to be discovered and crawled, while whether a page is indexed, when, and whether it stays indexed is decided by Google.

The service is scoped by URL batch rather than by month. The published tiers run from a small batch of priority pages up to very large sites, and the reference indexing window lengthens as the batch grows.

Batch of submitted URLsPublished reference indexing window
1 to 100Indexing starts within 2 days
1 to 1,000Indexing starts within 2 days
10,0007 to 15 days
100,000 or more7 to 30 days

Two qualifiers sit next to that table. The published indexing-rate range is 60% to 90%, and the page states that this is a reference interval rather than a guarantee for any batch of URLs. The published periods are service references affected by page quality, site status and Google's processing, and the page states that no fixed number of days is promised.

Pages that can be discussed for suitability include a homepage, topic pages, product pages, article pages, and external pages you are authorised to operate. Service scope, observation period, verification method and reporting are confirmed before cooperation. Where you have access to the property, Guangsuan points to Search Console's URL inspection and Page indexing report for verification rather than a site: query, which does not necessarily list every indexed URL and is not an exact index total.

You can read how the scope and the verification terms are described on Guangsuan's GSI Google indexing service page, and how the crawl-side resources are described on Guangsuan's GPC crawler pool page.

What such a service does not promise

  • Not a guaranteed place in the index. Guangsuan states that submission is not indexing, and indexing is not ranking. Whether a page is indexed, when, and whether it stays indexed is Google's decision.
  • Not a ranking. Indexing is eligibility to appear in results rather than a position, and an indexed page does not necessarily show up in every relevant search.
  • Not a permanent index entry. A URL can leave the index later because of content quality, duplicate pages, canonical changes, access status or Google's own judgement.
  • Not a repair for the page. Crawl blocks, noindex directives, redirects and duplicate content are site-level problems to fix first; crawl work cannot bypass Google's quality judgement.
  • Not risk-free by default. Guangsuan states that crawl guidance and any link operations should comply with Google's spam policies, and that no SEO operation can be promised as 100% risk-free.

What to prepare before you ask

Most of the value of a first conversation comes from the material you bring to it. Guangsuan asks for the site address, the specific URLs, the number of pages and their current status; the same list works well with any provider.

  • The exact URLs, not the domain and not the whole sitemap.
  • Each URL's current status in Search Console: indexed, discovered but not crawled, or crawled but not indexed, with the date you looked.
  • Confirmation that each URL loads for someone who is not signed in, with no login, 404 or soft 404.
  • Confirmation that nothing blocks the URL: no robots.txt rule, no noindex directive, no canonical pointing elsewhere.
  • The page type for each URL, since not every type is a good submission candidate.
  • Your verification plan: which tool, over which period, and what you will record.

Frequently asked questions

Does submitting a sitemap guarantee that my pages will be indexed?

No. Google states that a sitemap helps discovery but does not guarantee that everything in it will be crawled and indexed. Indexing is a separate decision made by the search engine.

My page says discovered, currently not indexed. Is that where an indexing service helps?

That is the direction such a service works on, provided the URL is reachable and free of an obvious crawl block, noindex or duplicate-canonical problem. If the page is blocked or duplicated, that is a site fix first.

My page says crawled, currently not indexed. What now?

That status means Google already fetched the page. The next step is to look at content value, duplicate pages, canonical choice and indexability rather than adding crawl activity.

How do I check whether the work changed anything?

Where you have access to the property, use Search Console's URL inspection tool and Page indexing report. Guangsuan states that the scope, observation period and reporting method are agreed before cooperation, and that a site: query is not a reliable index count.

Can pages I do not own be submitted?

Guangsuan states that external pages can be discussed if you have authorisation to operate them, and that the URL must be reachable and free of obvious crawl blocks or noindex directives.

Sources

Ship work that moves pipeline — every Tuesday, in your inbox.

Get the Weekly Traffic Brief