If you paste a sitemap URL
An address that ends in .xml or .xml.gz is used as it is.
If you paste a domain
- Link Signals reads the site’s
robots.txtand looks forSitemap:lines. Relative addresses are resolved against the domain. - If there are none, it tries
/sitemap.xml,/sitemap_index.xml,/sitemap-index.xmland/wp-sitemap.xml, in that order. A candidate only counts if the start of the file contains a<urlset>or<sitemapindex>element.
https://example.com/robots.txt
Sitemap: https://example.com/sitemap_index.xmlSitemap indexes and compressed files
A sitemap index lists other sitemaps. Link Signals follows them, up to 200 sitemap files, and stops as soon as it has enough pages for your plan. Gzip-compressed sitemaps are unpacked automatically, and each sitemap file can be up to 20 MB.
How many pages are scanned
If a sitemap lists more pages than your plan reads, a scan reads a sample spread across the whole site: the homepage, the most recently changed pages and an even spread of the rest. Monitoring reads pages in the order the sitemap lists them, up to your plan’s limit. If robots.txt declares more than 10 sitemaps, only the first 10 are used. The page count shown before you run a scan counts up to 100,000.
| Plan | Sitemap pages crawled | Links per scan | Link checks a month | Monitored sites |
|---|---|---|---|---|
| Free | 100 | 50 | 50 | None |
| Starter | 1,000 | 2,000 | 2,000 | None |
| Pro | 10,000 | 12,000 | 12,000 | 15 |
| Scale | 50,000 | 50,000 | 50,000 | 50 |
Fetching safely
- Addresses that point at private or internal networks are refused.
- Redirects are followed for up to ten hops, and each hop is checked again.
- Each page is fetched with a ten-second timeout and read up to about 1.5 MB.
- Site owners can opt out. See blocking the Link Signals crawler.
A worked example
Say you type example.com into the scan box. Link Signals asks for https://example.com/robots.txt and finds one line, Sitemap: https://example.com/sitemap_index.xml. That file is a sitemap index listing three child sitemaps: one for posts, one for pages and one for products. Each is fetched in turn, unpacked if it is gzip-compressed, and its page addresses are collected until your plan’s page limit is reached or the sitemaps run out.
If the same site had no Sitemap: line, the four common paths would be tried instead, and the first that returned a file starting with a <urlset> or <sitemapindex> element would be used. If none did, you would be asked for the sitemap address itself, and finding your sitemap URL explains where to look.
Why the dates in a sitemap matter
Each entry in a sitemap can carry a last-modified date. A one-off scan uses those dates to pick the most recently changed pages when it has to sample. Monitoring uses them to decide which pages to read again. A sitemap with accurate dates therefore gives better coverage for the same allowance. Checking your sitemap dates shows how to tell whether yours are trustworthy.
Sampling in practice
Suppose a site lists far more pages than your plan reads. The scan does not simply take the first pages in the file, because those are often one section or the oldest content. It takes the homepage, then alternates between an even spread across the whole sitemap and the most recently changed pages, so a scan that stops early has already seen a mix. The report says that it is a sample and how many pages the sitemap listed. See sampled scans for the definition.
When discovery fails
| What happens | Likely cause | What to try |
|---|---|---|
| No sitemap found for a domain | There is no Sitemap: line in robots.txt and the sitemap isn’t at one of the four common paths. | Paste the sitemap address itself, ending in .xml or .xml.gz. |
| The sitemap is found but lists no pages | It is an index whose child files couldn’t be read, or it is empty. | Open the sitemap in a browser and check that the child addresses load. |
| Only some pages appear | A child sitemap returned an error or timed out. Files that can’t be read are skipped. | Run the scan again. If it persists, check that the file loads for visitors and isn’t blocked by a firewall rule. |
| The scan is refused | The address points at a private or internal network, or the site owner has opted out of being requested. | Use the public address of the site. |
Questions
Does Link Signals read pages that robots.txt disallows?
Robots.txt is read only to find Sitemap: lines. Link Signals does not use its allow and disallow rules to filter pages. It reads the pages your sitemap lists, because you asked for them to be checked, and it identifies itself as LinkSignalsBot so a site owner can recognise the visits. A site owner who doesn’t want to be requested can opt out, as described in blocking the Link Signals crawler.
What does Link Signals call itself when it visits?
Requests identify as LinkSignalsBot with a link to the site. If a destination refuses that, Link Signals tries once more with a browser-style user agent before deciding what to report. It does not try to get past bot protection beyond that single retry.
What if my site has several sitemaps?
If robots.txt lists several, the first 10 are used. A sitemap index is followed into its children. If your sites are split by language or section, scan the sitemap index, or scan each section’s sitemap separately.
Are images and videos in the sitemap scanned?
Only the page addresses are used. Image and video entries are not fetched as pages.
Paste your sitemap and Link Signals reads it the way described here.
