Technical SEO best practices are constantly evolving, which means you need to be sure that your website is optimized for search engines on an ongoing basis.
It’s important to make sure your on-page features, such as title tags, meta descriptions, and internal links are tailored to fit user needs but are you sure that all your web pages even exist in the eyes of search engines?
How do you make sure that changes to your website don’t affect your ability to rank? How do you let Google know that you have a new website that needs crawling? After all, without crawlability, the hard work you put into making your site SEO friendly goes completely unnoticed.
If you’re searching for the answers to any of these questions, look no further. A well-maintained XML sitemap allows you to communicate with search engines like Google and Bing to make sure that if you have something to tell them, you can.
What is an XML Sitemap?
An XML sitemap is a file that lists all pages on your website that you deem important for search engines to see. This file can be accessed via a link, typically www.example.com/sitemap.xml (try it out for your favorite website). Below is Google.com’s XML sitemap, which shows how websites are formatted within the file.
What is a Sitemap Used For?
XML sitemaps are used to create a link of communication between search engines and website owners. Once created, they are submitted to search engines like Google through Google Search Console. Google can then read the sitemap to collect information about all pages in the file such as what pages you want to be crawled and which pages have been recently updated. Many website owners will add the link to their XML sitemap to the bottom of their robots.txt file as well to make it as visible as possible to search engine bots.
How Do Bots Use the Sitemap to Crawl Websites?
Bots first discover and crawl websites based on links they follow from previously crawled websites.
With that said, any web page needs at least one link to point to it in order to be crawled by search engines. This is extremely important because if a web page cannot be crawled, it cannot be indexed, which is what allows it to appear in search engine results pages.
XML sitemaps aid in avoiding this problem as they make these ‘undiscoverable’ pages more ‘discoverable.’
When a URL is listed on your sitemap, the sitemap acts as the ‘link’ telling search engines like Google that they should look at a specific page for crawling and indexing. It is important to note, however, that including a URL in your sitemap will not guarantee that it will be crawled – listing it only notifies search engines that you believe it should be crawled.
How does an XML sitemap Benefit Your Website?
The most impactful thing an XML sitemap can do for your website is to increase the visibility of your pages by boosting SEO. A few of the specific SEO benefits your website can realize are listed below.
Initial crawling and indexing
As mentioned, if a website can not be crawled, it can not be indexed and can not appear in search results. If your website doesn’t appear in search results, it is very unlikely that you will see the amount of traffic you desire come to that page.
By hinting to search engines that you want them to crawl specific pages on an XML sitemap, you instantly increase your chances of having each of your pages crawled, indexed, and ranked on results pages.
Ongoing crawling and indexing
XML sitemaps help ensure that your web pages get re-crawled and re-indexed when you make changes to them. This can prevent your pages from being removed from search engine results, or help you continue to get more pages to appear. So even if you have links to every page on your website, you can still benefit from using an XML sitemap.
Who Needs an XML Sitemap?
The bottom line is – everyone should have an XML sitemap. There’s no real harm to having one for your website and it can end up benefiting you in a big way. However, Google specifically recommends XML sitemaps for websites that are “really large, have a large archive of content pages that are isolated or not well linked to each other, new and have few external links to it, and/or use rich media content.”
One thing to keep in mind is that there is a limit on the number of URLs that can be listed within a single XML file.
That limit is 50,000 – if you have more than 50,000 URLs on your website, you will need to create multiple XML sitemaps. Many larger websites need to do so, and you can simply submit the additional sitemap(s) to Google Search Console and/or list them underneath the original sitemap in your robots.txt.
What Pages Should Be Included in an XML Sitemap?
Every page that you want search engines to crawl and index should be included in your XML sitemap. Pages that should be excluded from sitemaps may include those that are not user-friendly, not meant for users to land on via search, or those with sensitive information that you likely have blocked from bots already.
A key point to make here is that you want to be sure you are sending search engines a consistent, uniform message about the URLs on your website.
Including a URL on your XML sitemap tells search engines that you want the page to be crawled and indexed, therefore those pages should be easily accessible to bots. One example to especially keep in mind is that you should not include URLs that are blocked off to spiders on your robots.txt file, for example, in your XML sitemap as that sends a conflicting message to search engine bots and will cause the site to not be crawled.
How Do You Create an XML Sitemap?
It is important to note that an XML sitemap should not replace a clear, easy-to-follow navigation structure on your website.
The sitemap acts as a guide to search engine bots; it helps them find the URLs you’d like them to crawl. When the bots begin to crawl your site, however, they need to be able to navigate on their own without any trouble.
With that said, the best way to go about creating an XML sitemap that will best benefit your website is to start with ensuring your website navigation is as user-friendly as possible.
Once clear navigation is in place on your website, you can feel confident that when search engine bots follow your XML sitemap, they’ll easily be able to understand where they’re going and crawl your site most effectively.
When it comes to actually building the sitemap, you can do so manually in HTML given that an XML sitemap is a type of document. If you’re looking for something a little faster, however, you can use tools such as Screaming Frog which will help quicken the process.
If you’re looking for more of a personal touch, we’d love to help you as well! Contact WebFX today to see how our team can help you create an XML sitemap that gets your pages noticed and explore even more opportunities to boost your SEO!