Documentation

Sitemaps

A sitemap provides search engines with information about the pages on your site and their relationships. Search engines like Google read sitemaps to crawl your site more efficiently, helping to improve search engine optimization. By default, search engines look for the sitemap at {your-website}/sitemap.xml.

Your commercetools Frontend project is pre-configured with sitemap generation that automatically generates the sitemap whenever you build your project using the yarn build command.

In this article, you'll learn how commercetools Frontend generates the sitemap for all the pages on your website.

Configuring a sitemap

commercetools Frontend uses the next-sitemap package to create sitemaps. The next-sitemap package generates a sitemap for your Next.js website using the sitemap configuration provided in the next-sitemap.config.js file.

The next-sitemap.config.js file provides next-sitemap with the configuration to build the sitemap using the following options:

PropertyDescription
siteURLYour website's origin URL used for creating the full paths.
excludeList of relative paths to exclude from the sitemap.
generateRobotsTxtIf true then the robots.txt file will be generated for web crawlers.
robotsTxtOptionsList of policies for the robots.txt file, and a list of all sitemaps (see Generating the sitemap).
alternateRefsList of language-specific alternative URLs.

You can provide different values to these configuration options by following the next-sitemap configuration documentation. For example, if you want to add a new locale to the sitemap, you need to update the languageMapper object.

/* Website URL */


const { languageMapper } = require('./project.config');


const siteUrl = process.env.SITE_URL;


const alternateRefs = [];
for (const [key] of Object.entries(languageMapper)) {
  alternateRefs.push({
    href: `${siteUrl}/${key}`,
    hreflang: key,
  });
}
/* Site map generator configuration */


/** @type {import('next-sitemap').IConfig} */
const config = {
  siteUrl,
  exclude: [
    '/sitemap-static.xml',
    '/sitemap-categories.xml',
    '/sitemap-products.xml',
    '/verify',
    '/__preview',
  ],
  generateRobotsTxt: true,
  robotsTxtOptions: {
    policies: [
      {
        userAgent: '*',
        allow: '/',
      },
      {
        userAgent: '*',
        disallow: '/api',
      },
      {
        userAgent: '*',
        disallow: ['/__preview'],
      },
    ],
    additionalSitemaps: [
      `${siteUrl}/sitemap-static.xml`,
      `${siteUrl}/sitemap-categories.xml`,
      `${siteUrl}/sitemap-products.xml`,
    ],
  },
  alternateRefs: alternateRefs,
};


module.exports = config;

Generating the sitemap

To generate the sitemaps, we need to fetch the list of all available paths, create a standard sitemap structure by adding necessary fields such as loc and lastmod to the sitemap response, and return the response in the XML format.

We use the helper methods from the frontastic package to get the list of paths and generate 3 sitemaps one for each: static pages, product pages, and category pages for easier maintenance.

Sitemap for static pages

To generate the sitemap for all the static pages created using the Studio, we create a new file, sitemap-static.xml/route.tsx, in the frontend/app/ directory. In this file, we implement the GET route handler method of Next.js that returns the sitemap response.

In the following code, we get the static pages structure using the sdk.page.getPages() method and convert the received data into ISitemapField structure required by next-sitemap. Then we pass the sitemap data fields to the generateSiteMap method of the next-sitemap package and return the response.

The generated sitemap is made available on {your-site}/sitemap-static.xml.

pages/sitemap-static.xml/route.tsxTypeScript
import { SiteMapField, generateSiteMap } from 'helpers/sitemap';
import { sdk } from 'sdk';


export async function GET() {
  const locale = 'en';


  const siteUrl = process.env.SITE_URL;


  sdk.defaultConfigure(locale);


  const fields = [] as SiteMapField[];


  const path = '/';
  const depth = 1;


  const res = await sdk.page.getPages({ path, depth });


  if (res.isError) return new Response(null, { status: 500 });


  const data = res.data;


  if (data?.pageFolderStructure) {
    fields.push(
      ...data.pageFolderStructure?.map((pageFolderStructureValue) => ({
        loc: `${siteUrl}/${locale}${(pageFolderStructureValue as any)._url}`, //eslint-disable-line
        lastmod: new Date().toISOString(),
        changefreq: 'daily' as const,
      }))
    );
  }


  return new Response(generateSiteMap(fields), {
    status: 200,
    headers: {
      'Cache-Control': 'public, s-maxage=86400, stale-while-revalidate',
      'Content-Type': 'application/xml',
    },
  });
}

Sitemap for products

To generate the sitemap for all product pages created using the Studio, we create a new file sitemap-products.xml/route.tsx in the frontend/app/ directory. In this file, we implement the GET route handler method of Next.js that returns the sitemap response.

In the following code, we get a list of all products by calling the extensions.product.query method and convert the received data into the ISitemapField structure required by next-sitemap. Then we pass the sitemap data fields to the generateSiteMap method of the next-sitemap package and return the response.

The generated sitemap is made available on {your-site}/sitemap-products.xml.

sitemap-products.xml/route.tsxTypeScript
import { Product } from 'shared/types/product/Product';
import { SiteMapField, generateSiteMap } from 'helpers/sitemap';
import { sdk } from 'sdk';


export async function GET() {
  const locale = 'en';


  const siteUrl = process.env.SITE_URL;


  sdk.defaultConfigure(locale);


  const fields = [] as SiteMapField[];


  let nextCursor: string | undefined;


  do {
    const extensions = sdk.composableCommerce;


    const response = await extensions.product.query({
      cursor: nextCursor,
      limit: 12,
    });


    const items = (response.isError ? [] : response.data.items) as Product[];


    fields.push(
      ...items.map((product) => ({
        loc: `${siteUrl}/${locale}${product._url}`,
        lastmod: new Date().toISOString(),
        changefreq: 'daily' as const,
      }))
    );


    nextCursor = !response.isError ? response.data.nextCursor : undefined;
  } while (nextCursor);


  return new Response(generateSiteMap(fields), {
    headers: {
      'Cache-Control': 'public, s-maxage=86400, stale-while-revalidate',
      'Content-Type': 'application/xml',
    },
  });
}

Sitemap for categories

To generate the sitemap for all category pages created using the Studio, we create a new file sitemap-categories.xml/route.tsx in the frontend/app/ directory. In this file, we implement the GET route handler method of Next.js that returns the sitemap response.

In the following code, we get a list of all categories by calling the extensions.product.queryCategories method and convert the received data into the ISitemapField structure required by next-sitemap. Then we pass the sitemap data fields to the generateSiteMap method of the next-sitemap package and return the response.

The generated sitemap is made available on {your-site}/sitemap-categories.xml.

sitemap-categories.xml/route.tsxTypeScript
import { Category } from 'shared/types/product/Category';
import { SiteMapField, generateSiteMap } from 'helpers/sitemap';
import { sdk } from 'sdk';


export async function GET() {
  const locale = 'en';


  const siteUrl = process.env.SITE_URL;


  sdk.defaultConfigure(locale);


  const fields = [] as SiteMapField[];


  let nextCursor: string | undefined;


  const extensions = sdk.composableCommerce;


  do {
    const response = await extensions.product.queryCategories({
      cursor: nextCursor,
      limit: 12,
    });


    const items = ((response.isError ? [] : response.data.items) ??
      []) as Category[];


    fields.push(
      ...items.map((category) => ({
        loc: `${siteUrl}/${locale}${category._url}`,
        lastmod: new Date().toISOString(),
        changefreq: 'daily' as const,
      }))
    );


    nextCursor = (!response.isError && response.data.nextCursor) as string;
  } while (nextCursor);


  return new Response(generateSiteMap(fields), {
    headers: {
      'Cache-Control': 'public, s-maxage=86400, stale-while-revalidate',
      'Content-Type': 'application/xml',
    },
  });
}

Final sitemap

Now that we've set up sitemap generation for all our pages, the next-sitemap package uses the sitemap configuration to generate an sitemap index that lists all the different sitemaps:

  • {your-site}/sitemap-static.xml
  • {your-site}/sitemap-products.xml
  • {your-site}/sitemap-categories.xml

The next-sitemap package runs as a postbuild script specified in the frontend/package.json file and generates the sitemap after the next build has finished.

After the build, the generated sitemap index is made available on {your-site}/sitemap.xml. The sitemap index provides the URL to all three sitemaps we created, allowing search engines and web crawlers to index your website.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>{your-site}/sitemap-static.xml</loc></sitemap>
  <sitemap><loc>{your-site}/sitemap-categories.xml</loc></sitemap>
  <sitemap><loc>{your-site}/sitemap-products.xml</loc></sitemap>
</sitemapindex>