Sitemaps

A sitemap provides search engines with information about the pages on your site and their relationships. Search engines like Google read sitemaps to crawl your site more efficiently, helping to improve search engine optimization. By default, search engines look for the sitemap at {your-website}/sitemap.xml.

Your commercetools Frontend project is pre-configured with sitemap generation that automatically generates the sitemap whenever you build your project using the yarn build command.

In this article, you'll learn how commercetools Frontend generates the sitemap for all the pages on your website.

Configuring a sitemap

commercetools Frontend uses the next-sitemap package to create sitemaps. The next-sitemap package generates a sitemap for your Next.js website using the sitemap configuration provided in the next-sitemap.config.js file.

The next-sitemap.config.js file provides next-sitemap with the configuration to build the sitemap using the following options:

Property	Description
`siteURL`	Your website's origin URL used for creating the full paths.
`exclude`	List of relative paths to exclude from the sitemap.
`generateRobotsTxt`	If `true` then the `robots.txt` file will be generated for web crawlers.
`robotsTxtOptions`	List of policies for the `robots.txt` file, and a list of all sitemaps (see Generating the sitemap).
`alternateRefs`	List of language-specific alternative URLs.

You can provide different values to these configuration options by following the next-sitemap configuration documentation. For example, if you want to add a new locale to the sitemap, you need to update the languageMapper object.

/* Website URL */

const { languageMapper } = require('./project.config');

const siteUrl = process.env.SITE_URL;

const alternateRefs = [];
for (const [key] of Object.entries(languageMapper)) {
  alternateRefs.push({
    href: `${siteUrl}/${key}`,
    hreflang: key,
  });
}
/* Site map generator configuration */

/** @type {import('next-sitemap').IConfig} */
const config = {
  siteUrl,
  exclude: [
    '/sitemap-static.xml',
    '/sitemap-categories.xml',
    '/sitemap-products.xml',
    '/verify',
    '/__preview',
  ],
  generateRobotsTxt: true,
  robotsTxtOptions: {
    policies: [
      {
        userAgent: '*',
        allow: '/',
      },
      {
        userAgent: '*',
        disallow: '/api',
      },
      {
        userAgent: '*',
        disallow: ['/__preview'],
      },
    ],
    additionalSitemaps: [
      `${siteUrl}/sitemap-static.xml`,
      `${siteUrl}/sitemap-categories.xml`,
      `${siteUrl}/sitemap-products.xml`,
    ],
  },
  alternateRefs: alternateRefs,
};

module.exports = config;

Generating the sitemap

To generate the sitemaps, we need to fetch the list of all available paths, create a standard sitemap structure by adding necessary fields such as loc and lastmod to the sitemap response, and return the response in the XML format.

We use the helper methods from the frontastic package to get the list of paths and generate 3 sitemaps one for each: static pages, product pages, and category pages for easier maintenance.

Sitemap for static pages

To generate the sitemap for all the static pages created using the Studio, we create a new file, sitemap-static.xml/index.tsx, in the frontend/pages/ directory. In this file, we implement the getServerSideProps method of Next.js that returns the sitemap response.

In the following code, we get the static pages structure using the frontastic.getStructure() method and convert the received data into ISitemapField structure required by next-sitemap. Then we pass the request context and sitemap data fields to the getServerSideSitemap method of the next-sitemap package and return the response.

The generated sitemap is made available on {your-site}/sitemap-static.xml.

pages/sitemap-static.xml/index.tsxTypeScript

import { GetServerSideProps } from 'next';
import { getServerSideSitemap, ISitemapField } from 'next-sitemap';
import { siteUrl } from 'next-sitemap.config';
import { mapLanguage } from 'project.config';
import { createClient } from 'frontastic';

export const getServerSideProps: GetServerSideProps = async (context) => {
  const fields = [] as ISitemapField[];
  const path = '';
  const depth = '';

  const frontastic = createClient();
  const data = await frontastic.getStructure(
    path,
    depth,
    mapLanguage(context.locale),
    context.req,
    context.res
  );

  if (data?.pageFolderStructure) {
    fields.push(
      ...data.pageFolderStructure?.map((pageFolderStructureValue) => ({
        loc: `${siteUrl}${pageFolderStructureValue._url}`,
        lastmod: new Date().toISOString(),
        changefreq: 'daily' as const,
      }))
    );
  }

  context.res.setHeader(
    'Cache-Control',
    'public, s-maxage=86400, stale-while-revalidate'
  );

  return getServerSideSitemap(context, fields);
};

export default function Sitemap() {}

Sitemap for products

To generate the sitemap for all product pages created using the Studio, we create a new file sitemap-products.xml/index.tsx in the frontend/pages/ directory. In this file, we implement the getServerSideProps method of Next.js that returns the sitemap response.

In the following code, we get a list of all products by calling the /action/product/query action extension using the fetchApiHubServerSide method and convert the received data into the ISitemapField structure required by next-sitemap. Then we pass the request context and sitemap data fields to the getServerSideSitemap method of the next-sitemap package and return the response.

The generated sitemap is made available on {your-site}/sitemap-products.xml.

import { GetServerSideProps } from 'next';
import { Product } from '@Types/product/Product';
import { Result } from '@Types/product/Result';
import { getServerSideSitemap, ISitemapField } from 'next-sitemap';
import { siteUrl } from 'next-sitemap.config';
import { mapLanguage } from 'project.config';
import { fetchApiHubServerSide } from 'frontastic';

export const getServerSideProps: GetServerSideProps = async (context) => {
  const fields = [] as ISitemapField[];

  let nextCursor: string;

  do {
    const products = (await fetchApiHubServerSide(
      `/action/product/query?cursor=${nextCursor}&limit=128&locale=${mapLanguage(
        context.locale
      )}`,
      {
        req: context.req,
        res: context.res,
      }
    )) as Result;

    fields.push(
      ...(products.items as Product[]).map((product) => ({
        loc: `${siteUrl}${product._url}`,
        lastmod: new Date().toISOString(),
        changefreq: 'daily' as const,
      }))
    );

    nextCursor = products.nextCursor;
  } while (nextCursor);

  context.res.setHeader(
    'Cache-Control',
    'public, s-maxage=86400, stale-while-revalidate'
  );

  return getServerSideSitemap(context, fields);
};

export default function SitemapCategories() {}

Sitemap for categories

To generate the sitemap for all category pages created using the Studio, we create a new file sitemap-categories.xml/index.tsx in the frontend/pages/ directory. In this file, we implement the getServerSideProps method of Next.js that returns the sitemap response.

In the following code, we get a list of all categories by calling the /action/product/queryCategories action extension using the fetchApiHubServerSide method and convert the received data into the ISitemapField structure required by next-sitemap. Then we pass the request context and sitemap data fields to the getServerSideSitemap method of the next-sitemap package and return the response.

The generated sitemap is made available on {your-site}/sitemap-categories.xml.

import next, { GetServerSideProps } from 'next';
import { Category } from '@Types/product/Category';
import { Result } from '@Types/product/Result';
import { getServerSideSitemap, ISitemapField } from 'next-sitemap';
import { siteUrl } from 'next-sitemap.config';
import { mapLanguage } from 'project.config';
import { fetchApiHubServerSide } from 'frontastic';

export const getServerSideProps: GetServerSideProps = async (context) => {
  const fields = [] as ISitemapField[];

  let nextCursor: string;

  do {
    const categories = (await fetchApiHubServerSide(
      `/action/product/queryCategories?cursor=${nextCursor}&limit=128&locale=${mapLanguage(
        context.locale
      )}`,
      {
        req: context.req,
        res: context.res,
      }
    )) as Result;

    fields.push(
      ...(categories.items as Category[]).map((category) => ({
        loc: `${siteUrl}${category._url}`,
        lastmod: new Date().toISOString(),
        changefreq: 'daily' as const,
      }))
    );

    nextCursor = categories.nextCursor;
  } while (nextCursor);

  context.res.setHeader(
    'Cache-Control',
    'public, s-maxage=86400, stale-while-revalidate'
  );

  return getServerSideSitemap(context, fields);
};

export default function SitemapCategories() {}

Final sitemap

Now that we've set up sitemap generation for all our pages, the next-sitemap package uses the sitemap configuration to generate an sitemap index that lists all the different sitemaps:

{your-site}/sitemap-static.xml
{your-site}/sitemap-products.xml
{your-site}/sitemap-categories.xml

The next-sitemap package runs as a postbuild script specified in the frontend/package.json file and generates the sitemap after the next build has finished.

After the build, the generated sitemap index is made available on {your-site}/sitemap.xml. The sitemap index provides the URL to all three sitemaps we created, allowing search engines and web crawlers to index your website.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>{your-site}/sitemap-static.xml</loc></sitemap>
  <sitemap><loc>{your-site}/sitemap-categories.xml</loc></sitemap>
  <sitemap><loc>{your-site}/sitemap-products.xml</loc></sitemap>
</sitemapindex>