Populating a Store-specific external search

The Store and Product Selection data models allow you to build Store-specific search indices that can be used for querying Store-specific product data to power Store-specific frontends. The following guide explains how a commercetools-external search service can be integrated with the commercetools Platform to build several Store-specific search indices and how to use commercetools Subscriptions and Messages to keep the search indices up to date.

This guide assumes that each search index contains the product data available for exactly one Store.

Initial Setup

This section explains which steps need to be completed to set up the integration between the commercetools Platform, a message queuing service, and an external search service. The first four steps are specific to the service providers and therefore not explained here in detail.

  1. Set up Store-specific search indices.
  2. Set up a message queue for notifications about changes on Products.
  3. Set up a message queue for notifications about changes on Product Selections.
  4. Set up a message queue for notifications about changes on Stores.

Once this is complete, you need to set up three subscriptions to get notified upon changes on product data related to a Store.

Set up Subscriptions

The following subscriptions need to be set up on the commercetools Platform:

Subscription for Changes on Products

Create a Subscription to listen to any updates on Product with the following changes field:

{
...
"changes": [
{
"resourceTypeId": "product"
}
]
}

Subscription for Changes on Product Selections

By using Message Subscriptions on Product Selection Messages, you get notified whenever a Product was added or removed to/from a Product Selection.

Subscription for Changes on Stores

By using Message Subscriptions on Store Messages, you get notified whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection is activated or deactivated for a Store.

When you are certain that the Stores' configuration does not change, you can skip this step since there is no need to listen to those changes.

Initial product data feed

You now have the search indices and message queues in place and you also set up the Subscriptions that make sure you will not miss any update on the product data from now on. What is left to do, is to fill the search indices with product data since the indices are still empty.

For each search index you have to find all Products available for the respective Store:

  1. first fetch all active Product Selections of the Store.
  2. for each Product Selection, iterate on all Products' References.
  3. for each Product, fetch the projected product in the Store to put this data into the search index.

Keeping the search indices up to date

Feeding the search indices with initial product data is now complete, but over time it is likely that product data or the Product Selection is changed on the commercetools Platform and you need to reflect those changes in the search indices to be up to date in the store front also.

Reacting to changes on Products

Upon every change on a Product, the commercetools Platform sends a Message to the Product queue you set up before. You react upon such message in the following way:

Checking which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step by going straight to Updating the search index. Otherwise you need to fetch all the Stores in which the updated Product is available by:

  1. finding out which Product Selections the Product belongs to.
  2. iterating over all returned Product Selections and checking which Stores are impacted.

In the second step you use following where query to filter for Stores that contain activated Product Selections with the changed Product:

productSelections(active=true and productSelection(id in :productSelectionIds))

and you pass productSelectionIds as input variables.

Updating the search index

Since you now found out to which Store(s) the updated Product belongs, you can now update the respective search indices with the changed product information.

For each impacted Store,

  • if the Product was deleted, it can be removed from the search index of the corresponding Store.
  • if the Product was updated, fetch the projected Product in the Store.
    • If the Product is available in the Store, put the returned data into the search index of the corresponding Store.
    • If a 404 Not Found error is returned, the Product is not available in this Store and you delete it from the search index of the corresponding Store.

As an optimization, we recommend comparing the version of the updated Product with the version stored in the search index, and to only update in case the version number from the received message is higher than the number in the search index.

Reacting to changes on Product Selections

In the section before you handled changes on the individual Products, but you also want to update our Store's search index whenever a new Product was added or removed from the Store's Product Selections. This case is not covered by changes on Products and you need to handle it by processing messages from the Product Selection queue you set up earlier like this:

Checking which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step by going straight to Updating the search index. Otherwise you need to fetch all the Stores impacted by the change.

From the ProductSelectionProductAddedMessage or the ProductSelectionProductRemovedMessage you get the ID of Product Selection.

You can check which Stores are impacted by using the following where query to filter for Stores that contain the updated Product Selection:

productSelections(active=true and productSelection(id = :productSelectionId))

and you pass productSelectionId as input variables.

Updating the search index

If a Product was added to a Product Selection,

  1. fetch the Product Projection in the Store and
  2. put the returned data into the search index of the corresponding Store.

If a Product was removed from a Product Selection, delete it from the search index of the corresponding Store.

Reacting to changes on Stores

You also want to update our Store's search index whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection was activated or deactivated in a Store. You handle this by processing messages from the Store queue you set up earlier like this:

Checking which Stores are impacted

You get the ID of the impacted Store from the Store Messages.

Updating the search index

In case a Store was created, feed all product data into the new search index as described in initial product data feed.

In case a Store was deleted, you can remove the corresponding search index.

In case the Product Selection configuration of a Store was changed, you have two options:

Store configuration is not expected to be changed frequently. As a consequence, the Store configuration is cached up to one minute. You have to respect this delay before fetching any projected product in the Store.

Best practices

Detect unnecessary updates

We recommend you to store the version of the Products in the search indices, and to use this information to detect if some product data is already up to date and does not need any further updates.

Reindex all data from time to time

A search index can become out of date in case an update was missed, or due to changes on a Store's configuration. To keep such inconsistencies to a minimum, we recommend reindexing all product data in a search index, not only on demand, but also from time to time.

The following steps suggest a way how to do that without impacting your live traffic for the search index of one Store:

  1. Create a second search index for this Store.
  2. Let the Product updates now apply to both search indices: the live index and the new index.
  3. Feed all product data into the new search index as described in initial product data feed.
  4. Once the new search index is ready, move your live traffic over to it.
  5. Delete the old live index that is not used anymore.
Developer Center
HTTP APIGraphQL APIPlatform Release NotesCustom ApplicationsBETASDKs & Client LibrariesImport & ExportSUNRISE Starter FrontendsTutorialsFAQ
Merchant Center
DocumentationRelease Notes
Sign upLog inTech BlogIntegrationsStatusSupportUser Research Program
Copyright © 2022 commercetools
Privacy PolicyImprint