# Populate a Store-specific external search

The [Store](/api/projects/stores.md) and [Product Selection](/api/projects/product-selections.md) data models allow you to build Store-specific search indices that can be used for querying Store-specific product data to power Store-specific frontends. The following guide explains how a commercetools-external search service can be integrated with commercetools to build several Store-specific search indices and how to use [Subscriptions](/api/projects/subscriptions.md) and [Messages](/api/projects/messages.md) to keep the search indices up to date.

This guide assumes that each search index contains the product data available for exactly one Store.

## Initial Setup

This section explains which steps need to be completed to set up the integration between commercetools, a message queuing service, and an external search service. The first four steps are specific to the service providers and therefore not explained here in detail.

1. Set up Store-specific search indices.
2. Set up a message queue for notifications about changes on Products.
3. Set up a message queue for notifications about changes on Product Selections.
4. Set up a message queue for notifications about changes on Stores.

Once this is complete, you need to set up three subscriptions to get notified upon changes on product data related to a Store.

### Set up Subscriptions

The following subscriptions need to be set up in your Project:

#### Subscription for Changes on Products

Create a [Subscription](/api/projects/subscriptions.md) to listen to any updates on Product with the following `changes` field:

```
{
  ...
  "changes": [
    {
      "resourceTypeId": "product"
    }
  ]
}
```

#### Subscription for Changes on Product Selections

By using [Message Subscriptions](/api/projects/subscriptions.md#messagesubscription) on [Product Selection Messages](/api/projects/messages/product-catalog-messages.md#product-selection-messages), you get notified whenever a Product was added or removed to/from a Product Selection or there have been a change on a Variant Selection.

#### Subscription for Changes on Stores

By using [Message Subscriptions](/api/projects/subscriptions.md#messagesubscription) on [Store Messages](/api/projects/messages/store-messages.md), you get notified whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection is activated or deactivated for a Store.

If the Stores' configuration does not change, you can skip this step since there is no need to subscribe to those changes.

### Initial product data feed

You have now the search indices and message queues in place and you have also set up the Subscriptions to ensure
you will not miss any update on Product data from now on. What is left to do, is to fill the search indices with Product data since the indices are still empty.

For each search index you have to find all Products available for the respective Store:

1. Given a [Store](/api/projects/stores.md), [iterate](/api/general-concepts.md#iterate-over-all-elements) over [all Product Selection Assignments](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-selection-assignments%3AGET) to get Product References.
   You can sort and filter on the `product.id` field for that purpose.
   Keep in mind that the response will include **duplicate Products** whenever **more than one active Product Selection** of the given Store includes a Product, so you might want to keep them distinct on your side.
2. For each Product, fetch the respective [Product Projection in the Store](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-projections%2F%7Bid%7D%3AGET) and add it to your search index.

```mermaid
sequenceDiagram
actor App
participant Product Selections
participant Product Projections
loop iterate over all Product Selection Assignments in a given Store
 App->>Product Selections: Query Products assigned to active Product Selections in Store
 activate Product Selections
 Product Selections-->>-App: ProductSelectionAssignment[]
 loop for each ProductSelectionAssignment
	App->>App: get product.id
	App->>Product Projections: Get ProductProjection in Store by ID
	activate Product Projections
	Product Projections-->>-App: ProductProjection
	App->>App: add Product to search index
 end
end
```

## Keep the search indices up to date

Feeding the search indices with initial product data is now complete. Over time it is likely that the Product data or the Product Selection will change.
You need to reflect these changes in the search indices to keep them up to date in the storefront, too.

### React to changes on Products

Any change on a Product triggers a Message to be sent to the [Product queue](/tutorials/store-specific-external-search.md#subscription-for-changes-on-products) you have set up before.
You react upon such messages in the following way:

#### Check which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step and [Update the search index](/tutorials/store-specific-external-search.md#update-the-search-index). Otherwise you need to fetch all the Stores in which the updated Product is available by:

1. finding out which [Product Selections the Product belongs to](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fproducts%2F%7Bid%7D%2Fproduct-selections%3AGET).
2. [iterating](/api/general-concepts.md#iterate-over-all-elements) over all returned Product Selections and [checking which Stores are impacted](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fstores%3AGET).

In the second step you use following `where` query to filter for Stores that contain activated Product Selections with the changed Product:

```
productSelections(active=true and productSelection(id in :productSelectionIds))
```

and you pass `productSelectionIds` as [input variables](/api/predicates/query.md#input-variables).

#### Update the search index

After you found out in which Store(s) the updated Product is available, you can now update the respective search indices with the changed product information.

For each impacted Store,

- if the Product was deleted, it can be removed from the search index of the corresponding Store.
- if the Product was updated, fetch the [projected Product in the Store](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-projections%2F%7Bid%7D%3AGET).
  - If the Product is available in the Store, put the returned data into the search index of the corresponding Store.
  - If a [Not Found](/api/errors.md#resourcenotfound) error is returned, the Product is not available in this Store and you delete it from the search index of the corresponding Store.

As an optimization, we recommend comparing the version of the updated Product with the version stored in the search index, and to only update in case the version number from the received message is higher than the number in the search index.

### React to changes on Product Selections

In the section before you handled changes on the individual Products, but you also want to update our Store's search index
whenever a new Product was added or removed to/from the Store's Product Selections. This case is not covered by changes on Products but by messages on the [Product Selection queue](/tutorials/store-specific-external-search.md#subscription-for-changes-on-product-selections) you set up earlier like this:

#### Check which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step and [Update the search index](/tutorials/store-specific-external-search.md#update-the-search-index-1). Otherwise you need to fetch all the Stores impacted by the change.

From the [ProductSelectionProductAdded](/api/projects/messages/product-catalog-messages.md#product-selection-product-added) Message,
the [ProductSelectionProductRemoved](/api/projects/messages/product-catalog-messages.md#product-selection-product-removed) Message,
or the [ProductSelectionVariantSelectionChanged](/api/projects/messages/product-catalog-messages.md#product-selection-variant-selection-changed) Message,
you get the ID of Product Selection.

You can [check which Stores are impacted](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fstores%3AGET) by using the following `where` query to filter for Stores that contain the updated Product Selection:

```
productSelections(active=true and productSelection(id = :productSelectionId))
```

and you pass `productSelectionId` as [input variables](/api/predicates/query.md#input-variables).

#### Update the search index

If a Product was added to a Product Selection,

1. fetch the [Product Projection in the Store](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-projections%2F%7Bid%7D%3AGET) and
2. put the returned data into the search index of the corresponding Store.

If the Variant Selection of a Product to a Product Selection was changed,

1. fetch the [Product Projection in the Store](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-projections%2F%7Bid%7D%3AGET) and
2. update the search index of the corresponding Store with the returned data.

If a Product was removed from a Product Selection, delete it from the search index of the corresponding Store.

### React to changes on Stores

You also want to update our Store's search index whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection was activated or deactivated in a Store. You handle this by processing messages from the [Store queue](/tutorials/store-specific-external-search.md#subscription-for-changes-on-stores) you set up earlier like this:

#### Check which Stores are impacted

You get the ID of the impacted Store from the [Store Messages](/api/projects/messages/store-messages.md).

#### Update the search index

In case a [Store was created](/api/projects/messages/store-messages.md#store-created), feed all product data into the new search index as described in [initial product data feed](/tutorials/store-specific-external-search.md#initial-product-data-feed).

In case a [Store was deleted](/api/projects/messages/store-messages.md#store-deleted), you can remove the corresponding search index.

In case the [Product Selection configuration of a Store was changed](/api/projects/messages/store-messages.md#store-product-selections-changed), you have two options:

- You ignore the details from the [StoreProductSelectionsChanged](/api/projects/messages/store-messages.md#store-product-selections-changed) Message and completely reindex all data for this Store as described in [reindex all data from time to time](/tutorials/store-specific-external-search.md#reindex-all-data-from-time-to-time).
- Alternatively, you are more specific with our updates and reindex only with those Products that are affected by the configuration change:
  1. For each Product Selection that was added, removed, activated or deactivated, you can [iterate](/api/general-concepts.md#iterate-over-all-elements) on [all Products' References](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fproduct-selections%2F%7Bid%7D%2Fproducts%3AGET).
  2. For each Product, you fetch the [projected product in the Store](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-projections%2F%7Bid%7D%3AGET).
  - If the Product is not found you remove it from the search index.
  - Otherwise you put this data into the Store's search index.

Store configuration is not expected to be changed frequently. As a consequence, the Store configuration is cached up to one minute.
You have to respect this delay before fetching any [projected product in the Store](/urn?urn=ctp%3Aapi%3Aendpoint%3A%2F%7BprojectKey%7D%2Fin-store%2Fkey%3D%7BstoreKey%7D%2Fproduct-projections%2F%7Bid%7D%3AGET).

## Best practices

### Detect unnecessary updates

We recommend you to store the version of the Products in the search indices, and to use this information to detect if some product data is already up to date and does not need any further updates.

### Reindex all data from time to time

A search index can become out of date in case an update was missed, or due to changes on a Store's configuration. To keep such inconsistencies to a minimum, we recommend reindexing all product data in a search index, not only on demand, but also from time to time.

The following steps suggest a way how to do that without impacting your live traffic for the search index of one Store:

1. Create a second search index for this Store.
2. Let the Product updates now apply to both search indices: the live index and the new index.
3. Feed all product data into the new search index as described in [initial product data feed](/tutorials/store-specific-external-search.md#initial-product-data-feed).
4. Once the new search index is ready, move your live traffic over to it.
5. Delete the old live index that is not used anymore.

## Related pages

- [Area overview page with navigation](/tutorials.md)
- [Previous page: Integrate external search](/tutorials/search-integration.md)
- [Next page: Integrate tax](/tutorials/tax-integration.md)