Integrate external search

Data flows in one direction. commercetools is the source of truth for the catalog, and the search engine holds a denormalized copy optimized for querying. commercetools never reads back from the engine.

Before you start, you need the following:

• a commercetools Project with published Products, and an API Client that can read them
• an account and API credentials for the search engine you intend to use
• somewhere to run the integration, such as Connect, a cloud function, or your own infrastructure

The following topics are out of scope: relevance tuning inside the engine, storefront rendering, and inventory as a live stock ledger. Each is covered as a boundary rather than as an implementation step.

Decide what you need

When an external search engine is justified

commercetools ships its own search. Rule it out before you commit to an indexing pipeline, because an external engine is a standing system to own, secure, monitor, and pay for.

• visual or AI-driven merchandising and manual curation
• synonym, redirect, or query-rule sets that merchandisers maintain themselves
• recommendations, such as "customers also bought"
• search analytics dashboards
• A/B testing of ranking
• learned or personalized ranking
• a discovery engine your storefront already uses

The reverse case matters too. Suppose your organization already licenses an engine and the merchandising team works in it daily. Staying with the built-in search is then rarely a real option, even when the query requirements look simple. The requirement is merchandising in the engine, which puts you past this gate.

Requirements that shape the design

The architecture follows from a handful of answers. Settle them before you write code or configure a Connector, because each one changes the shape of the search document or the applications around it.

Write the answers down and confirm them with whoever owns the storefront. Most later rework traces back to a price context or a record granularity that was assumed rather than decided.

Choose how to build it

Use, configure, fork, or build

Stop at the first of the following options that fits:

1. The built-in search is enough. Build nothing. See When an external search engine is justified.
2. A public Connector for your engine covers everything. Install and configure it. For installation, see the Connect getting started guide.
3. The right engine, but a gap that looks like a missing capability. Prove it isn't configuration first. Index name, indexed fields, locale, price context, and Store selection are usually settings rather than code. If configuration closes the gap, you are back at the previous option.
4. A genuine gap that configuration cannot close, and source is available. Fork the existing Connector, add only the difference, and deploy it as an Organization Connector.
5. No usable Connector for your engine. Build one, scaffolding from the Product export template.

The last two options are materially more work than the others. Decide explicitly rather than defaulting to a build.

Available Connectors and templates

The method in this guide is the same for any engine. Algolia, Constructor, Bloomreach, Coveo, Elasticsearch, Typesense, Meilisearch, and an in-house engine all share the same requirements and the same gate. They also share the same mapping method and the same two-application architecture. Only the client library and the index vocabulary change.

What the engine owns

Searchable-field weighting, ranking, tie-breaking, synonyms, redirects, query rules, merchandising, and A/B tests live in the search engine and are configured by merchandisers. They belong neither in commercetools nor in the mapping.

The integration has one job: keep each record equal to the current published Product. The engine decides relevance.

Design the search document

This is where a search integration succeeds or degrades over time. Moving the data is mechanical. The mapping decides whether the index stays correct, queryable, and affordable.

The tension is structural. The commercetools model is normalized and reference-based, so Products reference Categories, Prices, and Channels by identifier. A search engine queries most efficiently against a flat, self-contained record built for one query. The mapping is a transform and a filter, not a copy. Index only what the storefront searches, filters, sorts, or displays.

Read the published projection

This single choice prevents the most common leak in search integrations: unreleased names, prices, and draft Products appearing in customer-facing results.

Key every record for idempotent writes

Give each record a deterministic identifier derived from a stable commercetools identifier:

• the Product id for product-level records
• the Product id combined with the Variant id, or the SKU, for variant-level records

If you cannot reconstruct a record's key from a later message, you cannot update or delete that record. Settle the key before you write any synchronization code.

Choose product-level or variant-level records

This is a storefront design decision, and it shapes the entire document. Choose one of two granularities:

• Product-level, one record per Product: variant-specific values such as size and color become sets aggregated across Variants. A hit links to the product detail page. Fewer records, and the simpler default for most catalogs.
• Variant-level, one record per Variant: each color or size is its own hit with its own image and price. Needed when the results grid shows "red shirt" and "blue shirt" as separate tiles. Many more records, so check the engine's record-count pricing.

Match the choice to how the storefront displays results, and keep it consistent. Do not mix granularities in one index.

Resolve the price context

• Index one context, for example euros in Germany. The simplest option, and correct only for a single-market storefront. Select the context with the price-selection query parameters at map time.
• Index one field per context, for example price_EUR_DE and price_USD_US. One record with several price fields, and the storefront reads the field matching the shopper. This scales to a handful of contexts.
• Emit one record per context, with a context attribute you filter on. Use this when contexts are numerous, or when B2B Customer Group pricing must be searchable. It multiplies the record count.

State the choice explicitly. A mismatch here produces the classic wrong-price-in-search defect. The record exists and looks healthy, but it shows a price no shopper is entitled to.

Handle locales

• One index per locale, for example products_en and products_de. The cleanest option for language-specific relevance configuration, and the usual choice for multi-market and Store-specific setups.
• One index with per-locale fields, for example name_en and name_de. Fewer indices, and the storefront queries the fields for the shopper's language. Fine for a few locales.

Denormalize categories

Scope by Store, Product Selection, and Product Tailoring

• One index with a Store filter field. Each record carries the Stores or Product Selections it belongs to, and the storefront filters by the current Store. Simple, and a good fit when tailored content per Store is minimal.
• One index per Store. Driven by each Store's Product Selection, reading /in-store/key={storeKey}/product-projections. The in-store projection also resolves the Store's languages and distribution channels, so tailored names, descriptions, images, and assets arrive already applied. This is the shape the Product export template implements, with one Deployment per Store.

Decide where availability belongs

Keep the search index out of the role of live stock ledger, and keep per-unit inventory events out of it. That write volume overwhelms the engine and costs money without improving the shopper's experience.

Keep the mapping a pure function

Write the projection-to-document transform as a pure function with no network calls. Everything the engine needs to rank and display should already be in the record. The applications around the mapping should do nothing but move data.

Worked example

An apparel catalog with product-level records, two locales, one price context, and one global index.

That record schema is the deliverable. It is the same artifact whether a public Connector consumes it as configuration or a Connector you build implements it in code.

Build the two applications

A search integration is two jobs, so build two applications. Keep them separate rather than putting a mode switch in one application.

One rule spans both. The index is a derived copy of the published catalog. Every record must be reproducible from the current Product Projection. Every write is an upsert on the record key, and every removal targets that same key. Both full loads and Subscription deliveries are at-least-once. Running either application twice must therefore converge rather than duplicate or double-delete.

Full ingestion

Full ingestion rebuilds the entire index from commercetools. It is your initial load, your disaster-recovery path, and the periodic backstop that repairs whatever the incremental path missed.

Incremental updates

Incremental updates keep the index in step with catalog changes between full loads. Choose the trigger first.

Propagate deletions

Deletion is a first-class path, not an afterthought. Every trigger that should remove a record needs a defined action:

• a Product is unpublished or deleted
• a Product is removed from a Product Selection that scopes a Store's index
• a Product is no longer available in a Store, which the in-store projection reports as a ResourceNotFoundError
• a Store is deleted, which retires that Store's index

Skipping any of these leaves ghost records: results that still appear in search and link to a product detail page that no longer exists.

On the Store-scoped path, a Not Found response has two possible causes. The Product is genuinely outside the Store's assortment, or the Store configuration cache has not caught up. Wait out the cache before you treat a Not Found as a removal. Otherwise a routine assortment change removes Products that should stay indexed.

Repair drift

Even a correct incremental path drifts. A Category rename fans out to Products that generated no Product Message. A notification is lost during a deployment gap. A Store configuration change reshapes an assortment. Run a periodic full rebuild as the backstop, and reconcile record counts between the engine and the published catalog.

To rebuild without disturbing live traffic, follow these steps:

1. Create a second index for the same content.
2. Apply incremental updates to both the live index and the new one.
3. Run full ingestion into the new index.
4. Move live traffic to the new index once it is complete and its count checks out.
5. Delete the previous index.

To avoid unnecessary writes, store the Product version alongside each record. Skip the update when the version you already hold is not older than the incoming one.

Configure scopes, secrets, and the trigger

A search integration reads from commercetools and writes to the engine, so it needs no write scope on the catalog. Grant the following scopes:

• Grant view_products to both applications, and add view_product_selections and view_stores for a Store-scoped index.
• Grant manage_subscriptions only to the application whose deployment step registers the Subscription.
• Grant scopes per application rather than per Connector, and check the current list in Scopes. A search integration that holds manage_products is over-privileged.
• The engine needs no commercetools scope. It is reached with the engine's own API key.

Store the engine's API key as secured configuration rather than as plain configuration, and keep it out of logs and error responses. Index name, region, locale list, price context, and behavioral toggles are ordinary configuration.

What the templates leave you to build

Read the current source of any repository before you fork it, rather than relying on this table or on a previous reading. These are starting points that reduce the boilerplate, not Connectors you can deploy as they are.

Test the contracts you cannot see

Most of what breaks a search integration is invisible in a code review, so cover it with automated tests. Mock both boundaries, the engine and the commercetools APIs, so the suite runs with no deployment and no secrets.

For full ingestion, assert that:

• a trigger without authentication, or with an invalid signature, is rejected
• pagination uses the identifier cursor rather than an offset
• the rebuild is atomic, so the live index is never empty or partial mid-run
• the count check compares engine records against published Products and fails the run on a mismatch
• a Store-scoped load reads in-store projections and deduplicates Products held in more than one Selection

For incremental updates, assert that:

• ProductPublished upserts from the payload projection, and a second delivery of the same notification changes nothing
• ProductUnpublished and ProductDeleted remove the record by its key
• Store and Product Selection Messages add to and remove from the correct Store's index
• an out-of-order notification does not overwrite newer data
• the envelope is decoded, and handled, ignored, and retryable messages each return the intended status
• for the polling variant, the checkpoint advances and removals are covered elsewhere

Test the mapping directly as a pure function. Given a Product Projection, it should produce the exact record you expect, including the selected price context, the locale fields, and the denormalized categories.

Verify and operate

Treat the integration as unfinished until a change appears, a removal disappears, and a rebuild matches the catalog. Observe each of those in the engine's index rather than in your own logs.

Verify that a publish reaches the index

Publish a Product, or change and republish one, then confirm that:

• a record with the expected key exists. It carries the mapped fields: names and descriptions in each in-scope locale, the selected-context price, the denormalized categories, and the image
• searching for a word from the Product name returns it, and its facets are populated
• only published data is present, with no staged edits and no unpublished Products
• redelivering the same notification changes nothing, because the write is an upsert on a stable key

Verify that a removal disappears

Unpublish a Product, and separately delete one, then confirm the record is gone from the index and no longer returned by search. This is the check people skip, and skipping it is how ghost records reach production.

If you poll rather than subscribe, confirm that removals are covered by removal Messages or by the periodic rebuild.

Verify that a rebuild matches the catalog

Trigger full ingestion against a known catalog, then confirm that:

• the engine's record count equals the published Product count, multiplied by your granularity factor
• the index never went empty or partial during the rebuild, checked by querying it while a rebuild is running
• re-running full ingestion produces an identical index, with the same count and the same records

Verify that Store scoping holds

For a Store-scoped index, add a Product to one Store's Product Selection and confirm it appears in that Store's index only. Remove it and confirm it disappears from that index while remaining in the other Stores that still list it. Confirm that the in-store projection resolved the Store's languages and prices rather than the Project defaults. Where Product Tailoring applies, confirm that the tailored content arrived.

Signals that look like failures

Three correct behaviors read as defects, and one defect reads as flakiness. Recognize them before you escalate.

Symptom, cause, and resolution

Query the index from your storefront

Reading the index is your storefront's job, and it sits outside this integration. The engine's own client library handles querying, faceting, and pagination.

If you build on commercetools Frontend, two pieces connect a storefront to an external engine:

• Frontend components render the search experience, such as type-ahead search and a filterable product listing page. The Store Launchpad for B2C Retail ships Algolia components you can use as a reference. See Algolia integration.
• Extensions run in the API hub and orchestrate calls to the engine, to commercetools, and to any other service.

Some pricing rules cannot be expressed in the engine, such as price hints or Customer Group resolution. Keep those in your storefront or backend-for-frontend layer rather than encoding them in the index.