# Integrate an analytics destination

Export commerce data from commercetools to a data warehouse, customer data platform, or product analytics tool, and keep that copy correct as the data changes.

Learn more about integrations in the self-paced [Integration patterns](/learning-integrate-with-commercetools/integration-patterns/overview.md) module.

This guide covers the export pipeline that moves commerce data out of commercetools so that another system can report on it. It starts with deciding which kind of analytics you need, moves through choosing a destination and an integration path, and ends with the two applications that keep the destination data synchronized with your Project.

Data flows in one direction. commercetools is the source of truth, and the destination holds a copy shaped for querying. commercetools never reads back from the destination. This design runs the pipeline asynchronously outside the checkout path, so an outage produces stale reports rather than a broken checkout.

Before you start, you need the following:

- a commercetools Project with the data you want to export, and an API Client that can read it
- an account and ingestion credentials for the destination
- somewhere to run the integration, such as [Connect](/connect), a cloud function, or your own infrastructure
- an agreed answer to each question in [Identify requirements that shape the pipeline](/guides/analytics-export-integration.md#identify-requirements-that-shape-the-pipeline)

## Define analytics export requirements

The destination, the integration path, and the pipeline design depend on these decisions. Record them before you configure or build anything.

### Confirm you need a commerce data export

Three unrelated things are called analytics, and they route to different products. Confirm which one your requirement is before you read further, because only the first is an export pipeline.

| What you want | Where it comes from | This guide |
| --- | --- | --- |
| Transactional and state truth for reporting, such as Orders, Customers, Payments, Inventory, and catalog | An export pipeline you own, built on [Subscriptions](/api/projects/subscriptions.md) and API queries. | Covered here |
| API latency, error rates, request volume, and server-side request logs | [Platform Insights](/api/platform-insights.md), which forwards telemetry to an application performance monitoring provider. See [Set up Platform Insights](/guides/apm-integration.md). | Not covered |
| A governance record of who changed what and when | [Change History](/api/history/overview.md), a compliance log on a separate host. | Not covered |

The Change History boundary and the client-side tracking boundary are worth stating plainly, because both mistakes are expensive to reverse.

Change History is not an analytics feed. It runs on [its own hosts](/api/history/change-history.md#hosts), and it is rate limited: exceeding the limit returns [Too Many Requests](/api/errors.md#too-many-requests) with a `Retry-After` header. The documentation also advises you to [avoid making API calls in response to an event stream or message subscription](/api/history/overview.md#best-practices). Audit Log Basic records only changes that originate in the Merchant Center, so changes made through the [HTTP API](/api) and the [Import API](/api/import-export/overview.md) are absent unless you have the [Audit Log Premium Add-On](/offering/audit-log-premium.md). A pipeline built on Change History is both throttled and incomplete.

Client-side behavioral tracking is a storefront concern. Page views, clicks, and consent-managed pixels belong in your storefront with a tag manager, not in this pipeline. An export can feed the server-side ingestion endpoint of the same vendor, but it cannot produce browser events.

### Identify requirements that shape the pipeline

Answer and record the following, because each answer changes the design.

| Question | Design impact |
| --- | --- |
| Which destination, and which category? | Decides the ingestion mechanism and whether an existing integration might cover you. See [Choose the destination category](/guides/analytics-export-integration.md#choose-the-destination-category). |
| Which data domains and which fields? | Decides the Messages you subscribe to, the resources the backfill queries, and the read scopes you grant. |
| Does the destination need near-real-time data, periodic loads, or both? | Decides which applications exist. Most pipelines need both. |
| Is a one-time historical load required? | A historical load is a separate run from the ongoing delta, even though it shares the code path. |
| What is the destination schema grain? | One row per change in an append-only table, or one current-state row per resource. This decides the transform and the merge key. |
| What volume and change rate must it handle? | Sets page sizes, backoff, run duration, and whether the Subscription budget constrains the design. |
| Which exported fields are personal data? | Decides field minimization, and whether erasure has to propagate to the destination. |

Capture anything that does not fit these rows as its own requirement rather than forcing it into one. Each of the following can move the decision in the next section:

- consolidating several Projects or Regions into one destination
- data residency rules that restrict where the destination and the runtime may sit
- an existing extract, transform, and load tool or customer data platform that can already read the commercetools API
- a retention or erasure policy that the destination must honor
- near-real-time personalization built on the same feed, which raises the latency requirement

Unless a requirement says otherwise, the design that follows is the default: a streaming application for freshness, a scheduled job for history and gap repair, read-only scopes, destination credentials in secured configuration, and deduplication on the destination side.

### Choose the destination category

The category decides the mechanism. Vendor documentation is authoritative for ingestion endpoints, field names, and limits.

| Category | Examples | Mechanism | What flows |
| --- | --- | --- | --- |
| Data warehouse | BigQuery, Snowflake, Redshift, Databricks | Stream and batch. This is the default fit. | All transactional and state data, modeled downstream |
| Customer data platform | Segment, mParticle, RudderStack, Tealium | Stream. Check first whether the platform has its own commercetools source. | A curated set of Customer and Order events keyed to a stable identifier |
| Product analytics | Amplitude, Mixpanel, GA4, Snowplow | Stream a small set of conversion events into the server-side ingestion API. | Conversion and behavioral events, not Orders as rows |
| Business intelligence | Looker, Tableau, Power BI | None. These read the warehouse. | Nothing directly from commercetools |

Two of these rows carry a decision that is easy to get wrong.

Business intelligence tools do not integrate with commercetools directly. Point the pipeline at a warehouse and let the reporting tool read modeled tables from there. Building an extract for a reporting tool alone produces a pipeline with no history and no place to model.

Product analytics tools that are designed for browser instrumentation, such as GA4 and Mixpanel, expose a narrower server-side ingestion API than their browser SDK. Confirm what each one accepts before you treat it as a substitute for a warehouse. If the requirement is complete history and modeling, choose a warehouse and treat the analytics tool as an additional destination rather than the only one.

## Choose an integration path

Work through these options in order and stop at the first one that meets your requirements.

### Check for an existing analytics Connector

Check the current Connector inventory rather than relying on a remembered list, because published Connectors and their versions change. Use an API Client with the `view_connectors` scope to call the [Search Connectors](/connect/connectors.md#search-connectors) endpoint and filter on the analytics integration type:

```http title="Search for analytics Connectors"
GET https://connect.{region}.commercetools.com/connectors/search?integrationTypes=analytics&private=false
```

Pass `private=false` so the response covers all available Connectors rather than only the ones already assigned to your Project. For the host to use in each Region, see [Hosts and authorization](/connect/hosts-and-authorization.md). Search for your destination by name as well, because a Connector that writes to a warehouse may be registered under a different [IntegrationType](/search.md?urn=ctp:connect:type:IntegrationType).

Record the Connector key and version you evaluated, or record that none exists. Plan for a custom build until the search identifies an option that meets your requirements, and record that assumption in the effort estimate.

Two adjacent options exist and neither is a Connect deployment:

- A customer data platform may ship its own commercetools source, configured in that vendor's console. If it covers your domains, use it. Nothing in this guide's build and deploy sections applies.
- An extract, transform, and load tool that already reads the commercetools API may be the cheapest path if your organization runs one. It is a valid alternative, and it is not built or deployed through Connect.

The [technology partner directory](https://marketplace.commercetools.com/connectors) includes integrations with different deployment models. Confirm the deployment model of each candidate before you plan a Connect deployment. For installable Public Connectors, use [Connect in the Merchant Center](/merchant-center/connect.md).

### Prove that a gap is not configuration

When a Connector matches your destination but appears to be missing a behavior, confirm that configuration cannot close the gap before you consider forking it. Which Messages flow, how fields map to destination columns, and which table or dataset receives them are usually configuration.

### Fork or build

If a Connector matches your destination and has a genuine gap that configuration cannot close, and its source is available, fork it. Add only the difference and deploy the result as an Organization Connector.

If no Connector matches your destination, build one. Decide where the integration runs before you scaffold it. For a comparison of vendor middleware, integration platforms, cloud services, and Connect, see [Compare middleware and runtime options](/guides/erp-integration.md#compare-middleware-and-runtime-options).

Size the workload against Connect before you choose it. Connect names [batch processing, long running processes, and complex analytics or data processing jobs](/connect/best-practices.md#when-to-use-an-alternative-to-connect) as cases to host elsewhere. A delta export that streams pages to the destination and checkpoints its progress fits inside the Connect limits. A one-time load of years of Order history, or a run that assembles a dataset before writing it, does not. Run those on infrastructure you control and keep Connect for the ongoing delta.

The closest starting point in Connect is the [Product export template](/connect/templates/product-export.md), which already has the two-application shape this guide describes: a full export that reads through the API, and an incremental updater that reacts to Messages. The CLI registers this template under the name `product-ingestion`.

```bash title="Initialize a Connector from the Product export template"
commercetools connect init my-analytics-connector --template product-ingestion
```

The Product export template supplies application structure, not a working analytics export. The following observations apply only to the [audited template source](https://github.com/commercetools/connect-product-export-template/tree/1cba7d2e50b7d2e63bf45d67904b9735162330c8), inspected on 9 September 2026, and are not platform guarantees. Before production use, inspect the version you deploy and implement the following where needed:

- The destination calls themselves. The `saveProducts` and `removeProducts` functions in both applications log a count and return. No destination client exists.
- A scheduled backfill. The template has no `job` application, no `lastModifiedAt` window, and no checkpoint. The full export is a `service` application that runs on demand.
- Your own resources and Messages. The template exports Products from a single Store and subscribes to Product, Product Selection, and Store changes. An analytics export usually needs Orders, Customers, and Payments instead.
- Authentication on the full export endpoint. It accepts unauthenticated requests that trigger a complete export.
- Idempotent Subscription registration. The `postDeploy` script deletes the Subscription and creates a new one, so every change emitted between the two calls is lost.
- Least-privilege permissions. The template expects a pre-created API Client supplied through secured configuration rather than declaring `inheritAs.apiClient.scopes`.
- Bounded memory use. The full export collects the entire result set in memory before writing anything.
- A non-destructive rebuild. The full export removes the existing records before it writes the new ones, which leaves the destination empty for the length of the run.
- Meaningful tests. The suite asserts only that a response object exists.

Whichever option you choose, the pipeline design in the next section is the same.

## Design the export pipeline

An analytics export is made of two applications that write to the same destination. The streaming application keeps the destination fresh. The scheduled job loads history and repairs whatever the stream missed. Both produce the same rows, so both must be safe to run more than once.

```mermaid title="Stream and batch paths of an analytics export"
flowchart LR
	subgraph ct["commercetools"]
		Resources[Orders, Customers, and other resources]
		Subscription[Subscription]
		Resources -- Emit Messages --> Subscription
	end

	subgraph integration["Your integration"]
		Streamer["Streaming application"]
		Backfill["Scheduled backfill job"]
		Checkpoint[(Checkpoint)]
	end

	Destination[(Analytics destination)]
	Reporting[Reporting and modeling]

	Subscription -- Deliver notifications --> Streamer
	Resources -- Query by lastModifiedAt --> Backfill
	Backfill <-- Read and write cursor --> Checkpoint
	Streamer -- Rows with a deduplication key --> Destination
	Backfill -- Rows with a deduplication key --> Destination
	Destination --> Reporting
```

### Stream changes as they happen

The streaming application receives a notification, retrieves the current resource, transforms it, and writes it to the destination. Use an `event` application, which receives Messages through a Subscription and a Connect-provisioned message broker.

```mermaid title="Export one change to the destination"
sequenceDiagram
	participant CT as commercetools
	participant Sub as Subscription
	participant Streamer as Streaming application
	participant Dest as Analytics destination
	CT->>Sub: Emit Message
	Sub->>Streamer: Deliver encoded envelope
	Streamer->>Streamer: Decode and filter by Message type
	Streamer->>CT: Get the resource by resource.id
	CT-->>Streamer: Current resource and version
	Streamer->>Streamer: Transform to destination rows
	Streamer->>Dest: Write rows with the deduplication key
	Dest-->>Streamer: Ingestion result
	Streamer-->>Sub: Acknowledge within 10 seconds
```

[Integrate an order management system](/guides/oms-integration.md) is the canonical description of this delivery contract. Read the following sections there and apply them unchanged, because an analytics export makes the same guarantees and the same mistakes:

- [Retrieve the Order instead of reading the Message payload](/guides/oms-integration.md#retrieve-the-order-instead-of-reading-the-message-payload), which covers at-least-once delivery, the absence of an ordering guarantee, and the omitted payload that arrives as [PayloadNotIncluded](/search.md?urn=ctp:api:type:PayloadNotIncluded) when a Message exceeds the queue size limit or its format is not accepted by the messaging service.
- [Acknowledge deliberately](/guides/oms-integration.md#acknowledge-deliberately), which covers the status codes the message queue treats as an acknowledgment, the seven-day retention of unacknowledged messages, and the outcome-to-status-code mapping.

Three decisions are specific to an export.

**Subscribe to every Message that represents an exported change.** Select the types you need from the [Message catalog](/api/projects/messages.md), such as [Cart and Order Messages](/api/projects/messages/cart-order-messages.md) and [Customer Messages](/api/projects/messages/customer-messages.md). Order creation is emitted under three different types, and subscribing to only one of them produces a report that looks complete and is not:

| Message | Emitted when |
| --- | --- |
| [OrderCreated](/search.md?urn=ctp:api:type:OrderCreatedMessage) | The initial Order is created directly from a Cart. |
| [OrderCreatedFromRecurringOrder](/search.md?urn=ctp:api:type:OrderCreatedFromRecurringOrderMessage) | A Recurring Order produces an Order on its schedule. |
| [OrderImported](/search.md?urn=ctp:api:type:OrderImportedMessage) | An Order is imported. |

**Choose between a Message Subscription and a Change Subscription.** A [MessageSubscription](/search.md?urn=ctp:api:type:MessageSubscription) delivers named Message types, which keeps the feed narrow and typed. A [ChangeSubscription](/search.md?urn=ctp:api:type:ChangeSubscription) delivers every change to a resource in one Subscription, which suits an export that mirrors a whole resource. One Change Subscription can replace the several Message Subscriptions that resource would otherwise need. A Project allows a maximum of **** Subscriptions. This is a soft limit that can be raised after a performance impact review, but a design with one Subscription per Message type reaches it sooner than a design that fans out inside the handler.

The trade-off is that a change notification does not identify which Message caused it. A [ResourceUpdatedDeliveryPayload](/search.md?urn=ctp:api:type:ResourceUpdatedDeliveryPayload) carries the resource reference and the version, but no Message type, so a handler fed by a Change Subscription cannot route or filter by Message type. Keep a Message Subscription wherever the handler has to tell one kind of change from another.

The two styles also produce different deduplication inputs, which the next sections rely on. A [MessageDeliveryPayload](/search.md?urn=ctp:api:type:MessageDeliveryPayload) carries `sequenceNumber` alongside `version` and `resourceVersion`. A change notification carries `version`, and on an update also `oldVersion`, but no `sequenceNumber`.

**Keep the acknowledgment fast.** Retrieving a resource and writing to a destination inside the handler can exceed the [event acknowledgment timeout](/connect/deployment-behavior-and-environments.md#event). Measure the synchronous path against it, and move slow ingestion behind a queue of your own if it does not fit.

### Backfill and repair on a schedule

The scheduled job loads history and repairs what the stream missed. Query resources to load data that predates the Subscription because Message history has limited retention and is not enabled by default.

Use a `job` application and follow the documented [Job application behavior](/connect/deployment-behavior-and-environments.md#job) and [resource recommendations](/connect/best-practices.md#job-application). The application request times out after 30 minutes, and the CPU and memory ceilings are low enough that the job should stream pages to the destination rather than assemble a dataset in memory. For a large historical load, chunk the run and let successive runs resume from the stored position.

```mermaid title="Load a window of changes idempotently"
flowchart LR
	Start[Scheduled run] --> Lock{Acquire lock}
	Lock -- Held by an active run --> Skip[Log and exit]
	Lock -- Acquired --> Cursor[Open or resume a window]
	Cursor --> Query[Query the window since the cursor]
	Query --> Transform[Transform the page]
	Transform --> Deliver[Write rows with the deduplication key]
	Deliver --> Save[Save the page position]
	Save --> More{More pages}
	More -- Yes and time remaining --> Query
	More -- Yes and out of time --> Release[Release lock]
	More -- No --> Advance[Advance the cursor to the window end]
	Advance --> Release
```

#### Window on lastModifiedAt and page with an identifier cursor

Bound each run at both ends. Read the current time when the run starts, hold it as `:windowEnd` for the whole run, and query the half-open window between the last checkpoint and that value. Then page through the window on the resource identifier:

```http title="Query the first page of a window"
GET /{projectKey}/orders?withTotal=false&sort=id asc&limit=100&where=lastModifiedAt >= :cursor and lastModifiedAt < :windowEnd
```

```http title="Query each following page"
GET /{projectKey}/orders?withTotal=false&sort=id asc&limit=100&where=lastModifiedAt >= :cursor and lastModifiedAt < :windowEnd and id > :lastId
```

Pass the values as [input variables](/api/predicates/query.md#input-variables) with a `var.` prefix, such as `var.cursor=2026-09-01T00:00:00.000Z&var.windowEnd=2026-09-02T00:00:00.000Z`, rather than interpolating them into the predicate string. Repeat while the number of returned results equals the requested `limit`. The maximum `limit` is ****. The examples show the predicates unencoded for readability. URL-encode them in the request.

**The upper bound is what makes the window resumable.** Leave it out and the window keeps growing while the run pages through it. A resource whose identifier sorts before the page the run has already passed, and which is modified before the run finishes, still matches the open-ended predicate but is excluded by `id > :lastId`, so that run never exports it. Holding `:windowEnd` fixed puts the change outside the current window instead, where the next run collects it.

Offset pagination breaks on exports. The maximum offset is ****, so a window with more results than that leaves the export incomplete, and a resource modified mid-run shifts every later page. The identifier cursor is stable and resumable. For the full pattern, see [Iterate over all elements](/api/general-concepts.md#iterate-over-all-elements).

Place the most restrictive predicate first. For guidance on query cost, see [performance considerations](/api/predicates/query.md#performance-considerations).

#### Checkpoint the window

Keep two values, because they do different jobs.

Store `:windowEnd` and the identifier of the last row written in a [Custom Object](/api/projects/custom-objects.md) or in the destination itself, and save them after each page. A run that crashes or runs out of time then resumes inside the same window instead of restarting it.

Advance the cursor to `:windowEnd` only once the window is exhausted. Advancing it to the `lastModifiedAt` of the last row written moves it to a value chosen by identifier order rather than by time, which can step over changes the window never reached.

Start the next window slightly before the stored cursor rather than exactly on it. Deduplication on the destination absorbs the repeated rows, and the overlap covers resources whose modification time falls near a boundary.

The Connect documentation does not guarantee that scheduled runs cannot overlap, so take a durable lock with a time to live longer than the maximum run time. A crashed run then eventually releases the lock.

#### Do not poll the Messages API for history

Messages are not persisted for querying unless you [enable the feature](/api/projects/messages.md#enable-querying-messages-via-the-api) for the Project, and the default retention is 15 days even when it is enabled. See [Automatic cleanup](/api/general-concepts.md#automatic-cleanup). A backfill designed against the Messages API therefore returns nothing on most Projects, and at most 15 days of data on the rest. Window on the resources instead.

### Map resources onto the destination schema

The transform is where the value of the pipeline sits. Keep it a pure function with no network calls, so you can test it without a deployment.

Decide the grain first, because it determines everything else:

- **One row per change.** The destination holds an append-only record of what happened, and modeling happens downstream. This suits a warehouse and preserves history.
- **One current-state row per resource.** The destination holds the latest version of each resource, merged on its identifier. This suits a dashboard that only reports on what is true now, and it discards history.

Three shapes rarely map onto a flat table without a decision:

- **Localized strings.** A [LocalizedString](/search.md?urn=ctp:api:type:LocalizedString) holds one value per locale. Flatten the locales you report on into columns, or keep the whole object in a semi-structured column.
- **Money.** [CentPrecisionMoney](/search.md?urn=ctp:api:type:CentPrecisionMoney) carries `centAmount`, `currencyCode`, and `fractionDigits`. Carry all three. A `centAmount` alone is not comparable across currencies, and dividing it by 100 is wrong for currencies with a different exponent.
- **Nested collections.** Line Items, addresses, and Custom Fields are arrays. Either explode them into a child table keyed to the parent, or keep them as a semi-structured column and model them later.

Carry `resource.id`, the version or sequence value, and `lastModifiedAt` as columns on every row. Without them the destination cannot deduplicate, resolve a late arrival, or tell you when the copy was last correct.

### Deduplicate on the destination side

The same change reaches the destination more than once. Subscription delivery is [at least once with no ordering guarantee](/api/projects/subscriptions.md#delivery-guarantees), and a backfill window deliberately overlaps the stream. Duplicate rows are the expected outcome of a correct pipeline, not a delivery fault.

A Connect application cannot deduplicate on its own. Applications are [serverless and stateless](/connect/best-practices.md#when-to-use-an-alternative-to-connect) and are autoscaled, so an in-process record of what has been written is lost on restart and is not shared between instances. The destination has to absorb the repetition.

Because the streaming application retrieves the resource rather than reading the payload, both paths produce a resource with an `id`, a `version`, and a `lastModifiedAt` value. That gives you one merge key for both:

- **Append-only tables.** Make the primary key `id` plus `version`. A repeated write is then a no-op rather than a second row. For a raw event table fed only by the stream, `resource.id` plus `sequenceNumber` from the notification works as well, and `sequenceNumber` has the advantage of being sequential per resource.
- **Current-state tables.** Merge on `id` and keep the row with the highest `version`. A `version` value is not sequential, but it is comparable for the same resource, so a late arrival that carries a lower `version` is discarded instead of overwriting newer data.

Use the same key in both applications. If the stream and the backfill key their rows differently, every backfill run duplicates rows the stream already wrote.

### Propagate deletion and minimize personal data

Orders and Customers carry personal data, and an analytics destination is a common place for it to outlive its purpose. Treat the destination as a data processor and design for erasure before the first load:

- **Export only the fields the reports need.** A pipeline that copies whole Customer profiles by default takes on a data-protection obligation that the reports never required.
- **Subscribe to deletion.** [CustomerDeleted](/search.md?urn=ctp:api:type:CustomerDeletedMessage) is emitted when a Customer is deleted. Handle it by deleting or anonymizing the matching rows in the destination. A pipeline that only ever writes leaves orphaned personal data behind after an erasure request is satisfied in commercetools.
- **Account for deletion in the backfill too.** A query that windows on `lastModifiedAt` returns resources that still exist. A resource deleted since the last run does not appear in any result, so the backfill alone never removes anything. Deletion has to come from the stream, or from a periodic reconciliation of identifiers.
- **Keep personal data and credentials out of logs.** Log identifiers, decisions, and counts. Log a correlation key on every line, such as `resource.id` with `sequenceNumber`, so a run can be traced without recording its contents.

## Configure and deploy the Connector

Configure the applications, permissions, and Subscription as one contract.

### Grant only the scopes the flows use

Declare the scopes through `inheritAs.apiClient.scopes` so Connect generates a scoped API Client at installation, instead of asking an installer to supply credentials. The API Client that creates the Deployment needs `manage_connectors_deployments`, and it also needs `manage_api_clients` because the Connector generates its credentials automatically. See [Hosts and authorization](/connect/hosts-and-authorization.md#authorization) and [Modify a Connector](/connect/modify-connector.md).

An export reads from commercetools and writes to the destination, so the scopes are read scopes for the domains you export, plus `manage_subscriptions` for the registration script. Grant nothing more. See the canonical [API scopes](/api/scopes.md) reference for the full list.

```yaml title="Declare a least-privilege API Client"
inheritAs:
  apiClient:
    scopes:
      - manage_subscriptions
      - view_orders
      - view_customers
  configuration:
    standardConfiguration:
      - key: DESTINATION_TARGET
        description: Destination dataset, table, or source identifier
    securedConfiguration:
      - key: DESTINATION_CREDENTIALS
        description: Destination ingestion credential
```

`manage_subscriptions` grants permission to view, create, update, and delete Subscriptions, so the registration script needs no separate read scope. Confirm every scope you declare against the [API scopes](/api/scopes.md) reference: requesting a scope the API Client does not have returns a [400 `invalid_scope` error](/learning-developer-essentials/authentication-authorization/scopes.md) at runtime.

### Separate configuration from secrets

Classify destination settings according to the canonical [`connect.yaml` configuration rules](/connect/development.md#configure-connectyaml). Destination credentials belong in `securedConfiguration`. Never place them in code or logs.

Declare the streaming application as an `event` application and the backfill as a `job` application with a schedule:

```yaml title="Declare the two applications"
deployAs:
  - name: analytics-streamer
    applicationType: event
    endpoint: /analyticsStreamer
    scripts:
      postDeploy: npm ci --omit=dev && npm run connector:post-deploy
      preUndeploy: npm ci --omit=dev && npm run connector:pre-undeploy
  - name: analytics-backfill
    applicationType: job
    endpoint: /analyticsBackfill
    properties:
      schedule: "17 2 * * *"
```

Avoid scheduling the job on the hour. Scheduled jobs that start at the same moment across many Projects concentrate load and increase latency, so derive a fixed minute-and-second offset from your Project key and add a small random jitter to each run. See [Import, export, and sync integrations](/api/performance-tips.md#import--export--sync-integrations), which also documents the delta-plus-periodic-full-sync pattern this guide builds on.

Run the one-time historical load through the same job before you let the schedule take over, so the code path you depend on for recovery is the one you exercise routinely.

### Register the Subscription idempotently

Create the Subscription in `postDeploy` and remove it in `preUndeploy`. Declare both in the `scripts` block of `connect.yaml`, because [Connect runs only the scripts declared there](/connect/automation-scripts.md). A script file that exists in the repository but is not declared never runs.

Read the Subscription first and create it only when it is absent, updating it in place otherwise. A registration routine that deletes the Subscription and recreates it drops every notification emitted during the gap, and those changes reach the destination only if a later backfill happens to cover them.

Connect deploys to both Google Cloud and AWS Regions, and the message broker follows the Region. Read `CONNECT_SUBSCRIPTION_DESTINATION` to identify `SNS` or `GoogleCloudPubSub`, then use `CONNECT_AWS_TOPIC_ARN` or `CONNECT_GCP_TOPIC_NAME` for the active deployment Region. See [Automation scripts](/connect/automation-scripts.md) for the Event variables.

Creating, updating, or deleting a Subscription can take up to one minute to take effect. commercetools also sends a test notification to the Destination when a Subscription is created, so the handler must acknowledge a notification that carries no resource change.

## Verify the export

Add automated tests before you deploy. Mock both the commercetools API and the destination, and assert on what the code decided to write:

- The transform is a pure function that handles localized strings, money, and nested collections for a representative resource.
- Each subscribed Message type routes to the intended handler, and unhandled types are acknowledged without a destination call. A handler fed by a Change Subscription instead routes on the resource type, because the notification carries no Message type.
- The streaming application retrieves the resource by identifier and never reads the Message payload, including when the payload is omitted.
- Every row carries the deduplication key, and a redelivered notification produces one row rather than two.
- A transient destination failure returns a status outside the acknowledgment set, and a terminal failure is acknowledged and recorded for replay.
- The backfill pages with an identifier cursor, advances the checkpoint after each page, and resumes from the checkpoint on the next run.
- A backfill run over a window that the stream already covered adds no rows.
- A deletion removes or anonymizes the matching rows in the destination.
- `postDeploy` creates one Subscription, a repeated deployment does not duplicate it, and `preUndeploy` removes it.

After deployment, trace a real change end to end:

1. Confirm that one Subscription exists with the expected key, destination, and Message types.
2. Make a representative change, such as placing an Order or editing a Customer, and confirm that the row appears in the destination with the mapped fields correct.
3. Confirm that the row carries the deduplication key.
4. Redeliver the same notification and confirm that the destination still holds one row. To do this locally without a broker, post the encoded envelope to the application directly. See [Test an event application locally](/connect/test-applications-locally.md#test-an-event-application).
5. Run the backfill over a known window, then run it again over the same window, and confirm that the second run adds no rows and that the checkpoint advanced.
6. Confirm that the backfill reached the end of the window rather than stopping at the offset ceiling.
7. Delete a test Customer and confirm that the destination rows are removed or anonymized.
8. Make the destination fail temporarily and confirm that the work is retried rather than lost.

Some correct behavior looks like failure while you test. Confirm the deployment environment and the elapsed time before you treat any of these as a defect:

- Subscription delivery has [no guaranteed time frame](/api/projects/subscriptions.md#delivery-guarantees), and delays of several minutes happen. A row that has not appeared yet is not evidence of a dropped notification.
- A default `sandbox` deployment [scales to zero and needs time to boot again](/connect/deployment-behavior-and-environments.md#sandbox), which can exceed the event acknowledgment deadline. Early notifications are redelivered until the application is warm.
- Duplicate rows during a deliberate redelivery test are the expected input to deduplication. Judge the result after the merge, not before it.

In production, monitor Subscription health, acknowledgment failures, destination ingestion errors, backfill duration against the job timeout, checkpoint lag, and the row count each run writes. Define how an operator replays a window after a fix, and record it alongside the Connector.

## Troubleshoot observable symptoms

Match the symptom you observe to its likely cause and resolution:

| Symptom | Likely causes | Checks and resolution |
| --- | --- | --- |
| No rows arrive at all | The Subscription is missing. The `scripts` block was never declared in `connect.yaml`. Or `postDeploy` built a destination for the wrong broker and Subscription creation failed. | Query Subscriptions by key and inspect the health status. Compare the destination with `CONNECT_SUBSCRIPTION_DESTINATION`. Review [Deployment logs](/connect/deployment-logs.md). |
| Duplicate rows after a redelivery or a backfill run | The rows carry no deduplication key, or the stream and the backfill key them differently. | Assert the key on both paths and merge on it in the destination. |
| Most Orders arrive, but subscription and migrated Orders never do | The Subscription covers Order Created only. | Add Order Created From Recurring Order and Order Imported to the Subscription and the handler. |
| Rows are missing fields, and only the largest resources are affected | The handler transformed the Message payload, which is omitted above the queue size limit. | Retrieve the resource by `resource.id` and transform from current state. |
| Logs show success, but the destination has no rows | The handler acknowledged before the destination call, or swallowed the error and returned a success status. | Check destination-side ingestion, not only application logs. Let transient failures redeliver. |
| The same notification retries for hours and then disappears | The handler returns a status outside the acknowledgment set for a notification that can never succeed. | Acknowledge permanently unprocessable notifications, record them for replay, and alert instead of looping. |
| A backfill stops after a fixed number of rows | The job pages by `offset` and reached the maximum offset. | Page with an identifier cursor inside the `lastModifiedAt` window. |
| A backfill reloads everything on every run | The checkpoint is never saved, or it is saved only at the end of a run that times out. | Save the checkpoint after each page. |
| Two backfill runs process the same rows at once | Scheduled runs overlapped. | Take a durable lock with a time to live longer than the 30-minute job timeout. |
| A query against the Messages API returns nothing | Querying Messages is off by default, and retention is limited when it is on. | Window on the resources' `lastModifiedAt` instead. |
| Subscription registration fails on a new Deployment | The Project reached its Subscription limit. | Consolidate Message Subscriptions into a Change Subscription per resource, then route on the resource type because the notification carries no Message type. Or request a limit increase. |
| Change History queries return Too Many Requests, and API-origin changes are absent | The pipeline reads Change History. | Rebuild the feed on Subscriptions and API queries. Change History is a governance log. |
| Deleted Customers remain in the destination | Only the backfill runs, or the stream does not handle deletion. | Subscribe to Customer Deleted and delete or anonymize the matching rows. |
| Deployment succeeds but requests fail at runtime | The API Client scopes do not cover the exported domains, or a non-existent scope was declared. | Compare the exported domains against the declared scopes, and confirm each one exists in the [API scopes](/api/scopes.md) reference. |
| Reports disagree with commercetools on totals | Money was flattened to `centAmount` without `currencyCode` and `fractionDigits`. | Carry all three fields and convert in the destination. |
| The destination is empty during a rebuild | The full load clears the target before it writes. | Build into a new target and swap it in when the load completes. |

## Related pages

- [Area overview page with navigation](/guides.md)
- [Previous page: Integrate CPQ](/guides/cpq-integration.md)
- [Search documentation and API specs](/search.md)