Integrate an analytics destination

This guide covers the export pipeline that moves commerce data out of commercetools so that another system can report on it. It starts with deciding which kind of analytics you need, moves through choosing a destination and an integration path, and ends with the two applications that keep the destination data synchronized with your Project.

Data flows in one direction. commercetools is the source of truth, and the destination holds a copy shaped for querying. commercetools never reads back from the destination. This design runs the pipeline asynchronously outside the checkout path, so an outage produces stale reports rather than a broken checkout.

Before you start, you need the following:

• a commercetools Project with the data you want to export, and an API Client that can read it
• an account and ingestion credentials for the destination
• somewhere to run the integration, such as Connect, a cloud function, or your own infrastructure
• an agreed answer to each question in Identify requirements that shape the pipeline

Define analytics export requirements

The destination, the integration path, and the pipeline design depend on these decisions. Record them before you configure or build anything.

Confirm you need a commerce data export

Three unrelated things are called analytics, and they route to different products. Confirm which one your requirement is before you read further, because only the first is an export pipeline.

The Change History boundary and the client-side tracking boundary are worth stating plainly, because both mistakes are expensive to reverse.

Client-side behavioral tracking is a storefront concern. Page views, clicks, and consent-managed pixels belong in your storefront with a tag manager, not in this pipeline. An export can feed the server-side ingestion endpoint of the same vendor, but it cannot produce browser events.

Identify requirements that shape the pipeline

Answer and record the following, because each answer changes the design.

Capture anything that does not fit these rows as its own requirement rather than forcing it into one. Each of the following can move the decision in the next section:

• consolidating several Projects or Regions into one destination
• data residency rules that restrict where the destination and the runtime may sit
• an existing extract, transform, and load tool or customer data platform that can already read the commercetools API
• a retention or erasure policy that the destination must honor
• near-real-time personalization built on the same feed, which raises the latency requirement

Unless a requirement says otherwise, the design that follows is the default: a streaming application for freshness, a scheduled job for history and gap repair, read-only scopes, destination credentials in secured configuration, and deduplication on the destination side.

Choose the destination category

The category decides the mechanism. Vendor documentation is authoritative for ingestion endpoints, field names, and limits.

Two of these rows carry a decision that is easy to get wrong.

Business intelligence tools do not integrate with commercetools directly. Point the pipeline at a warehouse and let the reporting tool read modeled tables from there. Building an extract for a reporting tool alone produces a pipeline with no history and no place to model.

Product analytics tools that are designed for browser instrumentation, such as GA4 and Mixpanel, expose a narrower server-side ingestion API than their browser SDK. Confirm what each one accepts before you treat it as a substitute for a warehouse. If the requirement is complete history and modeling, choose a warehouse and treat the analytics tool as an additional destination rather than the only one.

Choose an integration path

Work through these options in order and stop at the first one that meets your requirements.

Check for an existing analytics Connector

GET https://connect.{region}.commercetools.com/connectors/search?integrationTypes=analytics&private=false

Record the Connector key and version you evaluated, or record that none exists. Plan for a custom build until the search identifies an option that meets your requirements, and record that assumption in the effort estimate.

Two adjacent options exist and neither is a Connect deployment:

• A customer data platform may ship its own commercetools source, configured in that vendor's console. If it covers your domains, use it. Nothing in this guide's build and deploy sections applies.
• An extract, transform, and load tool that already reads the commercetools API may be the cheapest path if your organization runs one. It is a valid alternative, and it is not built or deployed through Connect.

Prove that a gap is not configuration

When a Connector matches your destination but appears to be missing a behavior, confirm that configuration cannot close the gap before you consider forking it. Which Messages flow, how fields map to destination columns, and which table or dataset receives them are usually configuration.

Fork or build

If a Connector matches your destination and has a genuine gap that configuration cannot close, and its source is available, fork it. Add only the difference and deploy the result as an Organization Connector.

commercetools connect init my-analytics-connector --template product-ingestion

Whichever option you choose, the pipeline design in the next section is the same.

Design the export pipeline

An analytics export is made of two applications that write to the same destination. The streaming application keeps the destination fresh. The scheduled job loads history and repairs whatever the stream missed. Both produce the same rows, so both must be safe to run more than once.

Stream changes as they happen

• Retrieve the Order instead of reading the Message payload, which covers at-least-once delivery, the absence of an ordering guarantee, and the omitted payload that arrives as PayloadNotIncluded when a Message exceeds the queue size limit or its format is not accepted by the messaging service.
• Acknowledge deliberately, which covers the status codes the message queue treats as an acknowledgment, the seven-day retention of unacknowledged messages, and the outcome-to-status-code mapping.

Three decisions are specific to an export.

Backfill and repair on a schedule

The scheduled job loads history and repairs what the stream missed. Query resources to load data that predates the Subscription because Message history has limited retention and is not enabled by default.

Map resources onto the destination schema

The transform is where the value of the pipeline sits. Keep it a pure function with no network calls, so you can test it without a deployment.

Decide the grain first, because it determines everything else:

• One row per change. The destination holds an append-only record of what happened, and modeling happens downstream. This suits a warehouse and preserves history.
• One current-state row per resource. The destination holds the latest version of each resource, merged on its identifier. This suits a dashboard that only reports on what is true now, and it discards history.

Three shapes rarely map onto a flat table without a decision:

• Localized strings. A LocalizedString holds one value per locale. Flatten the locales you report on into columns, or keep the whole object in a semi-structured column.
• Money. CentPrecisionMoney carries centAmount, currencyCode, and fractionDigits. Carry all three. A centAmount alone is not comparable across currencies, and dividing it by 100 is wrong for currencies with a different exponent.
• Nested collections. Line Items, addresses, and Custom Fields are arrays. Either explode them into a child table keyed to the parent, or keep them as a semi-structured column and model them later.

Deduplicate on the destination side

• Append-only tables. Make the primary key id plus version. A repeated write is then a no-op rather than a second row. For a raw event table fed only by the stream, resource.id plus sequenceNumber from the notification works as well, and sequenceNumber has the advantage of being sequential per resource.
• Current-state tables. Merge on id and keep the row with the highest version. A version value is not sequential, but it is comparable for the same resource, so a late arrival that carries a lower version is discarded instead of overwriting newer data.

Use the same key in both applications. If the stream and the backfill key their rows differently, every backfill run duplicates rows the stream already wrote.

Propagate deletion and minimize personal data

Orders and Customers carry personal data, and an analytics destination is a common place for it to outlive its purpose. Treat the destination as a data processor and design for erasure before the first load:

• Export only the fields the reports need. A pipeline that copies whole Customer profiles by default takes on a data-protection obligation that the reports never required.
• Subscribe to deletion. CustomerDeleted is emitted when a Customer is deleted. Handle it by deleting or anonymizing the matching rows in the destination. A pipeline that only ever writes leaves orphaned personal data behind after an erasure request is satisfied in commercetools.
• Account for deletion in the backfill too. A query that windows on lastModifiedAt returns resources that still exist. A resource deleted since the last run does not appear in any result, so the backfill alone never removes anything. Deletion has to come from the stream, or from a periodic reconciliation of identifiers.
• Keep personal data and credentials out of logs. Log identifiers, decisions, and counts. Log a correlation key on every line, such as resource.id with sequenceNumber, so a run can be traced without recording its contents.

Configure and deploy the Connector

Configure the applications, permissions, and Subscription as one contract.

Grant only the scopes the flows use

inheritAs:
  apiClient:
    scopes:
      - manage_subscriptions
      - view_orders
      - view_customers
  configuration:
    standardConfiguration:
      - key: DESTINATION_TARGET
        description: Destination dataset, table, or source identifier
    securedConfiguration:
      - key: DESTINATION_CREDENTIALS
        description: Destination ingestion credential

Separate configuration from secrets

deployAs:
  - name: analytics-streamer
    applicationType: event
    endpoint: /analyticsStreamer
    scripts:
      postDeploy: npm ci --omit=dev && npm run connector:post-deploy
      preUndeploy: npm ci --omit=dev && npm run connector:pre-undeploy
  - name: analytics-backfill
    applicationType: job
    endpoint: /analyticsBackfill
    properties:
      schedule: "17 2 * * *"

Run the one-time historical load through the same job before you let the schedule take over, so the code path you depend on for recovery is the one you exercise routinely.

Register the Subscription idempotently

Read the Subscription first and create it only when it is absent, updating it in place otherwise. A registration routine that deletes the Subscription and recreates it drops every notification emitted during the gap, and those changes reach the destination only if a later backfill happens to cover them.

Creating, updating, or deleting a Subscription can take up to one minute to take effect. commercetools also sends a test notification to the Destination when a Subscription is created, so the handler must acknowledge a notification that carries no resource change.

Verify the export

Add automated tests before you deploy. Mock both the commercetools API and the destination, and assert on what the code decided to write:

• The transform is a pure function that handles localized strings, money, and nested collections for a representative resource.
• Each subscribed Message type routes to the intended handler, and unhandled types are acknowledged without a destination call. A handler fed by a Change Subscription instead routes on the resource type, because the notification carries no Message type.
• The streaming application retrieves the resource by identifier and never reads the Message payload, including when the payload is omitted.
• Every row carries the deduplication key, and a redelivered notification produces one row rather than two.
• A transient destination failure returns a status outside the acknowledgment set, and a terminal failure is acknowledged and recorded for replay.
• The backfill pages with an identifier cursor, advances the checkpoint after each page, and resumes from the checkpoint on the next run.
• A backfill run over a window that the stream already covered adds no rows.
• A deletion removes or anonymizes the matching rows in the destination.
• postDeploy creates one Subscription, a repeated deployment does not duplicate it, and preUndeploy removes it.

After deployment, trace a real change end to end:

1. Confirm that one Subscription exists with the expected key, destination, and Message types.
2. Make a representative change, such as placing an Order or editing a Customer, and confirm that the row appears in the destination with the mapped fields correct.
3. Confirm that the row carries the deduplication key.
4. Redeliver the same notification and confirm that the destination still holds one row. To do this locally without a broker, post the encoded envelope to the application directly. See Test an event application locally.
5. Run the backfill over a known window, then run it again over the same window, and confirm that the second run adds no rows and that the checkpoint advanced.
6. Confirm that the backfill reached the end of the window rather than stopping at the offset ceiling.
7. Delete a test Customer and confirm that the destination rows are removed or anonymized.
8. Make the destination fail temporarily and confirm that the work is retried rather than lost.

Some correct behavior looks like failure while you test. Confirm the deployment environment and the elapsed time before you treat any of these as a defect:

• Subscription delivery has no guaranteed time frame, and delays of several minutes happen. A row that has not appeared yet is not evidence of a dropped notification.
• A default sandbox deployment scales to zero and needs time to boot again, which can exceed the event acknowledgment deadline. Early notifications are redelivered until the application is warm.
• Duplicate rows during a deliberate redelivery test are the expected input to deduplication. Judge the result after the merge, not before it.

In production, monitor Subscription health, acknowledgment failures, destination ingestion errors, backfill duration against the job timeout, checkpoint lag, and the row count each run writes. Define how an operator replays a window after a fix, and record it alongside the Connector.

Troubleshoot observable symptoms

Match the symptom you observe to its likely cause and resolution: