commercetools catalog migration

The canonical catalog feed

The pipeline's source boundary. Adapters write these shapes; the pipeline reads nothing else. No source system is named anywhere in the contract, by design.

• Shape
• Variant axes
• Two rules that are load-bearing — axis codes vs labels, money as a decimal string
• Record types — channel / customerGroup, productSelection / store, category, attributeDefinition, product, variant, asset
• Resolve in the adapter what commercetools cannot express
• Choosing a code — externalId only maps on a category
• Validation

Shape

{"_type":"category","code":"tops","name":{"en-GB":"Tops"},"parent":"apparel","sourceOrder":1}
{"_type":"attributeDefinition","name":"colour","type":"lenum","level":"variant","axis":true,
 "values":[{"key":"BLK","label":{"en-GB":"Black"}}]}
{"_type":"product","code":"TEE-01","name":{"en-GB":"Cotton Tee"},"categories":["tops"],
 "axes":["colour","size"],"attributes":{"material":{"en-GB":"Cotton"}}}
{"_type":"variant","sku":"TEE-01-BLK-M","product":"TEE-01",
 "axisValues":{"colour":"BLK","size":"M"},
 "prices":[{"currency":"GBP","amount":"19.99","country":"GB"}]}

Variant axes

Three records share the work:

• an attributeDefinition with "axis": true declares that an attribute may carry variant identity — only legal at "level": "variant";
• a product lists the axes it varies by in axes, in order; absent or empty means a single-variant product;
• a variant gives its own coordinate in axisValues, with display text kept separately in axisLabels.

An axis is not merely an attribute whose value happens to differ between variants. Four things follow from marking one:

• The constraint is a one-way door. CombinationUnique can be relaxed to None later but never re-tightened or switched, because changeAttributeConstraint accepts only None. Getting it wrong means recreating the attribute and rewriting its data, which is why derive records every such choice as irreversible.
• Values must be language-independent codes, never localized text — the next section is that rule.
• Coverage and uniqueness are enforced. Every variant must carry a value for every axis its product declares, and no two variants of one product may share a combination.
• An attribute cannot be an axis in one product and a plain attribute in another product of the same ProductType. One definition carries one constraint, so the ProductType has to be split — see product-model.md.

Two rules that are load-bearing

These are enforced by the schema rather than left to discipline, because both failure modes are silent.

Axis codes and axis labels are separate fields

// Correct — identity is a code, display text is separate
{"axisValues":{"colour":"BLK"},"axisLabels":{"colour":{"en-GB":"Black","de-DE":"Schwarz"}}}

// Wrong, and the schema rejects an ltext axis: the same SKU would key
// differently per language and break when a second locale is added
{"axisValues":{"colour":"Black"}}

Money is a decimal string, never a JSON number

{"currency":"GBP","amount":"19.99"}   // correct
{"currency":"GBP","amount":19.99}     // rejected by the schema

Record types

channel and customerGroup — prerequisites, not imports

• it reads the project for the declared keys;
• absent ones are created through the platform API;
• present ones are left exactly as they are — never patched;
• a present channel lacking a role the plan needs is an error, reported and not modified, because roles govern stores and inventory too.

Three consequences follow:

• The code is the key, used verbatim — never prefixed. A price references the project's own channel key, and channels are often created by store setup long before a catalog migration runs. These two are the only deliberate exception to <prefix>-<sourceCode>. ProductTypes used to be a second, accidental one — keyed verbatim from the feed's productType code — and are now prefixed like everything else.
• A teardown scoped to keys.prefix will not remove them. If the load created one, it has to be removed by hand.
• Creating them needs scopes the import does not. manage_products covers channels; customer groups need view_customer_groups and manage_customer_groups, which manage_products does not grant.

productSelection and store — assortments, and where they apply

How prices, channels and stores actually relate

Price ──carries──> Channel (role: ProductDistribution)
                      ^
Store ──lists─────────┘  distributionChannels[]

Three consequences worth planning around:

• The channel is the pricing unit; the store is only the binding. Giving three stores three different prices means three channels. A store with no distribution channels sees only channel-less prices.
• Equal specificity is ambiguous, and the platform will not resolve it. If a store lists two distribution channels and a variant is priced in both, both match at the same fallback step, and the storefront has to disambiguate with setLineItemDistributionChannel. Modelling "one channel per price list and one per store" is legal and needs middleware — know that before choosing it.
• Cheapest wins within a step. A stray channel price silently undercuts the intended one rather than erroring.

Product selections decide which products exist

{"_type":"product","code":"TEE","selections":[{"code":"uk-assortment"}]}
{"_type":"product","code":"DRESS","selections":[{"code":"uk-assortment","includeSkus":["D-S","D-M"]}]}

Two things follow from the Import API's shape, and they are the reason the feed is authored this way round:

• Assignments cannot be split. The API replaces omitted fields, so all of a selection's assignments travel in one resource. plan inverts the product-side authoring and assembles the complete list per selection; there is no batching to fall back on, and an assortment of twenty thousand products is one large request. plan flags anything above a thousand for review.
• One line per record stays true. A selection carrying its assignments in the feed would be a single twenty-thousand-element line.

Stores decide where it all applies

Loading, and why the order inverts

category

attributeDefinition

Emit these when the source declares a type system. When the feed carries none, the pipeline infers definitions from observed values and forces an explicit review — inference is a fallback, not the happy path.

product

variant

Relative image URLs

asset

Each source:

This is worth stating because the opposite assumption — asset keys unique project-wide — makes the natural mapping look illegal and pushes an author into inventing per-SKU codes to satisfy a rule that does not exist. An over-strict reading here does not merely add noise; it changes the shape of the data.

Resolve in the adapter what commercetools cannot express

Choosing a code

The code becomes part of a key, so:

• Prefer a stable machine identifier over anything a merchandiser can edit.
• Never use a localized or display value.
• Preserve the raw source identifier in externalId as well — but read the next section first, because only a category externalId survives.

externalId only maps on a category

Validation

Integrity covers: duplicate codes and SKUs, missing category parents, category cycles, orphan variants, products without variants, missing category references, axis coverage and uniqueness, localized axes, attributes written but not declared, level mismatches, and multiple master-variant claims.

Every diagnostic names a file and a line. Fix the adapter, not the feed.

Categories, slugs and order hints

The tree

Slugs are unique across the Project per locale

Resolution appends the source code, which keeps the result traceable to the record that produced it:

mens-acc    →  accessories
womens-acc  →  accessories-womens-acc

This changes the URL the migration produces, so it is recorded as information loss. Generate the slug report from the plan, not from the loaded project: the old URL structure only exists before cutover, and the SEO owner needs to see the changes forced by project-wide uniqueness while there is still time to argue.

Product slugs follow the same uniqueness rule and the same pattern.

Order hints

The encoding used here pads to a fixed width and appends a non-zero terminal digit. Fixed width makes the values sort in the same order as the positions they encode; the terminal digit keeps them legal. Because every value has the same width, appending it cannot change the relative order.

position 1 of 3    →  0.11
position 2 of 3    →  0.21
position 1 of 12   →  0.011
position 10 of 12  →  0.101

If the source has no ordering at all, siblings are ordered by code and a decision is recorded saying so — navigation order is a merchandising decision, not a migration one.

Modelling questions worth raising

Other fields

• externalId preserves the source identifier alongside the key, so downstream systems can still join on it. Populated from the feed's externalId, falling back to the source code.
• description passes through when supplied.
• Meta title, description and keywords are not mapped today — they are SEO content, usually owned elsewhere, and inventing them from a name is worse than leaving them empty.

The engagement decision log

Two logs, and why they cannot be one

Mixing them fails both ways: a regenerated file cannot hold history, and an accumulated file cannot be trusted to describe the current plan.

Where it lives — and the layout it belongs to

<engagement-root>/
  .gitignore                    must ignore the pipeline checkout — see below
  ct-catalog-migration-pipeline/  the tool, cloned and gitignored. Never write
                                  engagement files here, and never edit it — local
                                  changes are lost on the next pull and are
                                  invisible to everyone else.
  source-export/                what the customer handed over
  migration/                    the engagement
    migration.config.json         written from the step-0 interview
    DECISIONS.md                  this log — append-only, committed
    MAPPING-PROPOSAL.md           the step-0b artefact
    adapter/                      the only code written per engagement
      package.json                  {"type":"module"} — see below
    feed/                         generated NDJSON; regenerated, disposable
    out/                          pipeline artefacts; regenerated, gitignored

{ "type": "module", "private": true }

/ct-catalog-migration-pipeline/
feed/
out/

cd ct-catalog-migration-pipeline
npm run pipeline -- plan --config ../migration/migration.config.json

Format

Numbered, so entries can reference each other and be superseded rather than edited:

# Migration decision log — acme-eu

Append-only. Newest last. Never regenerate; supersede instead.

## D001 — Catalog model: Classic
2026-09-22 · step 0a · decided by Priya (commerce lead)

The largest product has 14 variants, well inside Classic's 100.

Rejected: Modular. It would force standalone pricing and cannot import
`defaultVariant`, and nothing about this catalog needs the higher ceiling.

Status: accepted

## D002 — Colour is an lenum, not free text
2026-09-22 · step 0b · proposed from sample, confirmed by Priya

The sample has 11 distinct colour codes with German and English labels, so
`derive` proposed `lenum`. See `out/probe/MODEL-REVIEW.md`.

**Irreversible.** `changeAttributeConstraint` accepts only `None`, so if the
full export turns out to hold hundreds of colours this becomes an attribute
that has to be recreated and its data rewritten.

Status: accepted, pending confirmation against the full export

What to log, per step

Step 0 — the interview

Step 0b — discovery

Steps 1–2 — implementing the adapter

• a code derived from a display name, and the derivation rule
• a hierarchy level collapsed, and what happened to the intermediate
• an inherited price resolved, and from where
• a locale tag normalised (en_GB → en-GB)
• a sentinel or placeholder value skipped, passed through, or interpreted
• a field dropped, and why
• every silent cap — anything sampled, truncated or skipped

The adapter reference says a derived code "is a decision, not a detail" and to "report every silent cap". This is where those reports go.

validate and audit

preflight — and this one is an obligation

load and verify

What not to log

• Pipeline mapping decisions, copied. Reference the artefact.
• Fixes. A defect found and corrected leaves no decision behind.
• Anything the config already states. Log the why, not the value.
• Conversation. An entry is a decision with an owner and a rationale, not a transcript.

The discipline that makes it worth keeping

Write the entry when the decision is made, not at the end. A log reconstructed after the fact records what someone remembers deciding, which is reliably the subset that turned out well — and the entries that matter are the irreversible ones taken under uncertainty, which are the first to be forgotten.

Money and prices

The two things most likely to be wrong in a way that looks right.

Minor units

Convert on the digit string, never through a float

Amounts travel through the feed as decimal strings and are converted by string manipulation:

"19.99"  GBP (2)  →  1999
"4500"   JPY (0)  →  4500        not 450000
"12.345" BHD (3)  →  12345
"0.07"   EUR (2)  →  7

"29.999"  GBP (2)  →  error: 3 decimal places but GBP allows 2
"4500.5"  JPY (0)  →  error

Price scope

Adding an embedded price is rejected when:

• another embedded price has the same scope and the same validFrom / validUntil, or
• two prices have overlapping validity periods within the same scope.

Validity intervals are half-open, so one window ending exactly where the next begins does not overlap.

The audit gate checks all of this offline. Resolving inherited prices produces duplicate-scope shapes very easily, which is why it is worth catching before the API names one offending record and stops.

Field name

Embedded versus standalone

Mixing both types on one product is possible but degrades performance, so the pipeline takes a single choice.

Three things about standalone pricing that only bite at load time:

• It needs manage_standalone_prices. manage_products does not grant it, unlike every other import request this pipeline sends. A client set up for an embedded load fails on that one stage with a 403 and nothing else.
• sku is a plain string, not a KeyReference, and the API does not validate it. A price for a SKU no variant has is created successfully and prices nothing. The audit gate checks every SKU against the plan's own variants, because that is the only place it can be caught.
• priceMode on the product decides which prices are read. A product declaring Embedded whose prices are standalone resources imports with every operation reporting imported, and then shows no price anywhere. The gate treats any disagreement between the config, the products and where the prices actually are as an error.

Uniqueness differs between the two, and so does what a duplicate means:

Because standalone overlaps are accepted, the gate reports them as a warning rather than an error: the API will take them, but which price wins stops being something the migration decides. An open-ended base price alongside a dated promotion is not an overlap in either mode — that pair is the normal shape of a priced catalog.

Keep the embedded price count low by leaning on price-selection fallback: one price for a currency with no country beats one price per country, and a channel-less base price beats a price per channel.

Things with no embedded-price equivalent

Report these as information loss rather than approximating them:

• Quantity breaks / tiered pricing. Native tiers exist on Standalone Prices; an embedded price has no tiered equivalent. A quantity break usually becomes a Cart Discount, which is a modelling decision, not a translation.
• An unmappable customer-group price. Loading a trade or staff price unscoped shows it to every shopper and collides with the base price as a duplicate scope. Skip and report. Skipping is recoverable; wrong data in production is not.
• Net versus gross. Whether a price includes tax is usually a setting elsewhere in the source, not a property of the price row, so a price read on its own does not tell you what its number means. Getting it backwards changes every price on the site by the tax rate and nothing in the data contradicts you. It maps to a tax mode, which is a modelling decision.
• A tax code is not a tax rate. A product tax code is a reference into an external tax engine. Carry it across verbatim; never derive a rate from it.

Currency and locale acceptance

The product model

How ProductTypes and their attributes are derived, and which of those choices cannot be undone.

Declared or inferred

What inference can and cannot recover, on the same catalog:

Attribute constraints are irreversible

Product level versus variant level

• Product — a native product-level attribute. Supported by Product Search but not by Product Projection Search.
• Variant + SameForAll — works with both search APIs.

An attribute name is a Project-wide declaration

Two invariants hang on this, and neither is scoped to a ProductType.

A load declared material as text. An unrelated existing ProductType had it as ltext. Both operations were rejected.

There is no update action that changes an attribute's type. On a ProductType that already exists, the fix is to remove and re-add the attribute, which deletes its values on every product using it. Treat it as a migration decision.

Grouping products into ProductTypes

One ProductType per product is a modelling mistake — it defeats the point of a shared blueprint and burns through the project limit of 1000 ProductTypes — so the grouping never invents keys.

Three clashes are fatal, because a ProductType cannot express them:

• An attribute that is an axis for some products in the group and a plain attribute for others. Constraints belong to the ProductType, not the product. Split the ProductType, or make the axis consistent.
• An attribute at product level on some records and variant level on others. One attribute cannot be both SameForAll and free-varying.
• An attribute holding more than one kind of value. Declare it explicitly or fix the adapter.

Choosing types

Review the source model; do not just translate it

The most valuable output of a migration is often a defect found in the source. Worth raising rather than silently carrying across:

• A variant axis with no stable code, only a localized name.
• Categories that are really facets.
• Attributes that encode business rules.
• Attributes declared but never populated — usually a field the adapter dropped, and reported as such.
• A 300-attribute ProductType serving eight facets.

Output

• out/product-types.json — the drafts, for the load stage and for diffing.
• out/decisions.json — the machine-readable decision log.
• out/MODEL-REVIEW.md — the sign-off artefact, grouped into irreversible choices, information loss, and guesses. Nobody reviews a JSON file.

Every decision carries a rationale. The rationale is not a comment; it is the deliverable. A mapping added without one is an incomplete change, because the next reader cannot tell whether the choice was considered or accidental.

Running the pipeline

• Getting the pipeline
• The stages
• What is built
• validate
• derive
• plan
• audit — including the freshness digest
• preflight — unprefixed ProductType keys, the project-wide attribute constraint
• load — platform stages, prerequisites, catalog size ceiling, batching, failure states, SDKs, credentials and scopes
• verify
• Configuration
• What shapes the load
• Fixtures

Getting the pipeline

warning: adding embedded git repository: ct-catalog-migration-pipeline
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.

So the order matters:

# from the engagement root, BEFORE cloning
echo '/ct-catalog-migration-pipeline/' >> .gitignore

git clone https://github.com/commercetools/ct-catalog-migration-pipeline
cd ct-catalog-migration-pipeline
npm install            # first time; Node 20+
npm test               # no network, no credentials — a few seconds

git rm -r --cached -f ct-catalog-migration-pipeline

Run the tests even when cloning a known-good commit. "The four stages ran clean on my feed" is the tempting substitute and a much weaker signal: a pipeline that works on one feed says nothing about whether the checkout is intact. A few seconds against a stage misbehaving later for a reason you cannot localise.

One line is enough:

Pipeline: ct-catalog-migration-pipeline @ <short sha> (git rev-parse --short HEAD)

The stages

npm run pipeline -- validate --config ../migration/migration.config.json
npm run pipeline -- derive   --config ../migration/migration.config.json
npm run pipeline -- plan     --config ../migration/migration.config.json --payloads
npm run pipeline -- audit    --config ../migration/migration.config.json

Every stage exits non-zero when it finds an error, so the sequence composes in a shell or CI.

What is built

validate

Schema conformance, then catalog-wide integrity — and integrity only runs once every record parses, because a rejected record drops a product from the index and makes the integrity pass invent orphans that are really just consequences of the earlier failure.

Catches: malformed JSON, schema violations, unknown record types, duplicate codes and SKUs, missing category parents, category cycles, orphan variants, products without variants, missing category references, axis coverage and uniqueness, localized axes, axes declared at product level, attributes written but not declared, level mismatches, multiple master-variant claims.

• A price scoped to an undeclared channel or customer group. An error: load creates the ones the feed declares, so an undeclared reference is one nothing will create, and it expires unresolved after 48 hours taking the price with it. A price channel also needs the ProductDistribution role — an error under standalone pricing, which the API rejects outright, a warning under embedded.
• A dangling or self-defeating assortment. A product assigned to an undeclared selection (the assignment is silently dropped — nothing fails, the assortment is just wrong); a store listing an undeclared channel, or one whose roles do not match the list it is in; an empty Individual selection, which offers nothing; and a store whose selections are all inactive with at least one Individual, which offers no products at all where an empty list would have offered everything.
• Relative image or asset-source URLs with no media.baseUrl. An error, not a warning: the host is not in the export, a wrong one 404s every image undetectably, and the contract previously forced authors to invent one by rejecting relative paths outright.
• A price in a currency market.requiredCurrencies does not list, reported once per currency with the first price that uses it. Left to plan it would surface once per affected variant, two stages later.
• Which catalog model the catalog requires. Classic caps at 100 variants per product, Modular at 10,000, and that is purely a function of variant counts — so the feed can settle it with no credentials and no plan. The report states the largest product and the model it needs even when Classic fits, because "Classic is enough" is the answer someone is looking for.

Diagnostics name a file and a line. Fix the adapter, not the feed.

derive

plan

Two structural points worth knowing when reading a plan:

• Product-level attributes appear on every variant, because SameForAll enforces that variants agree rather than distributing a value.
• Axis values appear as ordinary attribute values holding the enum key. axisLabels never reach a variant.

Output:

Plans are deterministic: the same feed produces a byte-identical plan, which is what makes a diff between runs meaningful.

audit

That independence is the point. A verifier sharing logic with the writer confirms the writer's bugs rather than finding them, and reading the file means the gate checks what will actually be sent. It also keeps working on a hand-edited or replayed plan.

Reading from disk has one cost, and it is checked

Cascades are suppressed: a product whose ProductType does not exist is not also reported for every missing required attribute.

preflight

A handful of GETs that answer one question: will this project accept this plan? Four always — project settings, and counts of product types, categories and products — plus one each for channels and customer groups when the feed declares any, and a paginated read of every ProductType with its attributes. Every failure it catches would otherwise surface as a wall of rejected Import Operations, hours into a load, one error per record.

cp .env.example .env
npm run pipeline -- preflight --config ../migration/migration.config.json
npm run pipeline -- preflight --config ../migration/migration.config.json --apply

It prints the project it is about to write to first, with existing resource counts, because loading into the wrong project is a real hazard.

A project loaded before ProductType keys were prefixed

On a throwaway project, delete the products and re-load. On one holding data worth keeping, this is a migration decision: log it and decide deliberately.

The one project-wide constraint only preflight can see

• A differing type is an error. There is no update action that changes an attribute's type: on an existing ProductType it would have to be removed and re-added, deleting the values on every product using it. Preflight says so explicitly when the clash is on a ProductType the plan itself loads, because the remedy is different — a migration decision, not an edit.
• Enum values may differ freely. The documentation's own example gives Color a different value set on Jeans than on T-Shirt. Only the type is constrained, so preflight compares type names — and, for a set, its element type; for a reference, its target.
• A matching type with a differing isSearchable is a warning. The load succeeds, and the attribute then becomes unavailable for search, filters and facets across every ProductType — including ones working today. Nothing errors; the facet simply stops existing.

load

npm run pipeline -- load --config ../migration/migration.config.json              # dry run
npm run pipeline -- load --config ../migration/migration.config.json --execute
npm run pipeline -- load --config ../migration/migration.config.json --execute --wait

Two platform stages, at opposite ends of the run

Three stages do not go through the Import API at all, and they do not all run at the same time:

Prerequisites run first, and not through the Import API

The rule is read first, then create only what is missing:

• an absent channel or group is created;
• an existing one is left untouched — not patched to match the plan;
• an existing channel lacking a role the plan needs is an error: roles govern stores and inventory as well as prices, so widening them is a project decision, not something a catalog load does in passing;
• a failed read creates nothing, because creating without knowing what exists is how duplicates happen.

They are reported separately from the containers, because they have no container, no operation states and no 48-hour window — the call either succeeded or it did not.

How large a catalog this handles

The feed held in memory is the lesser cost: ~14 KB of RSS per variant, so a 125,000-variant run peaks near 1.8 GB against a 4.3 GB default heap.

Batching

• At most 20 resources per Import Request, clamped in code.
• Containers named by resource type (mig-category, mig-product-draft) and reused across runs; a run-scoped name would exhaust the 1000-container limit and hide the previous run's work. Each is restricted to its resourceType, so a mis-routed batch is rejected.
• A stage beyond the per-container operation limit splits into numbered containers, and no request straddles two.

Do not serialise on polling

Which states are failures

The official SDKs

The pipeline depends on three first-party packages, and the split is worth knowing when reading the code:

• @commercetools/importapi-sdk supplies every draft type in the plan. These are generated from the API's own specification, so a field that moves becomes a compile error rather than a rejected operation. Do not hand-write import resource types alongside them.
• @commercetools/platform-sdk supplies the Project type and preflight's request builders.
• @commercetools/ts-client owns transport: the client-credentials flow, token lifecycle, retry with backoff, a concurrency queue, and correlation IDs.

Three shapes are easy to get wrong by hand and impossible to get wrong with the generated types:

Credentials

• The file and the environment naming different projects is refused. The environment would win, so --env would silently not do what it looks like.
• No file at all, while the environment names a project, is also refused. That is the case a disagreement check cannot catch — there is nothing to disagree with — and it is how a run in a directory with no credentials of its own borrows a project nobody chose for it. The only thing that catches it otherwise is preflight printing the project name, which is luck rather than a check.
  Set CTP_AMBIENT_OK=1 when the ambient environment is deliberate. CI and containers should; a working directory generally should not.

verify

The last stage, and the only credentialed one that cannot change anything — every call is a GET.

npm run pipeline -- verify --config ../migration/migration.config.json

• A stale failure that is already fixed still shows, because the old operation has not aged out yet. Reads as worse than reality.
• A failure that has aged out is simply gone. Query a container more than 48 hours after the run and every surviving operation may well say imported — not because nothing failed, but because the ones that failed were deleted. Reads as better than reality, and it is indistinguishable from a clean run.

• missing — planned and absent. The load did not land, whatever it said.
• mismatched — present but not what the plan says.
• unexpected — a variant or attribute the plan never mentioned, which is how a stale earlier run becomes visible.

Two things it gets right that are easy to get wrong:

• It compares staged data, not current. The pipeline imports unpublished on purpose, so current is empty on a first load.
• It normalises read shapes first. An enum is written as a key and read back as {key, label}; references are written as keys and read back as ids. Enum values are normalised, and reference attributes are skipped rather than guessed at — a verifier that flags every enum in a catalog is one that gets switched off.

It reads stores and product selections back too, but only when the plan declares them — most engagements have neither, and an unconditional read costs two requests and two scopes for nothing. Two findings there are worth knowing in advance:

• selection-mode-differs cannot be fixed by re-running. With no changeMode action, the selection has to be deleted and recreated, and every store referencing it re-wired. Until then the assortment means the opposite of the plan.
• Store drift is reported per list, because the API replaces rather than merges each one. Supply-channel drift is only a warning: this pipeline imports no inventory, so that wiring affects nothing it loaded.

Configuration

The field-by-field shape:

What shapes the load

• 20 resources per Import Request; fewer than 200,000 Import Operations per container; max 1000 containers per project (soft).
• A container with no retentionPolicy is deleted 72 hours after creation. Set a TimeToLiveRetentionPolicy on create for a longer engagement.
• Organise containers by resource type, not by temporal batch.
• Import Operations are deleted 48 hours after creation, and an unresolved KeyReference resolves automatically if its target arrives inside that window — so prices may be imported before their products. Do not serialise the stages on polling; frequent polling of the summary endpoint actively slows the import. Retry only rejected; other states are retried internally, up to five times.
• Import Resources accept only KeyReferences, never ids.
• ProductVariantImport and StandalonePriceImport delete omitted fields on update. ProductVariantPatch is the only partial update.
• Load in dependency order — channels and customer groups, then product types, then categories, then products, then Standalone Prices if the price mode calls for them — and tear down in reverse, scoped to keys.prefix. Channels and customer groups are outside that scope and outside the Import API entirely: created through the platform API, and removable only by hand.
• Standalone Prices need manage_standalone_prices and customer groups need manage_customer_groups; manage_products covers every other request here.
• Containers, Operations and Summaries are generally available; the per-resource Import Requests are public beta.

Fixtures

Run any of them to see what a finding looks like before writing an adapter:

npm run pipeline -- validate --config fixtures/broken-integrity/migration.config.json
npm run pipeline -- audit --config fixtures/audit-violations/migration.config.json \
  --out fixtures/audit-violations/out

Source discovery

Profiling the real export — a sample of it, or all of it — before any mapping is designed, and turning it into a reviewable mapping proposal.

The rule

Do not teach the pipeline to read sources

What to look for, and which decision it settles

When the handover is the whole export

• Nothing to extrapolate to. The probe exists to predict what the full export will do; with the full export in hand, that prediction is just the answer.
• If onMissingDefinitions is require, the pipeline's inference never runs at all, and inference is most of what the probe is for. A source that declares its own types — a hybris items.xml, a PIM schema export — takes that path, which is exactly the case where a complete small export is likely.

The rest of this page still applies. Profiling, the observation table, the mapping proposal and the sign-off are all independent of how much data you have; only the probe step is conditional.

Say which path you took in the mapping proposal. "Read the complete export, 8 products, 90 SKUs" and "profiled a 500-row sample of ~40,000" support very different amounts of confidence, and the reader cannot tell them apart afterwards.

The probe feed

For a large export, where a sample is all you will get up front. The part that makes discovery evidence rather than opinion.

cd ct-catalog-migration-pipeline
npm run pipeline -- validate --config probe.config.json
npm run pipeline -- derive   --config probe.config.json --out out/probe

What this buys that an agent's analysis does not:

• The type, enum/lenum and attribute-constraint recommendations come from the same code that will run in production, so they cannot quietly disagree with it.
• mixed-value-types fires on any field holding more than one kind of value — the case where inference is not merely uncertain but wrong for some records.
• Every irreversible choice is already flagged: changeAttributeConstraint accepts only None, so a SameForAll or CombinationUnique proposed here is a one-way door, and derive marks it as such.
• validate failures on the probe feed are themselves findings. An axis with no stable code, a localized value used as identity, a duplicated axis combination from a collapsed hierarchy — each is a real property of the source, surfaced before the real adapter exists.
• It rehearses the adapter workflow at small scale, so the first real adapter is the second one written rather than the first.

The mapping proposal

Discovery ends in a named artefact, not a conversation, because it is the thing a human signs off. Write it next to the config:

# Mapping proposal — <engagement>

Sample: <what, how many records, from when>

## Source model as observed
<products vs variants, hierarchy depth, how categories are represented>

## Field mapping
| Source | commercetools | Type | Notes |
| :--- | :--- | :--- | :--- |
| ARTICLE_NO | variant sku | — | stable across exports; confirmed on two exports |
| COLOR_CODE / COLOR_NAME | attribute `colour`, axis | lenum | code + label present, so lenum |
| LONG_DESC_<locale> | product description | ltext | tags are `en_GB`; adapter re-keys to `en-GB` |

## Decisions needing sign-off
<each irreversible or lossy choice, with the cost of getting it wrong>

## Not migrating
<what is deliberately dropped, and why>

## Open questions
<what the sample could not answer>

The last two sections are the ones that get skipped and matter most. A migration claiming zero information loss is not being honest, and "not migrating" is where that honesty lives.

A sample is not the catalog

Skip this section if you read the complete export — the limits below are properties of sampling, not of discovery.

• Maximum variants per product — the constraint that can make a catalog unloadable is a long-tail property, and a 500-row sample will usually miss the worst case entirely.
• Attribute cardinality — a field with six values in the sample and four thousand in production is text, not an enum, and the enum choice is irreversible.
• Mixed value kinds — one bad record in fifty thousand is enough to make an inferred type wrong, and a sample is unlikely to contain it.
• Sentinel values — rare by nature.

State this when presenting the proposal. A mapping proposal offered as conclusions rather than hypotheses is how a migration commits to an irreversible attribute constraint on the strength of 500 rows.

The step-0 interview

• Order of work
• 0a. Can this pipeline serve the catalog at all?
• 0b. Discover the source
• 0c. The remaining judgements
• 0c-conditional. If slugs collide, ask who owns the URLs
• 0c-conditional. If 0b found relative media URLs, ask about the host
• 0d. The project facts
• Then open the decision log

Order of work

1. Size it (minutes): file list, line counts, largest variant count. Structural only — no mapping, no types, no decisions. Sizing is not the source work 0a means; writing an adapter is.
2. Ask 0a with that number in hand. Both its answers can end the engagement, and both are cheaper to learn now than after an adapter exists.
3. Discover the source (0b), then ask the rest (0c), because the data answers some of them.

0a. Can this pipeline serve the catalog at all?

Two questions, and they interact.

Resolve the two together, and say the outcome plainly:

• Pricing is Standalone only. Modular has no embedded prices at all, so target.priceMode must be standalone and the config refuses anything else. That also means the API Client needs manage_standalone_prices, which manage_products does not grant.
• defaultVariant cannot be imported. Modular replaces the master variant with Product.defaultVariant, which appears nowhere in the Import API. The pipeline still chooses and logs it deterministically, then cannot write it: products load with no default variant, the plan records the loss, and setting them needs an HTTP API pass this pipeline does not do.

0b. Discover the source

0c. The remaining judgements

Now informed by 0b. Each is irreversible or fails silently, and a wrong-but-consistent choice passes every offline stage:

The first two are storefront decisions with no evidence in the source, so they stay questions for whoever owns the implementation. The last two are the ones most often skipped — they feel like project management rather than modelling, and get discovered while writing the adapter, once the derivation is chosen.

0c-conditional. If slugs collide, ask who owns the URLs

Do not raise it when nothing collides.

0c-conditional. If 0b found relative media URLs, ask about the host

Do not raise this otherwise — it is not a standing question.

0d. The project facts

Then open the decision log

Writing a source adapter

Everything source-specific belongs here. Nothing source-specific belongs downstream.

Establish what the source is before writing any mapping

A migration designed against an assumed source model gets rewritten. Answer as much of this as the export allows, from the export itself rather than from documentation about the source system.

• How many products, variants, categories, and prices? Orders of magnitude, not exact counts — it decides whether the load design matters.
• Is this the whole catalog or a sample? A sample that behaves differently from production is worse than no sample.
• Which locales and currencies actually appear in the data, as opposed to which are configured somewhere?

• What is the stable machine identifier for a product? For a sellable SKU?
• Can it change between exports? If yes, keys derived from it will duplicate rather than update, and that has to be solved before anything else.
• Is there a separate identifier downstream systems join on? It goes in externalId.

• Does the source declare attribute types, or are they implied by values? Declared means emitting attributeDefinition records; implied means the pipeline has to infer, and every inference needs review.
• Are attributes split across more than one place? Many platforms keep some attributes in a product type definition and most of them in a separate classification or taxonomy structure. Mapping only the first half is a common and expensive mistake — check for a second source before designing anything.
• Which attributes vary per variant, and which are invariant per product?
• Which attributes constitute variant identity?

• How many levels does the hierarchy have? commercetools has exactly one, so anything deeper collapses in the adapter.
• What distinguishes two variants of the same product? Is that value a stable code, or display text?
• Are there non-sellable intermediate levels? They usually become nothing.

• Is a price inherited from anywhere? commercetools does not inherit between variants, so inheritance resolves here.
• What dimensions scope a price — currency, country, customer group, channel, date range, quantity?
• Are prices net or gross? This is often a setting elsewhere in the source, not a property of the price row, and getting it backwards changes every price by the tax rate with nothing in the data to contradict you.
• Quantity breaks have no embedded-price equivalent. Expect to report them.

• Are image paths absolute or site-relative? If relative, the host is not in the export — it is a question for whoever owns the storefront or CDN.
• How many renditions per shot? More than one means assets, not images.
• Does the source carry real dimensions, or will they default to 0×0?

• Is the tree navigation, or is it really a set of facets? A taxonomy used for filtering is usually better as attributes.
• Are names unique? Retail taxonomies reuse them across departments, which forces slug disambiguation.
• Is there a sibling ordering worth preserving?

Shape of an adapter

read source  →  assemble  →  emit
   parse         rebuild      one NDJSON
   the grammar   hierarchies  record per line
                 resolve
                 inheritance

Emit in any order. The contract allows forward references, so there is no need to buffer the whole catalog to get ordering right.

Rules that matter

Media

Images are a normal migration concern and they have three decisions in them, none of which the data answers.

If no host can be obtained, drop the images in the adapter and report them as not migrated. An absent image is recoverable; a wrong URL on every product is not, because it looks like success.

If you do have to reduce several renditions to one image, larger is the safer default: a storefront can scale down and cannot scale up. Record the choice — "the middle one" is a decision a retina storefront will disagree with.

Iterating

ERROR catalog.ndjson:17 [axis-combination-duplicate]
      Variants 'COLLAPSED-M-1' and 'COLLAPSED-M-2' share the axis combination
      size=M. Axes become CombinationUnique, so commercetools will reject the
      product. Either the hierarchy was collapsed wrongly, or an axis is missing
      from the declaration.

Fix the adapter, never the feed. The feed is regenerated on every run.

What good output looks like

• Codes are stable machine identifiers, and externalId preserves the raw source identifier.
• Axis values are codes; display text is in axisLabels.
• Every SKU has an explicit price, with inheritance already resolved.
• Locale tags are IETF form and the default locale is populated everywhere.
• Money is a decimal string with no more precision than its currency allows.
• Anything the adapter could not express is reported rather than approximated.