# Import API best practices To best utilize the Import API, we recommend the following best practices. ## Effective use of Import Containers ### Organize Import Containers You should plan how you use [Import Containers](https://docs.commercetools.com/api/import-export/import-container.md) deliberately. Do not create Import Containers arbitrarily. For example, avoid creating one-off Import Containers for small batches without reuse or cleanup. Projects have a maximum of 1000 Import Containers and new Import Containers expire by default after 72 hours unless you set a [retention policy](https://docs.commercetools.com/api/import-export/import-container.md#retentionpolicy). Aim for fewer, meaningful containers: more containers with bounded data per container are often better than very few containers holding huge operation counts. Every new container counts toward the per-Project limit and inherits the default TTL when you omit `retentionPolicy`. - When importing full data sets, a dedicated [Import Container](https://docs.commercetools.com/api/import-export/import-container.md) for that run can separate one import from another, provided the work finishes within the default 72-hour window or you set an appropriate `retentionPolicy`. - When performing routine imports, reuse a stable [Import Container](https://docs.commercetools.com/api/import-export/import-container.md) (for example, per resource type or integration) instead of minting a new key for every run. - When importing from multiple sources, one [Import Container](https://docs.commercetools.com/api/import-export/import-container.md) per source can clarify monitoring and errors. - When performing high-volume, short-lived imports, rely on the default 72-hour lifetime or tune `timeToLive` via [TimeToLiveRetentionPolicy](https://docs.commercetools.com/api/import-export/import-container.md#timetoliveretentionpolicy). Typical layouts include organizing by resource type, reusing a container for recurring jobs (within TTL or with a longer policy), or separating by data source. (a CSV formatted table follows. The first line are the column names.) Use Case,Possible Import Container organization Import Product and Category,Create separate Import Containers for Product and Category Import Price changes daily at 5 PM,Create a reusable Import Container. If there are more than 200000 imports per day, this may be broken down by some other business logic or temporary Import Container for excess counts. Import Product changes from multiple sources,Create one Import Container per source for Product imports. ### Optimize performance To achieve the best performance with the Import API, we recommended having fewer than 200000 [Import Operations](https://docs.commercetools.com/api/import-export/import-operation.md) per [Import Container](https://docs.commercetools.com/api/import-export/import-container.md). This way, monitoring activities at the container level will not be costly. As [Import Operations](https://docs.commercetools.com/api/import-export/import-operation.md) are automatically deleted 48 hours after they are created, you can reuse [Import Containers](https://docs.commercetools.com/api/import-export/import-container.md) over time. An example schedule is as follows: (a CSV formatted table follows. The first line are the column names.) Day,Import Operation total count,Import Containers (Import Operation count) Day 1,100,000,container-a (100,000) Day 2,500,000,container-a (100,000), container-b (200,000), container-c (200,000) Day 3,400,000,container-a (0), container-b (200,000), container-c (200,000) Day 4,200,000,container-a (200,000), container-b (0), container-c (0) On Day 3, container-a will be empty as all the Import Operations have reached 48 hours. container-a is now ready to be reused. Similarly container-b and container-c can be reused from Day 4. ### Clean up data from Import Containers and remove unused Import Containers [Import Operations](https://docs.commercetools.com/api/import-export/import-operation.md) inside an Import Container are automatically deleted 48 hours after they are created. That does not remove the [Import Container](https://docs.commercetools.com/api/import-export/import-container.md) itself. You should consider Import Containers to be short-lived workspaces tied to an import job or integration flow. An Import Container lifetime is determined when you create it: if you omit `retentionPolicy`, the default lifetime is 72 hours; if you set a [TimeToLiveRetentionPolicy](https://docs.commercetools.com/api/import-export/import-container.md#timetoliveretentionpolicy), the lifetime is defined by its `timeToLive` value (between 1 hour and 30 days). If you create an Import Container without a `retentionPolicy`, the Import Container is deleted 72 hours after creation. The Import Operations associated with this Import Container are also removed at this time. You can also [delete Import Containers](https://docs.commercetools.com/api/import-export/import-container.md#delete-importcontainer) manually at any time. That immediately deletes all [Import Operations](https://docs.commercetools.com/api/import-export/import-operation.md) in the container. Data already imported into your Project is not affected. Set a buffer in `timeToLive` so imports and monitoring finish before `expiresAt`. If you need a container to live longer than 30 days, you must create a new container or adjust your workflow. Newly created containers without a suitable policy do not have an unlimited default lifetime. #### Automatic deletion and retention policy When [creating an ImportContainer](https://docs.commercetools.com/api/import-export/import-container.md#create-importcontainer), the response includes `expiresAt` for containers that follow the default or TTL-based lifecycle. Omitting `retentionPolicy` means the default 72-hour TTL. Setting a [TimeToLiveRetentionPolicy](https://docs.commercetools.com/api/import-export/import-container.md#timetoliveretentionpolicy) sets `expiresAt` from your `timeToLive` value. When `expiresAt` is reached, the ImportContainer and its Import Operations are permanently deleted. ## Import large data sets You can import 20 [resources](https://docs.commercetools.com/api/import-export/import-resources.md) per [Import Request](https://docs.commercetools.com/api/import-export/overview.md#what-are-importrequests). If you have a huge number of resources to import, we recommend using thread optimization to send your data as fast as possible to an [Import Container](https://docs.commercetools.com/api/import-export/import-container.md). You can import as much data you need using the Import API, keeping in mind the best practices for [rate limits](https://docs.commercetools.com/api/import-export/best-practices.md#rate-limits) and [JSON document size limit](https://docs.commercetools.com/api/limits.md#json-document-size). Please note that the asynchronous import process starts as soon as the first [Import Request](https://docs.commercetools.com/api/import-export/overview.md#what-are-importrequests) is received by the [Import Container](https://docs.commercetools.com/api/import-export/import-container.md). For more information on arranging Import Containers for importing large data sets, see [optimize performance](https://docs.commercetools.com/api/import-export/best-practices.md#optimize-performance). ### Rate limits The Import API does not have rate limits, but to ensure the best performance we recommend that you send 300 API calls per second, per Project, to the Import API. For example, if you send 300 [Import Requests for Categories](https://docs.commercetools.com/api/import-export/import-requests.md#for-categories) per second, and each [CategoryImportRequest](https://docs.commercetools.com/api/import-export/import-requests.md#categoryimportrequest) contains 20 [CategoryImport](https://docs.commercetools.com/api/import-export/import-resources.md#categoryimport) items, this means that up to 360,000 Categories can be sent to the Import API every minute. ## Choose the right Product import endpoint When updating an existing Product or ProductVariant by import, you must include the existing values of all fields you want to keep in [ProductDraftImport](https://docs.commercetools.com/api/import-export/import-resources.md#productdraftimport), [ProductImport](https://docs.commercetools.com/api/import-export/import-resources.md#productimport), and [ProductVariantImport](https://docs.commercetools.com/api/import-export/import-resources.md#productvariantimport). The import process will remove the values of omitted fields. The Import API has multiple endpoints for importing Product data. The following table summarizes what can be imported by each endpoint. (a CSV formatted table follows. The first line are the column names.) Endpoint,What can be imported [Product Drafts](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-drafts),Product data including Product Variants and Prices. [Products](https://docs.commercetools.com/api/import-export/import-requests.md#for-products),Product data without Product Variants and Prices. [Product Variants](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-variants),Product Variant data without Prices. [Product Variant Patches](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-variant-patches),Product Variant Attribute data. [Embedded Prices](https://docs.commercetools.com/api/import-export/import-requests.md#for-embedded-prices),Price data for a specific Product Variant. The following information explains common use cases for each endpoint. ### Product Drafts Use import requests for [Product Drafts](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-drafts) for large payloads that include complete sets of Product data including Product Variants and Embedded Prices. Effective use of this endpoint can remove the need to create three separate import requests for ([Products](https://docs.commercetools.com/api/import-export/import-requests.md#for-products), [Product Variants](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-variants), and [Embedded Prices](https://docs.commercetools.com/api/import-export/import-requests.md#for-embedded-prices)). Common use cases: - when you want to update a Product with 10+ new Product Variants, each with 20+ Embedded Prices. - when you want to update a large number of Product Variants with new Attribute values and Prices. ### Products Use import requests for [Products](https://docs.commercetools.com/api/import-export/import-requests.md#for-products) to create or update Products without Product Variant or Price data. As [ProductImport](https://docs.commercetools.com/api/import-export/import-resources.md#productimport) does not import any [ProductVariant](https://docs.commercetools.com/urn?urn=ctp%3Aapi%3Atype%3AProductVariant) or Price data, it will result in better performance due to the smaller payload. ### Product Variants Use import requests for [Product Variants](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-variants) to create or update Product Variants without Price data. ### Product Variant Patches Use import requests for [Product Variant Patches](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-variant-patches) to update the Attributes of existing Product Variants. ### Embedded Prices Use import requests for [Embedded Prices](https://docs.commercetools.com/api/import-export/import-requests.md#for-embedded-prices) to update the Price data of existing Product Variants. ## Manage published state of Products Both [ProductImport](https://docs.commercetools.com/api/import-export/import-resources.md#productimport) and [ProductDraftImport](https://docs.commercetools.com/api/import-export/import-resources.md#productdraftimport) have the field `publish` which accepts a boolean value. The result of using this field varies based on whether you are importing data to create or update Products. ### When importing data to create a Product If `true`, the Product is created and published immediately. If `false`, the Product is created but not published. ### When importing data to update an existing Product The result of updating existing Products depends on if you are including changes in your import request, and if the Product currently has staged changes. (a CSV formatted table follows. The first line are the column names.) Value of `publish`,Does the import request have changes?,Does the Product have staged changes before importing?,Result `true`,No,Yes,The staged changes are applied to the current projection and the Product is published. `true`,No,No,If the Product is currently unpublished, it is published to the current projection. `true`,Yes,N/A,The changes are applied to both the current and staged projection. When using [ProductImport](https://docs.commercetools.com/api/import-export/import-resources.md#productimport): if the Product is currently unpublished, it is not published. When using [ProductDraftImport](https://docs.commercetools.com/api/import-export/import-resources.md#productdraftimport): if the Product is currently unpublished, it is published. `false`,No,N/A,The Product is unpublished. `false`,Yes,N/A,The changes are applied to the staged projection, the Product is unpublished, and `hasStagedChanges` becomes `true`. ## Utilize the lifetime of Import Operations [Import Operations](https://docs.commercetools.com/api/import-export/import-operation.md) are kept for 48 hours to allow you to send other resources, including those referenced by other resources (unresolved references), during this time period. For example, one of your teams is responsible for Product import but the business validation usually delays the Product import for 1-2 days, and another team that imports Prices is very fast in importing the data, the Import API keeps the Price data for up of 48 hours and waits for the Product to be imported. ## Handle retries You only need to retry if your [Import Operation](https://docs.commercetools.com/api/import-export/import-operation.md) has the `rejected` status. In other cases, the Import API will handle the retry internally without you needing to do anything. ## What not to do Do not send duplicate import requests concurrently. Since the Import API imports data asynchronously, the order is not guaranteed. It may also lead to a concurrent modification error. If [errors](https://docs.commercetools.com/api/errors.md#import-api-specific-errors) occur, you should fix them before [querying ImportOperations in ImportContainer](https://docs.commercetools.com/api/import-export/import-operation.md#query-importoperations-in-importcontainer) or [getting the ImportSummary of ImportContainer](https://docs.commercetools.com/api/import-export/import-container.md#get-importsummary-of-importcontainer). Frequently requesting these endpoints may slow down the import process. The data returned by [Get ImportSummary of ImportContainer](https://docs.commercetools.com/api/import-export/import-container.md#get-importsummary-of-importcontainer) is [eventually consistent](https://docs.commercetools.com/api/general-concepts.md#eventual-consistency), so do not rely on it for real-time monitoring. For assistance in issues or errors, [query the Import Operation](https://docs.commercetools.com/api/import-export/import-operation.md#query-importoperations-in-importcontainer) and consult the `errors` field. ## Avoid concurrency errors When importing [Product Variant Patches](https://docs.commercetools.com/api/import-export/import-requests.md#for-product-variant-patches), provide the reference to the [Product](https://docs.commercetools.com/urn?urn=ctp%3Aapi%3Atype%3AProduct) that contains the Product Variant. The value for the `product` field in the [ProductVariantPatch](https://docs.commercetools.com/api/import-export/import-resources.md#productvariantpatch) minimizes concurrency errors during the import process. If you set the `product` field on one ProductVariantPatch, you have to set it for every ProductVariantPatch in the same [ProductVariantPatchRequest](https://docs.commercetools.com/api/import-export/import-requests.md#productvariantpatchrequest). Otherwise, the API returns an [InvalidField](https://docs.commercetools.com/urn?urn=ctp%3Aimport%3Atype%3AInvalidFieldError) error. ## Related pages - [Section overview page](https://docs.commercetools.com/api.md) - [Previous page: Overview](https://docs.commercetools.com/api/import-export/overview.md) - [Next page: Hosts and authorization](https://docs.commercetools.com/api/import-export/hosts-and-authorization.md)