# Data orchestration Every Composable Commerce implementation has special nuances when it comes to importing data into a Project. Some integrations only need to get data from a single system, such as an enterprise resource planning (ERP) system, while others may have the data reside across many different systems. This could consist of any combination of the following systems for Product data: - Product information management (PIM) systems for Category and core product information. - Digital asset management (DAM) systems for product images. - Inventory management systems (IMS) for product inventory. - Price optimization and management (PO\&M) systems for B2B and customer-targeted pricing. - Enterprise resource planning (ERP) for core unenriched product data. Pulling different parts of product data from each system, merging that data together, and sending the product payload using the HTTP API can be burdensome to orchestrate when many systems are involved. To simplify this task, we can use the Composable Commerce [Import API](https://docs.commercetools.com/api/import-export/overview.md). The Import API is an asynchronous endpoint that allows you to send different parts of a resource at a time, and the Import API will resolve the relationships between each piece of data and import the whole resource as one entity. Every part of this reference data must be sent to the Import API within 48 hours and you can use the Import API to batch create or update resources. ### What does this look like in practice? An implementation that has multiple systems involved for Product could have the integration sending images from the DAM to the Import API first, then the product and category data from the PIM, and last of all the inventory data from the IMS. Once the Import API has received the data, it will process it and start to create or update the Product in Composable Commerce with all of the corresponding core product information, images, and inventory. For example, you can submit Products for import before their images exist in the platform. When you later import the images, the Import API automatically finds and links them to the correct Products. As the Import API is creating or updating the resource for you, you don’t need to provide the version number. This again simplifies the import process because you don’t need to do a read-ahead request to get the version number (this is a common synchronous practice that involves fetching the existing record version from Composable Commerce before initiating updates). In essence, the Import API is a powerful tool designed not only for the initial import of diverse data sources but also for ongoing, efficient data management in Composable Commerce. Let’s get started learning how to use the Import API! ## Terminology Before we continue, let’s clarify some terms to better understand the import API: ### Operation Synchronization We say that we are synchronizing an identified resource from an external system to a Composable Commerce Project when you - Create the resources in your Project if it does not exist, or - Update the resource in your Project so that it matches all fields as given by an external system. ### Import Container To import resources using the Import API, you must first create an [Import Container](https://docs.commercetools.com/api/import-export/import-container.md). Import Containers serve as the entry point for all Import API requests. Each Import Container can optionally specify a resource type to restrict which resources can be imported through it. Projects have a default limit of 1000 Import Containers. To stay within this limit, organize containers by resource type rather than by temporal batches, and delete obsolete containers to free up capacity. For details on container organization strategies, see [Import API best practices](https://docs.commercetools.com/api/import-export/best-practices.md#effective-use-of-import-containers). Container lifetime: Import Containers created without a `retentionPolicy` expire automatically 72 hours after creation. If a hands-on exercise takes longer, set a [TimeToLiveRetentionPolicy](https://docs.commercetools.com/api/import-export/import-container.md#timetoliveretentionpolicy) when creating the container or create a new container before the previous one expires. ### Import Request An [Import Request](https://docs.commercetools.com/api/import-export/overview.md#what-are-importrequests) pushes data into your Import Container. Different Import Request types exist depending on the resource type they contain, and each Import Request can contain up to 20 resources. For the full list of supported resource types and their request formats, see [supported resources](https://docs.commercetools.com/api/import-export/overview.md#supported-resources). For example, a [CustomerImportRequest](https://docs.commercetools.com/api/import-export/import-requests.md#customerimportrequest) to import a single Customer looks like this: ```json { "type": "customer", "resources": [ { "key": "imported-customer-01", "email": "sdavis@example.com", "password": "secret123", "firstName": "Sam", "lastName": "Davis", "addresses": [ { "key": "imported-customer-01-address", "country": "DE" } ] } ] } ``` ### Import Response, Import Operation, and Import Summary After you submit an Import Request, the Import API returns an [Import Response](https://docs.commercetools.com/api/import-export/overview.md#what-are-importresponses) confirming that the data has been received. Receiving an Import Response does not mean the resources have been imported. The response confirms that the Import API is processing the data asynchronously. The Import API creates one [Import Operation](https://docs.commercetools.com/api/import-export/overview.md#what-are-importoperations) per resource in the Import Request. Each Import Operation tracks the [ProcessingState](https://docs.commercetools.com/api/import-export/import-operation.md#processingstate) of its resource. If a resource cannot be imported due to missing references, the Import Operation has the `unresolved` state until those references are provided. Import Operations are automatically deleted 48 hours after creation. To monitor the overall progress of an Import Container, use the [Import Summary](https://docs.commercetools.com/api/import-export/import-container.md#get-importsummary-of-importcontainer) endpoint. It provides an aggregated count of Import Operations in each processing state. ## Import customer data Let’s come back to the focus of this Developer Essentials learning path: the Customer resource. How can the Import API be used to manage a merchant’s customer data? Imagine that you have 100,000 Customers to import into your Project. By "import" we mean that some of them have to be created but others may already exist and just need to be updated. Could we solve this problem by using the HTTP API? Absolutely! The process would, probably, look something like this: ```mermaid sequenceDiagram autonumber participant CRM participant Your backend participant Composable Commerce/Customers CRM->>Your backend: Push 100,000 customers loop iterate for 100,000 customers Your backend->>Composable Commerce/Customers: Get(Query) Composable Commerce/Customers-->>Your backend: CustomerPagedQueryResponses[] end loop iterate for 100,000 customers loop Until ok loop For new Customers Your backend->>Composable Commerce/Customers: POST(CustomerDraft) end loop For existing Customers Your backend->>Your backend: Create updateActions[] Your backend->>Composable Commerce/Customers: POST(UpdateActions) end Your backend->>Your backend: inspect logs, handle errors, handle failed connections end end ``` If we look at it in more detail, we can see that this is a complex and challenging process. After receiving a large amount of Customers to be imported into Composable Commerce you must perform the following steps: - You need to query the existing Customers in your Composable Commerce Project. By doing this you will know whether or not you need to update a Customer or create a new Customer. This query also gives you the `version` of the Customer, which you need to update Customers. - Now that you know which Customers to create or update in Composable Commerce, you can write an application that will send as many POST requests as needed. Each containing a CustomerDraft to create a new Customer. - For all remaining Customers, you need to compare the Customer data provided by the CRM versus the Customer data fetched from Composable Commerce. For all the fields that must be updated, you add it to a list of update actions for the Customer. - Remember that you must wait for a response for each of those requests. As the Internet is a loosely coupled system, a small number of requests will not reach Composable Commerce (or the answer won’t reach you). Hence, you need to check your logs, analyze them and in some cases, retry the requests. - Another possible step stems from whether or not the Customers are assigned to Customer Groups that don’t exist yet. Those calls would of course fail and you would have to send them again once you have created the missing Customer Groups. This seems like a time-consuming task! With the Import API, this process can be made smoother and easier. The process using the Import API would look a lot more like this: ```mermaid sequenceDiagram autonumber participant CRM participant Your backend participant ImportContainers participant ImportContainerKey participant ImportOperations Your backend->>ImportContainers: POST (ImportContainerDraft) ImportContainers->>ImportContainers: Create Import Container ImportContainers-->>Your backend: ImportContainer loop For every 20 customers CRM->>Your backend: Fetch batch of customers Your backend->>ImportContainerKey: POST (CustomerImportRequest) ImportContainerKey-->>Your backend: ImportResponse end Your backend->>ImportOperations: GET ImportOperations-->>Your backend: ImportOperationPagedResponse ``` Using the Import API, the same task we tried before can now be accomplished with the following steps: - Create an Import Container that will accept your new Customers and pass those to the Project. - Write an application that would get a batch of Customers from the CRM and add 20 to the Import Container until all Customers are looped through (Why only 20? Because that is the [limit](https://docs.commercetools.com/api/import-export/overview.md#what-are-importrequests) of the Import API). - Monitor the status of the Import Operations to know when all Customers have been imported. - What if some Customers are assigned to Customer Groups that don’t exist yet? That’s OK. The Import API will wait 48 hours and the moment those Customer Groups are added, it will create those Customer resources. - What if the create or update operation of a Customer fails? The Import API will retry up to five times. Isn’t it so much easier? No complex loops, no need to check your internal logs. Great! ### The importance of keys While Composable Commerce resources can be identified and referenced using various [identifier fields](https://docs.commercetools.com/learning-composable-commerce-developer-essentials/manage-resources-with-the-sdk/manage-resources-crud.md#identify-resources), the Import API only uses `key`. Assigning `key` values to your resources is critical for effective data management with the Import API. - Stable identification for updates: The Import API uses the `key` attribute to identify existing resources. If you import data with a key that matches an existing resource, the Import API updates that resource instead of creating a new one. Without keys, you risk creating duplicate entries. - Idempotency: Keys make import operations idempotent. If an import is interrupted and retried, the Import API correctly updates existing resources without creating duplicates, ensuring the desired state is eventually reached. - Dependency resolution: Keys are vital for resolving dependencies between resources. For example, when importing Products, you can reference Categories by `key` (using a [KeyReference](https://docs.commercetools.com/urn?urn=ctp%3Aimport%3Atype%3AKeyReference)). The Import API then automatically links the resources. - Merchant Center compatibility: The Merchant Center's import functionality also relies on keys to identify and update existing resources. Without keys, the Merchant Center import process [creates new resources instead of modifying existing ones](https://docs.commercetools.com/merchant-center/import-data.md#add-keys-to-existing-resources-and-objects). ## Import API workflow Now that you have a better understanding of the terminology used when discussing the Import API, you can confidently go over the [Import API workflow](https://docs.commercetools.com/api/import-export/overview.md#how-to-use-the-import-api) in the docs before giving it a try yourself in the next section. Make sure you understand the [ProcessingStates](https://docs.commercetools.com/api/import-export/import-operation.md#processingstate) as well! ## Advantages of the Import API Based on comparing the two solutions to the problem in the previous section, you can see that the Import API is the preferred choice for importing large amounts of data. Let us summarize the advantages: - Each API call can be used to create/update up to 20 resources. - Regardless of whether you are trying to create or update a resource, the request is the same when using the Import API. - The Import API is asynchronous, meaning that you do not have to wait for responses to your calls. The data import is processed in the background and you can always query the Import API to check the import status. - The Import API supports dynamic dependency handling, which means that if some of the imported resources are missing certain dependencies, the Import API will wait up to 48 hours and once those dependencies are added, it will process the resources. - Update actions are not needed when updating a resource. You only need to specify the fields that are changing. However, one of the advantages of the Import API has consequences for the management of your Project. You cannot expect your data to immediately change. For example, if you need to remove a product from your store immediately due to incorrect data, you should use CRUD (and therefore, synchronous) operations, triggered either from your application or the Merchant Center. ## Supported resources Thus far we have focused on Products and Customers, however you can use the Import API with a large number of other resources. A full list of resources that can be imported using the Import API can be found in the [documentation](https://docs.commercetools.com/api/import-export/import-container.md#importresourcetype). As a consequence of the complexity of importing Products you might notice that there are different requests related to product resources (Product, Product Draft, Product Variant, Product Variant Patch). Each of them exists to help you perform a specific import scenario: - [Product](https://docs.commercetools.com/api/import-export/import-resources.md#productimport) - Used to only import Product data (without variants). - [Product Draft](https://docs.commercetools.com/api/import-export/import-resources.md#productdraftimport) - Used to import Products (including their variants). - [Product Variant](https://docs.commercetools.com/urn?urn=ctp%3Aimport%3Atype%3AProductVariantImport) - Used to import only Product Variants for existing Products. - [Product Variant Patch](https://docs.commercetools.com/api/import-export/import-resources.md#productvariantpatch) - Used to import updates for existing Product Variant Attributes. ## Related pages - [Section overview page](https://docs.commercetools.com/learning-composable-commerce-developer-essentials.md) - [Previous page: Overview](https://docs.commercetools.com/learning-composable-commerce-developer-essentials/import-api/overview.md) - [Next page: Use the Import API](https://docs.commercetools.com/learning-composable-commerce-developer-essentials/import-api/use-the-import-api.md)