Data orchestration

What does this look like in practice?

An implementation that has multiple systems involved for Product could have the integration sending images from the DAM to the Import API first, then the product and category data from the PIM, and last of all the inventory data from the IMS. Once the Import API has received the data, it will process it and start to create or update the Product in Composable Commerce with all of the corresponding core product information, images, and inventory.

As the Import API is creating or updating the resource for you, you don’t need to provide the version number. This again simplifies the import process because you don’t need to do a read-ahead request to get the version number (this is a common synchronous practice that involves fetching the existing record version from Composable Commerce before initiating updates).

In essence, the Import API is a powerful tool designed not only for the initial import of diverse data sources but also for ongoing, efficient data management in Composable Commerce. Let’s get started learning how to use the Import API!

Terminology

Before we continue, let’s clarify some terms to better understand the import API:

Operation Synchronization

We say that we are synchronizing an identified resource from an external system to a Composable Commerce Project when you

• Create the resources in your Project if it does not exist, or
• Update the resource in your Project so that it matches all fields as given by an external system.

Import Container

To import resources using the Import API you must create at least one Import Container. Import Containers are the entry point of Import API requests, and serve as data containers for asynchronous API calls.

Each Import Container can have a defined resource type that specifies what resource types can be imported through it. If the resource type is not specified, the Import Container can import any resource supported by the Import API.

Here is a JSON snippet of an Import Container with a defined resource type:

{
  "key": "my-customer-import-container",
  "resourceType": "customer"
}

Import Request

An Import Request pushes data into your Import Container. Different types of Import Requests exist depending on which resource type they contain. Remember that each Import Request can contain a maximum of 20 resources.

{
  "type": "customer",
  "resources": [
    {
      "key": "imported-customer-01",
      "email": "sdavis@example.com",
      "password": "secret123",
      "firstName": "Sam",
      "lastName": "Davis",
      "addresses": [
        {
          "key": "imported-customer-01-address",
          "country": "DE"
        }
      ]
    }
  ]
}

Import Response

Import Responses are returned by the Import API once an Import Request is received. It contains a list of operationId and the current state of newly created Import Operations.

{
  "operationStatus": [
    {
      "operationId": "980b7a5d-c1a8-4b74-8584-1bbd6cf0eb01",
      "state": "processing"
    }
  ]
}

Import Operation

Import Operations describe the processing state of a specific resource being imported. For example, if a resource cannot be imported due to missing references it will have the unresolved status (and the missing references will be listed in the Operation’s unresolvedReferences attribute).

Import Operations are automatically deleted 48 hours after they have been created, and that is also how long they can wait for missing references to be added to the Import Request.

{
  "version": 2,
  "importContainerKey": "my-customer-import-container",
  "resourceKey": "imported-customer-01",
  "id": "980b7a5d-c1a8-4b74-8584-1bbd6cf0eb01",
  "state": "imported",
  "resourceVersion": 1,
  "createdAt": "2023-11-23T09:37:54.153Z",
  "lastModifiedAt": "2023-11-23T09:38:46.795Z",
  "expiresAt": "2023-11-25T09:37:54.153Z"
}

Import Summary

Import Summaries allow you to see exactly how many resources an Import Container has in each processing state. This is perfect for monitoring the progress of your import operations.

{
  "states": {
    "processing": 0,
    "validationFailed": 0,
    "unresolved": 0,
    "waitForMasterVariant": 0,
    "imported": 1,
    "rejected": 0,
    "canceled": 0
  },
  "total": 1
}

Importing customer data

Let’s come back to the focus of this Developer Essentials learning path: the Customer resource. How can the Import API be used to manage a merchant’s customer data?

Imagine that you have 100,000 Customers to import into your Project. By "import" we mean that some of them have to be created but others may already exist and just need to be updated. Could we solve this problem by using the HTTP API? Absolutely! The process would, probably, look something like this:

If we look at it in more detail, we can see that this is a complex and challenging process. After receiving a large amount of Customers to be imported into Composable Commerce you must perform the following steps:

• You need to query the existing Customers in your Composable Commerce Project. By doing this you will know whether or not you need to update a Customer or create a new Customer. This query also gives you the version of the Customer, which you need to update Customers.
• Now that you know which Customers to create or update in Composable Commerce, you can write an application that will send as many POST requests as needed. Each containing a CustomerDraft to create a new Customer.
• For all remaining Customers, you need to compare the Customer data provided by the CRM versus the Customer data fetched from Composable Commerce. For all the fields that must be updated, you add it to a list of update actions for the Customer.
• Remember that you must wait for a response for each of those requests. As the Internet is a loosely coupled system, a small number of requests will not reach Composable Commerce (or the answer won’t reach you). Hence, you need to check your logs, analyze them and in some cases, retry the requests.
• Another possible step stems from whether or not the Customers are assigned to Customer Groups that don’t exist yet. Those calls would of course fail and you would have to send them again once you have created the missing Customer Groups.

This seems like a time-consuming task!

With the Import API, this process can be made smoother and easier. The process using the Import API would look a lot more like this:

Using the Import API, the same task we tried before can now be accomplished with the following steps:

• Create an Import Container that will accept your new Customers and pass those to the Project.
• Write an application that would get a batch of Customers from the CRM and add 20 to the Import Container until all Customers are looped through (Why only 20? Because that is the limit of the Import API).
• Monitor the status of the Import Operations to know when all Customers have been imported.
• What if some Customers are assigned to Customer Groups that don’t exist yet? That’s OK. The Import API will wait 48 hours and the moment those Customer Groups are added, it will create those Customer resources.
• What if the create or update operation of a Customer fails? The Import API will retry up to five times.

Isn’t it so much easier? No complex loops, no need to check your internal logs. Great!

Import API workflow

Now that you have a better understanding of the terminology used when discussing the Import API, you can confidently go over the Import API workflow in the docs before giving it a try yourself in the next section. Make sure you understand the Processing states as well!

Advantages of the Import API

Based on comparing the two solutions to the problem in the previous section, you can see that the Import API is the preferred choice for importing large amounts of data. Let us summarize the advantages:

• Each API call can be used to create/update up to 20 resources.
• Regardless of whether you are trying to create or update a resource, the request is the same when using the Import API.
• The Import API is asynchronous, meaning that you do not have to wait for responses to your calls. The data import is processed in the background and you can always query the Import API to check the import status.
• The Import API supports dynamic dependency handling, which means that if some of the imported resources are missing certain dependencies, the Import API will wait up to 48 hours and once those dependencies are added, it will process the resources.
• Update actions are not needed when updating a resource. You only need to specify the fields that are changing.

However, one of the advantages of the Import API has consequences for the management of your Project. You cannot expect your data to immediately change. For example, if you need to remove a product from your store immediately due to incorrect data, you should use CRUD (and therefore, synchronous) operations, triggered either from your application or the Merchant Center.

Supported resources

Thus far we have focused on Products and Customers, however you can use the Import API with a large number of other resources. A full list of resources that can be imported using the Import API can be found in the documentation.

As a consequence of the complexity of importing Products you might notice that there are different requests related to product resources (Product, Product Draft, Product Variant, Product Variant Patch). Each of them exists to help you perform a specific import scenario:

• Product - Used to only import Product data (without variants).
• Product Draft - Used to import Products (including their variants).
• Product Variant - Used to import only Product Variants for existing Products.
• Product Variant Patch - Used to import updates for existing Product Variant Attributes.

Test your knowledge