Developing a data source extension

A data source extension makes data available to your site which:

  1. Can be configured and influenced from studio

  2. Is available immediately when the site loads (also during Server Side Rendering)

  3. Can be connected simultaneously to multiple Frontend components on the same page

You can develop a data source extension by following the steps below:

1. Specify a data source type


A data source needs to be specified so the studio knows that it exists and can expose configuration to its users. So, a data source schema (see the schemas article for more information) is created. The schema defines the data source type identifier (which is used to resolve the data source to your extension code) and a list of fields for configuration. For example:

{
"customDataSourceType": "example/star-wars-movie",
"name": "Star wars movie",
"category": "Content",
"icon": "stars",
"schema": [
{
"name": "Settings",
"fields": [
{
"label": "Movie ID",
"field": "movieId",
"type": "string",
"translatable": false
}
]
}
]
}

You can write this schema directly in the studio. If you want to store it, you can store it wherever you like as it's only required by the studio and not by the code.

The customDataSourceType field is required and needs to comply with the format <vendor>/<identifier>. You'll usually use your company name or project name as the <vendor> and use a speaking identifier for the data source.

The name specifies how the data source is referenced in the studio and should be understandable for studio users. The category and icon are specific schema fields for visualization in the studio.

The schema part holds a list of configuration sections where each section consists of some fields. This area is typical for schema definitions. More details are in the schemas article.

This example exposes a single input field where a studio user can enter the ID of a Star Wars movie episode. The data source will then fetch information about the selected movie from a backend API.

2. Announce the data source type to the studio


To let the studio know about your new data source, you need to create your schema in the studio. To do that:

  1. Open the studio and click Developer on the dashboard

9ce9876 Click Developer on the dashboard

  1. Click Data sources

65995a4 Click data sources in the developer area

  1. Copy the schema example from earlier, and click the Create schema button

76e7b3f Click the create schema button

  1. Paste the schema into the schema editor and click Validate

73c5c4e Paste in JSON and click validate

If you include the dataSourceType in your JSON, you don't need to add it to the required field input.

  1. Click Publish

2548517 Click publish

3. Configure a data source on a page folder


A data source is always connected to a page folder. With that, the data delivered by it is made available to the currently active page in it and so to all components that exist on this page.

  1. Go to the Site builder

02552db Click the site builder icon on the dashboard

  1. Hover on a page folder and click the settings icon 00b9bb4 Click the settings icon on a page folder

  2. Expand the Data source section and click + Add data source filter

a2c087c Click add data source

  1. Select your new data source from the dropdown

e7bcc83 Select your data source from the dropdown

This opens the data source filter editor.

  1. Enter the Movie ID for the episode number (ZmlsbXM6MQ==), change the filter name if you want to, and then click Save

618a068 Enter the movie id and click save

  1. Check your data source filter is there, then click Save

90d9dd2 Check your data source then click save

4. Create a component that consumes the data source


commercetools Frontend won't load data from a data source unless there's at least 1 Frontend component assigned to the live page version which consumes it. So, such a component must be specified and placed on a page version.

A simple test component like the below works:

{
"tasticType": "example/star-wars-opening-crawl",
"name": "Star Wars opening crawl",
"icon": "movie_filter",
"category": "general",
"schema": [
{
"name": "Data source",
"fields": [
{
"label": "Star Wars movie",
"field": "movieData",
"type": "dataSource",
"dataSourceType": "example/star-wars-movie"
}
]
}
]
}

This component has only 1 configuration field of type dataSource of which the dataSourceType references the previously specified data source. This means: When this component is used, it needs a corresponding data source on the page folder and needs to be attached to it.

5. Place the component onto a page


  1. Click Components on the left-hand navigation

ce9eb05 Click components on the left hand

  1. Click the Create schema button, this opens the schema editor

7db78b4 Click create schema to create a component

  1. Paste the JSON example into the editor, click Validate and preview, then Publish and download JSON

b4def0e Paste the json then validate and publish

  1. Click Site builder on the left-hand navigation

272ebd5 Click the site builder icon

  1. Select the page folder you added the data source to earlier and click New, then select Create page version

9e1511c Select page folder then create a new page version

  1. Click the blue add icon in the middle section

c9a84f3 Click the add icon

  1. Select the 1 layout element

8a90fb4 Select the layout element

  1. Find your Frontend component in the section on the left and drag it into the layout element

ae76607 Drag in the component

  1. Select the data source filter you created earlier in the page folder settings

3ca21b5 Select the data source filter

  1. Click Save

473929e Click save

  1. Click the more icon on your draft page version and select Make default

fd00e3e Click more then select make default

6. Test the data source


Now that a page folder exists with a configured instance of the data source and a page version that has a Frontend component to consume it, the data source can be tested. This works best using your favorite HTTP client, like Postman, HTTPie, or curl.

To access the API hub on your sandbox, look up its public URL in studio. To do this, open the studio, click Developer, then Sandboxes, find your sandbox, then click See public URLs, and then click the copy icon next to the URL.

Then you can fire a GET request to the path /frontastic/page on your sandbox URL. The request needs to be sent with the correct Accept header for application/json and the following 2 query parameters:

  • path holds the URL path to the page folder you created (for example, from above /data-source)
  • locale holds a valid locale for your project (for example, en_US).

For example:

curl -X 'GET' -H 'Accept: application/json' 'https://<sandbox-public-url>/frontastic/page?path=/data-source/lea&locale=en_US'

This call will return a full page payload of which only the following parts are significant for the data source test:

{
...
"data": {
"_type": "Frontastic\\Catwalk\\NextJsBundle\\Domain\\PageViewData",
"dataSources": {
"ee92bc20-f4b5-4e9d-98cf-b8c8d7300563": {
"message": "No stream handler for data source type example/star-wars-movie configured.",
}
}
},
"page": {
...
},
"pageFolder": {
"_type": "Frontastic\\Catwalk\\NextJsBundle\\Domain\\Api\\PageFolder",
...
"dataSourceConfigurations": [
{
"configuration": {
"movieId": "ZmlsbXM6MQ=="
},
"name": "Star wars movie",
"streamId": "ee92bc20-f4b5-4e9d-98cf-b8c8d7300563",
"type": "example/star-wars-movie"
}
],
...
}
}

The data key of the payload contains the page folder global data, which comes from data sources. In this example, there's only 1 data source that contains an error message (because it wasn't implemented yet). page is out of scope here. The pageFolder, among other information, contains the dataSourceConfigurations where the configuration of the desired Star Wars episode can be found.

This part is especially noteworthy because the data source implementation will receive this payload as in the configuration.

7. Implementing the data source code


You need to register the data source in the extension configuration to implement the data source code. This is usually the index.ts file in packages/<project>/backend.

import {
DataSourceConfiguration,
DataSourceContext,
DataSourceResult,
} from '@frontastic/extension-types';
export default {
'data-sources': {
'example/star-wars-movie': (
config: DataSourceConfiguration,
context: DataSourceContext
): DataSourceResult => {
return {
dataSourcePayload:
'We should fetch episode ' +
(config.configuration.episode || 'UNKNOWN'),
} as DataSourceResult;
},
},
};

A data source function is defined by its 2 parameters:

  • The configuration config (of type DataSourceConfiguration) and its context (of type DataSourceContext)
  • The return value of type DataSourceResult

As can be seen from the example, the config contains the configuration values from studio as specified earlier by the data source schema.

studio allows users to store configuration even if mandatory parameters are missing. So, you should always code defensive when using configuration values.

Repeating the test HTTP call will now reveal a data source result:

{
"data": {
"_type": "Frontastic\\Catwalk\\NextJsBundle\\Domain\\PageViewData",
"dataSources": {
"ee92bc20-f4b5-4e9d-98cf-b8c8d7300563": {
"dataSourcePayload": "We should fetch episode 4"
}
}
},
...
}

While the above example illustrates the types used with a data source, you wouldn't want to only return the data source configuration. So, here's an actual implementation of an API.

We're using the Axios library to perform HTTP requests here. To reproduce this example, you need to add this as a dependency, for example, using yarn add axios. You can use any HTTP library that works with Node.js, the native Node.js HTTP package, or an SDK library of an API provider.

import {
DataSourceConfiguration,
DataSourceContext,
DataSourceResult,
} from '@frontastic/extension-types';
import axios from 'axios';
export default {
'data-sources': {
'example/star-wars-movie': async (
config: DataSourceConfiguration,
context: DataSourceContext
): Promise<DataSourceResult> => {
return await axios
.post<DataSourceResult>(
'https://swapi-graphql.netlify.app/.netlify/functions/index',
{
query:
'{film(id:"' +
config.configuration.movieId +
'") {title, episodeID, openingCrawl, releaseDate}}',
}
)
.then((response): DataSourceResult => {
return {
dataSourcePayload: response.data,
} as DataSourceResult;
})
.catch((reason) => {
return {
dataSourcePayload: {
ok: false,
error: reason.toString(),
},
} as DataSourceResult;
});
},
},
};

This fetches the corresponding movie from the Star Wars API GraphQL example implementation and returns the corresponding payload as the result of the data source. Repeating the testing HTTP call will show it as part of the page payload:

{
"data": {
"_type": "Frontastic\\Catwalk\\NextJsBundle\\Domain\\PageViewData",
"dataSources": {
"ac8a8700-6a84-4c02-9160-4fc2fced3fe9": {
"dataSourcePayload": {
"data": {
"film": {
"title": "A New Hope",
"episodeID": 4,
"openingCrawl": "It is a period of civil war.\r\nRebel spaceships, striking\r\nfrom a hidden base, have won\r\ntheir first victory against\r\nthe evil Galactic Empire.\r\n\r\nDuring the battle, Rebel\r\nspies managed to steal secret\r\nplans to the Empire's\r\nultimate weapon, the DEATH\r\nSTAR, an armored space\r\nstation with enough power\r\nto destroy an entire planet.\r\n\r\nPursued by the Empire's\r\nsinister agents, Princess\r\nLeia races home aboard her\r\nstarship, custodian of the\r\nstolen plans that can save her\r\npeople and restore\r\nfreedom to the galaxy....",
"releaseDate": "1977-05-25"
}
}
}
}
}
}
...
}

8. Consume the data source


The API hub will automatically make the data source payload available through the data property to the Frontend component implementation:

{
"_type": "Frontastic\\Catwalk\\FrontendBundle\\Domain\\Tastic\\Configuration",
"mobile": true,
"tablet": true,
"desktop": true,
"movieData": {
"_type": "Frontastic\\Catwalk\\NextJsBundle\\Domain\\Api\\TasticFieldValue\\DataSourceReference",
"dataSourceId": "ee92bc20-f4b5-4e9d-98cf-b8c8d7300563",
"dataSource": {
"data": {
"film": {
"title": "A New Hope",
"episodeID": 4,
"openingCrawl": "It is a period of civil war.\r\nRebel spaceships, striking\r\nfrom a hidden base, have won\r\ntheir first victory against\r\nthe evil Galactic Empire.\r\n\r\nDuring the battle, Rebel\r\nspies managed to steal secret\r\nplans to the Empire's\r\nultimate weapon, the DEATH\r\nSTAR, an armored space\r\nstation with enough power\r\nto destroy an entire planet.\r\n\r\nPursued by the Empire's\r\nsinister agents, Princess\r\nLeia races home aboard her\r\nstarship, custodian of the\r\nstolen plans that can save her\r\npeople and restore\r\nfreedom to the galaxy....",
"releaseDate": "1977-05-25"
}
}
}
}
}
const StarWarsOpeningCrawl = ({ data }) => {
console.log(data);
return (
<marquee direction="up" style={{ whiteSpace: "pre-wrap" }}>
{data.movieData.dataSource.data.film.openingCrawl}
</marquee>
);
};

You have your opening crawl available on <http://localhost:3000/data-source>

2eeebf6 opening crawl

Further reading