Datacenter User Guide

Version 2.0 | Published November 02, 2023 ©

Filtering a Dataset

Introduction

Filtering can be used to process the incoming data before it is sent to the outputs, allowing flexible data manipulation. Unlike other data manipulation operations (scripting and flattening), filtering acts on the data schema rather than the payload. That is, filtering acts on an object representation rather than on its instance.

Info: When filtering a dataset, the incoming data must be a valid JSON object. When this is not the case, filtering cannot be applied to a dataset.

For example, given the incoming data below:

Incoming object
[
{
"id": "457e14f6-6244-4fb5-882a-eafc874ac384",
"label": "Soccer",
"UUID": "457e14f6-6244-4fb5-882a-eafc874ac384"
},
{
"id": "1f66f5cf-978e-4fc1-9c39-7f2bde73778c",
"label": "Basketball",
"UUID": "1f66f5cf-978e-4fc1-9c39-7f2bde73778c"
},
{
"id": "fb238641-7e4b-4578-b34f-bdd172be03bf,
"label": "Tennis",
"UUID": "fb238641-7e4b-4578-b34f-bdd172be03bf"
}
]

Filtering it to remove "id" and "UUID" results in the data held in the datacenter to be:

Incoming object
[
{
"label": "Soccer"
},
{
"label": "Basketball"
},
{
"label": "Tennis"
}
]

On subsequent data, should the array contain more entries and assuming the same filtering is applied, then the Datacenter only reads the property "label" for all entries.

Currently, the following providers support filtering:

These are the following topics:

Accessing the Filter Page

To access the filtering page, select a dataset of one of the supported providers, hover on an instance of a template and in the hamburger menu that appears, select Filter:

images/download/attachments/120745845/image-2023-8-17_12-11-29.png

This opens a dialog, where the "schema" (an abstract representation that describes your existing data format) is shown as an interactive tree.

images/download/attachments/120745845/image-2023-8-17_12-11-56.png

Interacting with the Tree: Filtering Nodes

The tree shown in the filtering dialog can be panned and zoomed: use your mouse's left-button to pan, and the mouse scroll wheel to zoom.

A node in the tree can be in one of the following three states:

State

Active (filtered in)

Inactive (filtered out)

Disabled


images/download/thumbnails/120745845/image-2023-8-17_12-18-1.png

images/download/thumbnails/120745845/image-2023-8-17_12-18-38.png

images/download/thumbnails/120745845/image-2023-8-17_12-19-9.png

  • Active: The node is added to the dataset data. If a node represents an object, its children are also added if no further filtering is applied, or the individual filtering on the children is considered.

  • Inactive: The node is not added to the dataset data. If a node represents an object, all its children is excluded from the dataset.

  • Disabled: No interaction is possible with the node.

When adding a new data source, by default, the incoming data is set to active. That is, all fields in the incoming object are added to the dataset data. To start filtering nodes in the tree, click on the nodes you wish to add to the dataset.

When clicking on a node, there are three possible scenarios:

  1. A leaf node that is directly connected to the root is clicked: a single node is added to the dataset.

    images/download/attachments/120745845/image-2023-8-17_12-25-11.png
  2. A leaf node that is a children of an object is clicked: the leaf node, its parents, and all other parents up to the root are added to the dataset.

    images/download/attachments/120745845/image-2023-8-17_12-27-5.png
  3. A node representing an object is clicked: the object node, its children, and its parents and all other parents up to the root are added to the dataset.

    images/download/attachments/120745845/image-2023-8-17_12-28-26.png

Once you start filtering a dataset, you can click on a node to toggle its state. That is, clicking on a node for the first time sets its state to active. Clicking on it again sets its state to inactive.

To further illustrate how filtering works, consider the following filtered tree:

images/download/attachments/120745845/image-2023-8-17_13-49-53.png

This selection results in the dataset to hold the following (flattened) values.

images/download/attachments/120745845/image-2023-8-17_13-50-51.png

Searching Nodes and Inspecting Sample Values

Nodes in the tree can be searched using the Search nodes field in the top-left of the dialog. Start typing a node name in the text field to see all possible matches. By selecting one of the results, the corresponding node briefly changes color and size.
images/download/attachments/120745845/image-2023-8-17_12-39-33.png

Note: Searching a node triggers highlighted animations on the tree and has no effect on the final dataset. That is, selecting a node name from the search text field does not add the node to the dataset output.

To get an idea of the incoming data, sample payload can be inspected for each node in the tree. To reveal the sample content of a node, hover on its name.

images/download/attachments/120745845/image-2023-8-17_12-33-7.png