End-To-End Use Case

This step-by-step tutorial is meant to be an introduction to using the XMPro platform. Completing it will give you a solid foundation to understand the more advanced concepts and detailed how-to guides. This tutorial will explain how to create and design a Data Stream, configure Stream Objects to ingest, analyze, transform, and perform actions on data. You will also learn how to set up a Recommendation to generate alerts based on rule logic, create and design an App, create Data Sources and Connections, and configure a simple Data Grid and Chart.

Use Case

Let's assume there is a power plant that uses a heat exchanger to keep the turbine cool and at the optimum temperature. The heat exchanger circulates water between the cooling tower and the heat exchanger to dissipate heat. To keep a proper circulation of liquid, there are three pumps [A, B, C] installed. Each Pump has a sensor that provides live data for Flow Rate (L/m) and Temperature (°C) using MQTT.

Unless the Pump is under maintenance the Flow Rate should be above 15000 L/m and Water temperature should be below 130°C .

Engineers should be alerted if the average flow rate falls below 250 L/s. If the average temperature starts to rise above 130°C then a critical level alert should be raised.

Engineers should be provided a view to check the history of pump telemetry, maintenance records, and reservoir level to enable them to take necessary action.

1. Design Data Streams with Real-Time Data Sources

The Use Case requires that we gather the Flow Rate and Temperature data from three pumps constantly, and pass it on to be analyzed and have actions performed on the data. We will achieve this with the use of Data Streams. A Data Stream is a visual representation of a flow of data. It is created through the Data Stream Designer.

To access the Data Stream Designer, log into your XMPro Account and press the button in the top-left corner of the screen and click on the Data Stream Designer item.

A Data Stream has four components:

1. Ingesting data through Listener Agents

2. Contextualizing sensor data/telemetry through Context Provider Agents

3. Analyzing and transforming data through Transformation and AI Agents

4. Performing actions or outputting data to other integrations through Recommendation and Action Agents

We will follow those four steps below.

Read & Wrangle Live and Context Data

In this section, we will simulate reading data from pump sensors and a metadata store, and combine the data together into a single flow.

To begin, we will need to create a new Data Stream. To create a Data Stream, follow the steps below:

1. Open the New Data Stream page from the left-hand menu.

2. Give the Data Stream a name. For example, "Pump Condition Monitoring"

3. Select the Type "Streaming". Data Streams of the Streaming type will run polling Agents at a set interval, for instance, every 10 seconds, whereas Recurrent Data Streams run on a customizable schedule, for instance, once a day at 12am. The recurring type only applies to polling-based Stream Objects, which we won't use in this example.

4. Select the category under which the Data Stream is to be added.

5. Feel free to load a suitable icon. If you do not, the default icon will be used. Sample icons can be found in the Icon Library.

6. Select a collection that will be used to publish Data Stream.

7. Enter a description to best describe the Data Stream.

8. Click on "Save".

In a production environment, Data Streams would integrate with external data emitters through Agents like OSIsoft PI or MQTT Listeners. However, for the sake of keeping the example simple, we won't be using any Agents that require an environment to be set up. Instead, we will be simulating the data with the Event Simulator, Calculated Field, and CSV Context Provider Agents.

To simulate the telemetry from the pumps, follow the steps below:

Drag into the canvas one of each of the following Agents:

You can search for the Agent in the search bar, and click and drag the Agent into the canvas to add it. An instance of an Agent added to the canvas is referred to as a Stream Object.

Once you have all four Stream Objects in the Data Stream canvas, rename them as follows:

• Event Simulator as "Simulate Pump Data"

• Calculated Field as "Add Pump Identifier"

• CSV as "Simulate Context Data for Assets"

• Join as "Contextualize Data"

To change the name of a Stream Object, click the text and edit it.

Your stream should end up looking like this:

Once you have renamed all four Stream Objects, connect them with arrows as follows:

1. "Simulate Pump Data" to "Add Pump Identifier"

2. "Add Pump Identifier" to "Contextualize Data" (first input)

3. "Simulate Context Data for Assets" to "Contextualize Data" (second input)

To connect two Stream Objects, click and drag the green rectangle (Output) at the bottom edge of the first Stream Object, move the cursor to the green rectangle on the left edge of the second Stream Object (Input).

Your connected Stream Objects should look like this:

Now we will configure the added Stream Objects. Save your Data Stream now and after every change to propagate the changes throughout the Data Stream.

Simulate Pump Data

We will need to simulate ingesting data about flow rate and temperature from sensors in the pumps. We can achieve this with the Event Simulator Agent. The "Simulate Pump Data" Event Simulator will constantly generate data defined by the Event Definitions at a rate defined by the Events per Second property.

Edit the "Simulate Pump Data" Stream Object and click the + button to the right of the Event Definition grid to add event definitions.

Add two event definitions as follows:

• Name: WaterTemperature Type: Range Minimum Value: 100 Maximum Value: 160 Spike Value: 0

• Name: FlowRate Type: Range Minimum Value: 14000 Maximum Value: 16000

  Spike Value: 0

Ignore the Spike Value and Generate Spike options, as they are not relevant to the current scenario.

Change the Events per Second to 1. Click "Apply" on the Simulate Pump Data configuration page. Then click "Save" on the Data Stream page.

Add Pump Identifier

We need to add a way to simulate having three different pumps. At the moment the data is not identified, so we will need to add a range of identifiers to the data. This can be achieved with the Calculated Field Agent. The "Add Pump Identifier" Calculated Field will add a "PumpId" field to the data generated with values "A", "B", and "C" for each subsequent row.

To configure the Stream Object, double click on "Add Pump Identifier" to open its configuration. Or, you can also highlight the Stream Object and click on the "Configure" option at the top of the Data Stream.

Keep "Append to Current" as the "Results Returned As" value. This will add the value calculated by the expression to each row instead of creating a new row with the identifier.

Click the + button to the right of the Expressions grid to add the following expression:

• Calculated Field: "PumpId" - The field won't exist yet in the dropdown, so you must enter it yourself. Expression: ReadingNo % 2 == 0 ? "A" : ReadingNo % 3 == 0 ? "B" : "C" Data Type: String

Press "Apply" on the PumpId expression and the Add Pump Identifier configuration pages, and press "Save" on the Data Stream page.

Simulate Context Data for Assets

There is often metadata associated with assets that is not part of the live data from the sensors. In this case, metadata includes whether the pump is currently under maintenance, the manufacturer, and the last service date. We must retrieve this data from elsewhere. In a production environment, this might be an SAP EAM system, but for this example, we can achieve this through the CSV Context Provider Agent.

Double-click on the "Simulate Context Data for Assets" Stream Object to open the configuration menu. You can also highlight the Stream Object and click on the "Configure" option at the top of the Data Stream.

Download the provided file. The contents of the file are below the download link.

PumpId,UnderMaintenance,Manufacturer,ServiceDate
A,FALSE,Bosch,2020-10-12
B,FALSE,Bosch,2020-08-06
C,FALSE,Bosch,2020-01-04

Then under Data check the Use Uploaded File? checkbox and upload the file into the CSV Context Provider. The CSV Definition will be automatically detected and filled.

Change UnderMaintenance from a String to a Boolean, using the options from the dropdown. Also change ServiceDate from a String to a DateTime, using the dropdown. Leave the Limit Rows, Filter Criteria, and Sort by properties as their default values.

When completed, press the "Apply" button at the top of the configuration, and then save the Data Stream.

Contextualize Data

The metadata about each pump needs to be appended to each row of sensor data received from the pumps. This can be achieved with the Join Agent.

The "Contextualize Data" Join will join together the data from the CSV Context Provider and the Calculated Field using the PumpId as the common field. Configure it as follows:

• Behavior: Context - we want to join some context data to our row.

• Context Endpoint: Right - we must tell the Stream Object which input has Context data. The Context Data is received by the Right endpoint, as shown in the image below.

• Select List: all fields except R_PumpId (as the same data will be in L_PumpId).

• Join Type: Inner Join.

• On: L_PumpId = R_PumpId

Press "Apply" on the Contextualize Data configuration page, and press "Save" on the Data Stream page.

Create Analytics and Calculations

In this section, we will add some analytics and calculations that will find exceptions, transform the units of the data and find the average level across 5 seconds.

Ignore Pumps Under Maintenance

We want to only pass data onward in the Stream if the current pump is not under maintenance. This can be achieved with the Filter Agent. To do this, drag in a Filter Agent and connect the "Contextualize Data" Join endpoint to the Filter.

Rename the Filter to "Ignore Pumps Under Maintenance", and save. Double-click on the Stream Object to open the configuration menu. Click on the + symbol to add a new rule for the filter. Select "Add Condition", and configure the Filter to have the logic R_UnderMaintenance Equals false.

The configuration and Data Stream should look like this:

There are two green outputs to the Filter Stream Object, the left output is where the data is output to when the filter is true. The right output is where the data is output to when the filter is false. The left True Output should be the output that you connect to the next Stream Object.

Press "Apply" on the Ignore Pumps Under Maintenance Data configuration page, and press "Save" on the Data Stream page.

Change Unit to L/s

The data from the Pump Data has different units than what we want to use - it is measured in L/m and we want the units to be in L/s. This can be solved with the Calculated Field Agent. To transform the data, drag in a Calculated Field Agent, and rename it to "Change Unit to L/s".

Connect the "Ignore Pumps Under maintenance" Filter endpoint to the Calculated Field and Save. Make sure you connect the left True Output of the "Ignore Pumps Under maintenance" Stream Object to the Calculated Field's input.

Configure the Calculated Field as follows:

• Calculated Field: L_FlowRate

• Expression: L_FlowRate / 60

• Data Type: Double

This will divide the flow rate by 60 to make the value in L/s instead of L/m.

Press "Apply" on the Change Unit to L/S configuration page, and press "Save" on the Data Stream page.

Average across 5 seconds

The Use Case requires that engineers should be alerted if the flow rate averaged over 5 seconds falls below 250 L/s, and if the temperature averaged over 5 seconds also starts to rise above 130°C then a critical level alert should be raised. This can be achieved with the Aggregate Agent.

To calculate the average temperature and flow rate over 5 seconds, drag in the Aggregate Agent and name it "Average across 5 seconds". Connect the "Change Unit to L/S" Calculated Field endpoint to the Aggregate Agent and save.

Configure the Aggregate Agent as follows:

• Attributes to group on: L_PumpId

• Aggregate:

  • Average (of) L_FlowRate (as) FlowRateAvg

  • Average (of) L_WaterTemperature (as) CoolantTemperatureAvg

• Unit: Second

• Size: 5

Press "Apply" on the Average across 5 seconds configuration page, and press "Save" on the Data Stream page.

Data Conversion

We want the data for the average flow rate to be in integer format to display it more easily. This can be achieved through the Data Conversion Agent. To do this, drag in a Data Conversion Agent and rename it to "Data Conversion". Connect the "Average across 5 seconds" endpoint to the Data Conversion Agent and press "Save" on the Data Stream page.

Configure the Data Conversion Agent with the following two rows (You may need to maximize the page again). Click on the + symbol to add each row:

First row:

• Input Column: "FlowRateAvg"

• Output Alias: "FlowRateAvg"

• Data Type: "Int64"

Second row:

• Input Column: "CoolantTemperatureAvg"

• Output Alias: "CoolantTemperatureAvg"

• Data Type: "Int64"

The input columns should already have FlowRateAvg and CoolantTemperatureAvg as options listed in the dropdown menu.

This will replace the FlowRateAvg and CoolantTemperatureAvg with values converted to Integer format.

Press "Apply" on the Data Conversion configuration page, and press "Save" on the Data Stream page.

Output Formatting and Action Integrations

In this section, we will integrate our Data Stream with the App Designer to trigger Recommendations and send data to Apps.

Run Recommendation

First, we want to trigger Recommendations with the data from the Data Stream. This can be achieved with the Run Recommendation Agent. To do this, drag in a Run Recommendation Agent and rename it to "Run Recommendation". Connect the "Data Conversion" endpoint to the Run Recommendation Agent and press "Save" on the Data Stream page

Configure the Run Recommendation Stream Object as follows:

• Url: the URL of the App Designer site

• Key: the App Designer Key.

• Output on first occurrence Only?: true

• Entity Identifier: L_PumpId - this is for the Recommendation to create separate Alerts for each Entity

• Columns To Return: Leave empty (Return all columns)

Otherwise, the App Designer URL and Key can be found by following these steps:

1. Open the App Designer in a new tab by clicking the "Waffle" button (a.ka. "App Launcher") in the top left corner of the page and clicking "App Designer".

2. Copy the App Designer URL from the browser's address bar and paste it into the Url field in the Run Recommendation configuration.

3. Click the "Settings" button in the top bar and click the "Copy" button to the right of the Integration Key and paste it in the Key field in the Run Recommendation configuration. You will only be able to see this if you have Admin access. If you do not have Admin access, you can ask an Admin to share the key with you.

This is how your Data Stream and configuration should look:

Press "Apply" on the Run Recommendation configuration page, and press "Save" on the Data Stream page.

XMPro App

We want to send data to an App to be displayed as a decision support dashboard for the engineers. This can be achieved through the XMPro App Agent.

Drag two XMPro App Agents onto the Data Stream and name them "Post Pump Overview" and "Post Pump Specifics". One will send an overview of the data for all pumps, and the other will send a large cached amount of data for each pump.

Now we run into a problem; we want to connect multiple agents to the same data. To solve this, drag a Broadcast Agent into the Stream, and rename it to "Broadcast". Disconnect the "Ignore Pumps Under Maintenance" input arrow and connect it to the new Broadcast Stream Object. You can disconnect the arrow by highlighting the arrow itself and clicking on the "Delete" button at the top of the Data Stream.

Alternatively, you can click on the green rectangle (input) on the "Ignore Pumps Under Maintenance" Stream Object, and drag the arrow to the green rectangle (input) of the "Broadcast" Stream Object. Connect the Broadcast endpoints to the two XMPro App Stream Objects and the "Ignore Pumps Under maintenance" Filter, as shown in the video below:

Press the "Save" button at the top of the Data Stream. Your Data Stream should now look like this:

Configure the "Post Pump Overview" Stream Object to store in cache and output only one row per pump as follows: (Follow the steps given for the Run Recommendation Agent above to get the Url and Key.)

• Url: the URL of the App Designer site

• Key: the App Designer Key.

• Cache Size: 1

• Replace Cache: false

• Cache Per Entity: true

• Entity Identifier: L_PumpId

• Primary Key: L_PumpId

Press "Apply" on the Post Pump Overview configuration page and press "Save" on the Data Stream page.

Configure the "Post Pump Specifics" Stream Object to cache and output 20 rows per pump as follows:

• Url: the URL of the App Designer site

• Key: the App Designer Key. For more detail on how to find the key in the Site Settings, see this article.

• Cache Size: 20

• Replace Cache: false

• Cache Per Entity: true

• Entity Identifier: L_PumpId

• Primary Key: L_PumpId and L_ReadingNo

Press "Apply" on the Post Pump Specifics configuration page and press "Save" on the Data Stream page.

Your Data Stream is now complete. To start the stream, click on the "Publish" button. To see the data flow at each Stream Object, press the "Live View" button and select all the Stream Objects. Alternatively, you can also select specific Stream Objects. For example, if you just want to see the data flowing through to the XMPro App, select the XMPro App Stream Object.

Troubleshooting the Data Stream

To see if data is flowing properly within the Data Stream, you will need to Publish the Data Stream. Before publishing, you want to make sure there are no errors in the configurations of the Stream Objects. Click on the "Integrity Check" option at the top of the Data Stream. If any errors are present, the Stream Object with the errors will turn red. Hovering over the Stream Object will show you the list of errors. Once these errors are fixed, you will need to run the Integrity Check again.

Once all Stream Objects have passed the Integrity Check, you can click on "Publish", then "Live View", on the top of the Data Stream. The Live Data will open on the side, and you can then click on "Select Views" to click on the Stream Objects you want to troubleshoot.

If data is displaying for the Stream Object, that means the Stream Object should be working correctly. If not, you can recheck your configuration values for the Stream Object. You can also check if you have a Stream Host running. There are also other ways that you can troubleshoot Data Streams.

2. Create Event Rules & Recommendations

The Use Case requires that engineers should be alerted if the flow rate averaged over 5 seconds falls below 250 L/s, furthermore, if the temperature averaged over 5 seconds also starts to rise above 130°C then a critical level alert should be raised. To achieve this we will use Recommendations.

Recommendations can be found in the App Designer. Open the App Designer in a new tab by clicking the button in the top left corner of the page and clicking "App Designer".

To access the Recommendation management section of the App Designer, click on the "Recommendations" button in the left menu and press the "Manage Recommendations" button on the page.

Create Event Rule and Rule Logic

To trigger the required Alerts, we will be creating a Recommendation with two Rules. First, create a Recommendation called "Pump Flow Threshold". The Data Stream should be the same "Pump Condition Monitoring" stream we created previously.

To create a new Recommendation, follow the steps below after navigating to the Recommendation management page:

1. Click "New".

2. Specify a name and category for your new Recommendation.

3. Choose a Data Stream to receive data from.

4. Click "Save".

This Rule will notify Engineers when the Flow Rate is lower than 250 L/s and give them instructions and resources to help resolve the issue.

Select the Enable Execution Order checkbox, since we want the more critical rule to override the medium rule. Create a Rule by pressing the + button to the right of the Rules list.

Give the Rule the following properties:

The Rule Logic determines whether a Recommendation Alert will be created on receiving data from the Data Stream.

Create recommendations & Triage Instructions

There should be some instructions for the engineers to follow to help resolve the issue when it occurs. This can be provided through the Triage Instructions.

Continue creating the Rule with the following properties: