Sample data tactics for Folder Capture in OL Connect Automate

When creating flows in OL Connect Automate, you typically use sample data for development and testing. If a flow starts with a folder capture node, you might manually copy and paste one of your sample data files to that folder to trigger it. This can be time-consuming since you have to switch between the application and the file system. However, you can automate this process and trigger it directly from the flow editor. This tutorial will show you how to automate the copying of sample data files into flows using the folder capture node in OL Connect Automate.

Prerequisites:

  • Files and Folders package version 0.9.14 or later (@objectif-lune/node-red-contrib-files-and-folders). This package provides the functionality to manipulate files and folders in the flow.

The concept

The basic setup consists of two nodes: an Inject node and a File Operations node (from the Files and Folders package). The Inject node allows you to manually start a flow by clicking its button in the editor. The File Operations node is configured to copy a file from a source location to a target location.

The following image shows this flow. The File Operations node is configured with two fixed paths for copying a sample data file from it’s source location to the folder monitored by the Folder Capture node.

This setup demonstrates the basic concept, but it can be improved to handle various test files and utilize a base path (or workspace folder) from a global environment variable. The latter would simplify deploying the flow across different machines.

Multiple test scenarios

The two paths could also be configured in the Inject node, allowing the setup to be expanded with multiple Inject nodes. Each Inject node can trigger the flow with a different sample data file, such as files containing varying numbers of records (e.g., 1, 10, or 1,000 records), data for testing sorting and grouping, or even an incorrect file to test error handling.

The image below shows how the Inject node is configured with fixed paths for the msg.source and msg.target properties. You can add multiple Inject nodes, each referencing a different sample file. Use the optional name field to give each Inject node a custom label that describes the test scenario. The Inject nodes are linked to the same File Operations node.

Next, configure the File Operations node to copy the file from the path in msg.source to the path specified in msg.target.

While this setup allows you to test the flow with different sample files, it’s not the most efficient approach. At this stage the Inject nodes contain the full path to both the source and target files, as does the Folder Capture node. This makes it hard to relocate the flow to a different machine where the files might be located in a different place or on a different volume. In a scenario like this, all paths must be updated manually on the respective setup.

Let’s explore how to optimize this by using Global Environment Variables.

Global Environment Variables

In an OL Connect Automate Global environment variables allow you to store and manage configuration settings or reusable values outside of your flow, making your flows more flexible and easier to adapt to different environments (e.g., development, testing, production). These variables are accessible within the flow and can be used in nodes to replace hard-coded values.

To setup a Global Environment Variable:

  1. In the editor, click the menu in the upper right corner and choose Settings.
  2. Under User Settings, select Environment.
  3. Click the +add button to create a new entry.
  4. Enter the name and value for the variable.

In the example below, we’ve created a variable called ENV_WORKSPACE and set its value to C:\workspace, which serves as the base path for the working directory of our solution.

Utilizing Environment Variables

Environment Variables can be utilized in various ways. In our setup, we will leverage the JSONata expression language to dynamically construct paths for the source and target files. JSONata is a powerful language for formatting, rearranging, and combining data, and many nodes support setting variable values using JSONata expressions.

For this approach, we will configure the following:

  • Inject nodes: These nodes will provide the name of the sample data file in the msg.topic property.
  • File Operations node: This node will use JSONata to combine the environment variable with solution-specific sub-paths and the file name, creating the full paths for both the source and target files.

Setting up the Inject node

Since we will be constructing the dynamic path in the File Operations node, the Inject node only needs to provide the file name of the sample data file. In our example, this is stored in the msg.topic property.

JSONata in the File Operations node

To enable JSONata for the Source file and Target file fields, change the input field to “J expression” (see the red arrow in the image below). Then, enter the expression in the input field or click the action icon (three dots) at the end of the field to open the JSONata Expression editor. This editor gives you access to the JSONata function reference.

In our example, we used the $env() function to get the value of the environment variable created earlier, then added the rest of the path by using the string concatenation operator (&) with the name of the sample data file stored in msg.topic.

Note! So far the examples used backslashes in the file paths following the Microsoft Windows format. Since OL Connect Automate is built on Node.js, you can also use forward slashes. On Windows, both forward slashes (/) and backslashes (\) work as path separators. However, when using backslashes in expressions, they need to be escaped with another backslash. For this reason, it’s often easier and clearer to use forward slashes.

The expression for the Source file:

$env("ENV_WORKSPACE") & "/patterns/sample data/samples/" & topic

The expression for the Target file:

$env("ENV_WORKSPACE") & "/patterns/sample data/in/" & topic

Repeat these steps for the other Inject nodes.

For the Folder Capture node you can use the same approach and change the Folder property to use JSONata. The following shows the expression for the watched folder:

$env("ENV_WORKSPACE") & "/patterns/sample data/in"

Conclusion

In this article, we demonstrated how using one or more Inject nodes to trigger a flow with different variants of sample data files can greatly enhance the testability of your flows and streamline the flow creation process.

By leveraging Global Environment Variables and JSONata, this approach becomes even more efficient and flexible. Dynamically constructing file paths allows the flow to adapt to different environments, eliminating the need for manual path adjustments when deploying across multiple systems. This solution not only simplifies the workflow but also enhances testing and development, ensuring smoother operation across various environments.

Tip! Global Environment Variables are part of the flow filebut can also be stored outside the flow file by configuring them in the Settings.js file or an external .env file. To add Global Environment Variables to your Settings.js file, include the following snippet at the beginning of the file, before the module.exports = line:

process.env.ENV_WORKSPACE="C:\workspace"

The Settings.js file is located in the ProgramData folder of OL Connect Automate. The use of .env files (similar to .ini files in OL Connect Workflow) will be covered in a separate post.

Tip! Any node property can be set with an environment variable by setting its value to a string of the form ${ENV_VAR}. When the runtime loads the flows, it will substitute the value of that environment variable before passing it to the node.

This only works if it replaces the entire property – it cannot be used to substitute just part of the value. For example, it is not possible to use ${ENV_WORKSPACE}\in\invoice.xml.