Skip to main content
Version: LOC v0.7 (legacy)

Create and Deploy Data Process

CLI allows you to create data process projects locally and deploy them with console commands.

tip

If you are using a code editor like VS Code, you can open the folder of CLI workspace (where the CLI binary is at).

New Template Project​

Create a new data process (named myproject in this example) using either JavaScript or TypeScript template:

./loc new myproject

A sub directory named myproject would be create under the CLI workspace:

🔍  Check Folder Name: myproject...

🎉  Generating Data Process Template folder...

🔨  Building fresh packages...
✨  Successfully created Data Process Template in myproject.
tip

If you haven't install SDK, see Install SDK. You can also install SDK and other dependencies under any data process' own directory.

Project Structure​

The generated template project would look like this:

/[local workspace]
    /myproject
        /generic-logic
            1.js
            2.js
        aggregator.js
        api-route-config.yaml
        config.yaml
        package.yaml

The main difference is that TypeScript projects use .ts for logic and has tsconfig.json.

tip

If you need a quick "Hello World" example, please refer to Quick Start.

CLI itself does not offer support for source or version control but you can use Git to commit your projects. See Source Control with Git for some information.

Configuring Data Process​

Config File​

You can edit some important information of the data process in config.yaml:

config.yaml (JavaScript)
version: 0.1.0
name: <project name>>
description: description
tagNames:
    - default-tag-from-loc-cli
timeoutSeconds: 180
genericLogics:
    - name: generic-1
      file: 1.js
      tagNames:
          - default-tag-from-loc-cli
    - name: generic-2
      file: 2.js
      tagNames:
          - default-tag-from-loc-cli
aggregatorLogic:
    name: aggregator
    file: aggregator-logic.js
agentConfiguration:
    database: []
    http: []
    fileStorage: []
    mail: []
logicVariables:
    DATA_PROCESS_VERSION: v0.1.0

Logic​

A data process must have one aggregator logic and at least one generic logic. You can create new .js or .ts file (depending on your project) and add them to config.yaml.

The generic logic will be executed by the order specified in the config file.

Tags​

Data process and logic can have multiple tags:

tagNames:
    - tag-1-data-process
    - tag-2-data-process
# ...
genericLogics:
    - tagNames:
          - tag-1-generic-logic-1
          - tag-2-generic-logic-1

Aggregator logic can have tags as well:

# ...
aggregatorLogic:
    tagNames:
        - tag-1-aggregator
        - tag-2-aggregator
        # ...

See Tags commands for additional operations after deployment.

Agent Configuration​

You can add agent configurations (secrets) by names:

agentConfiguration:
    # ...
    database: ["db-config-1", "db-config-2", ...]
    http: ["http-config-1", "http-config-2", ...]
    # ...

See Deploy Agent Configuration for how to deploy them.

info

If you are using local simple runtime, see Setting Up Agent Configurations.

tip

Unlike in Studio you have to use a reference name for agent configurations, here you'll have use their original name in the config files!

CLI will look for deployed configuations and load them at runtime. See Deploy Agent Configuration for how to deploy configurations.

Logic Variables​

You can set environment variables under logicVariables:

# ...
logicVariables:
    answer-to-life-universe-everything: "42"
info

The value of a local variable must be a string.

Then it can be read in the logic using logic variable agent:

const answer = LogicVariable.get("answer-to-life-universe-everything");

Deploy Data Process​

After implementing all the code you need in logic (see tutorials for details), you can deploy the data process to LOC.

First make sure you have logged in:

./loc login

Deploy a JavaScript project:

./loc deploy myproject
note

Deploy the same project again will create a new data process with a different PID.

If any error is thrown during the compile process, the data process will not be deployed.

Data processes deployed from CLI will be synchronised in Studio, which can be also undeployed or deleted, but since the code are already compiled you will not be able to inspect or modify the logic code in Studio.

See Project and Data Process commands for additional operations.

Delete Deployed Data Process​

After any data process (including the same data process with different revisions) is deployed to LOC, you can quickly delete them with one command:

./loc delete myproject

CLI will look for the name "myproject" in a local file .deploy-info.json (generated when you deploy data processes) and delete all data processes with this name.

Sharing Code Between Data Processes​

Create Utility Libraries​

If you are developing multiple data processes in CLI and they depends on the same third party packages or your own helper functions, you can export them via a .js or .ts file in another directory.

For example, we often have to use TextDecoder to decode Uint8Array array to string. We can instead create a utils.js (if you are using JavaScript projects) with the following content:

/[CLI workspace]/utils/common.js
// shared helper function
const UTF8ArrToStr = (aBytes) => {
    const utf8decoder = new TextDecoder();
    return utf8decoder.decode(new Uint8Array(aBytes));
};

// export functions
export { UTF8ArrToStr };

Then import it into a project's logic:

/[CLI workspace]/<your_project>/generic-logic/<your_logic_filename>/1.js
...
// import UTF8ArrToStr
import { UTF8ArrToStr } from "../../utils/common";

export async function run(ctx) {
  const payload = await ctx.payload();
  const data = payload.http.request.data;

  const parsed = JSON.parse(UTF8ArrToStr(data));

  const name = parsed?.name;  // assuming there is a name field

  // ...
}

CLI will automatically includes /utils/common.js during the complie process.

If you want to share a third party NPM package, simply install them with npm i or yarn add in the CLI workspace and wrap them inside your exported functions.

Sharing TypeScript Types​

For TypeScript developers, you can also share type definitions between logic using a .ts module, which is convinent to apply types on JSON objects:

/[CLI workspace]/utils/common.ts
// shared helper function with types
const UTF8ArrToStr = (aBytes: number[]): string => {
    const utf8decoder = new TextDecoder();
    return utf8decoder.decode(new Uint8Array(aBytes));
};

// shared type
type UserName = {
    name: string;
};

// export functions
export { UTF8ArrToStr, UserName };

Then import it into a project's logic:

/[CLI workspace]/<your_project>/generic-logic/<your_logic_filename>.ts
import { ..., HttpPayload } from "@fstnetwork/loc-logic-sdk";
...
// import UTF8ArrToStr
import { UTF8ArrToStr, UserName } from "../../utils/common";

export async function run(ctx) {
  const payload = await ctx.payload() as { "http": HttpPayload };
  const data = payload.http.request.data;

  const parsed: UserName = JSON.parse(UTF8ArrToStr(data));

  const name = parsed.name;  // inferred as string

  // ...
}