LOC Features Overview

Learning Objective

• To understand features in LOC and their hierarchy.
• To understand logic types in LOC.
• To understand what are triggers, tasks and executions.
• To understand what are (data) events.

Terminologies: Features, Assets and Resources

In LOC, we'll refer various functionalities with the following terminologies:

Type	Description
Features	Accessible functionalities or asset types that can be created in LOC
Assets	Created instance of features that owned by users
Resources	Allocated hardware (CPU and memory) for LOC

Features that can be created as assets:

• Projects and scenarios
• Logic
• Data processes
• Triggers
• Agent configurations
• Tags

Features that cannot be created as assets:

• Agents
• Task context and payload
• SDK types

A special case of asset is the workspace (unit), which is defined by your LOC license along with available hardware resources.

Project, Scenario and Data Process Hierarchy

In LOC, the hierarchy of projects, scenarios, data processes and logic are arranged as follows:

Term	Description
Unit	represents the topmost LOC development workspace for all projects.
Project	represents a business logic which may contain several scenarios.
Scenario	represents a use case in a business logic and may contain several data processes.
Data process	represents a data pipeline that have several logic linked to it.
Logic	represents a code module which contains scripts to perform data-processing tasks.

A project owner owns all scenarios and data processes under it (but not the linked logic).

LOC asset naming rules

The name or any LOC assets cannot be blank, cannot exceed 128 characters and can only contain the following characters:

• spaces
• dot: .
• dash: -
• underscore: _
• sharp: #
• tilde: ~
• single quote: '
• parentheses: ()
• square bracket: []

Spaces at both ends will be trimmed. All empty string is not allowed.

Whereas descriptions or assets have no such restriction and can be blank.

By default the entrypoint file path (the filename of source logic) is the same as the logic name.

Data Process and Logic

Logic Types

A data process always has at least one generic logic and only one aggreggaror logic, which are the two main logic types in LOC.

• Generic logic is the general purpose logic for performing any desired extracting, transformation or loading actions.
• Aggregator logic is responsible for returning finalised results from an executed data process. It is the last logic to be executed in a data process. Ideally, generic logic should pass their results to the aggregator logic via internal session storage variables.

Logic are designed to be reusable, so they are created and deployed independent of data processes. A data process is in fact a manifest of some metadata and a series of linked logic's IDs.

info

Currently LOC supports two languages for developing logic, both available in Studio and CLI:

• JavaScript (ES6/ES2015 and above)
• TypeScript (version 3.7.0 and above)

You can read the following tutorials to learn more about how to create and use logic:

• Logic and Session
• Tips on Error Handling

Logic Modularisation

The same logic can be shared between more thah one data processes:

LOC thus llows you to create common, reusable logic modules to reduce duplicated code, logic coupling as well as development time. Of course, how and what to modularise your code really depends on use cases of your business and Data.

Cloud Logic vs. Native Logic

Depending on where the logic code is developed and deployed, both generic and aggregator logic stored in LOC Core can have two types - cloud and native logic:

These two types of logic essentially work in the same way with a few difference:

Logic type	Deployed from	Source code editable on the cloud	Support 3rd-party libraries	Suitable For
Cloud logic	Studio	Yes	No	Demostraion or proof of concept
Native logic	CLI	No	Possible	General development

info

Source code - or "entry file" - of both cloud and native logic can be viewed in Studio. However only cloud logic entry files are editable.

The only difference is that cloud logic would store both source code and compiled code directly in LOC, while native logic only uploads the compiled code (with the source code files or source logic remains on the developer's local machine).

Despite not being editable in Studio, native logic have the advantages of combining with local developmenting, testing and source control tools so that you can design and test-run logic before pushing them to production. The deployed assets can then be managed easily via Studio's graphic interface.

For demostration purposes we will still mostly use cloud logic in our tutorials. See CLI Handbook to learn more about CLI.

warning

For maintaining data security, LOC's logic runtime does not allow third party packages to make external connections. You have to use SDK's (agents) to access external data sources. Other than that, you can add local code libraries and import them into different logic.

Agents

Each logic can access a range of internal and external data source functionalities, which are integrated into the LOC runtime. These "agents" allow logic to share data with other logic or access HTTP endpoints, file servers and databases, making LOC data processes a truly powerful data integration tool between a sea of data silos and legacy systems.

During an execution, the session storage agent can access the session storage and the event agent can access the event store.

You can find full detailed interface documentation at SDK Reference.

Agent configurations

For four of the agents that can access external data sources, they require special setups - agent configurations - for defining the data sources like web servers and databases.

You can learn how to set and use agent configurations in the following tutorials:

• Send HTTP Requests Using Agent Configuration
• Create a Data-Driven Microservice with Database

Tags

Tags are metadata that can be added to data processes and logic. They are useful for improve the readability and searchability in both Studio and CLI.

Triggers, Tasks and Executions

Triggers are features for invoking one or more data processes.

Trigger type	Trigger message source	Can has payload	Return result
API route (HTTP endpoint)	HTTP request	Yes	Synchronous/asynchronous
Message queue	MQ client	Yes	Asynchronous
Scheduler	LOC scheduler	No	None (execution history only)

The list of data processes linked to a trigger can be referred as the trigger manifest.

When a trigger send messages to LOC runtime, it starts an execution. Each data processes invoked in the execution is run as a task. All tasks would receive the same payload from the trigger if there is one.

Both execution and task would generate an unique ID, and the execution result/logs would be stored as execution history.

note

Execution Result and Task Result

After the execution, a trigger may return the execution result. A result in JSON format would look like this:

{
    // execution metadata
    "_status": 200,
    "_metadata": {
        "executionId": "...",
        "triggerType": "ApiRoute",
        "triggerId": "...",
        "creationTimestamp": "...",
        "completionTimestamp": "...",
        "status": "success"
    },
    "data": {
        // task result(s)
        "task_result_1": {
            // ...
        },
        "task_result_2": {
            // ...
        }
    }
}

An execution result contains

• execution metadata (auto-generated by LOC):
  • HTTP status code and execution status
  • Trigger ID and type
  • Execution timestamps
• At least one task results returned from each task (executed data process), which are fully customisable by user's logic

By default a trigger - synchronous API route or message queue - returns both execution metadata and task result(s). You can have an API route to return task results only by turning the encapsulation off.

If possible, the task result would be sent directly back to the trigger. The execution response (combined with task results from each data process) will also be stored in the execution histories, which can be queried later if the trigger is asynchronous.

You can learn how to create an API route with the following tutorial:

• Create and Use an API Route Trigger

info

LOC Studio and CLI only supports managing API routes.

MQ and scheduler were once available to manage via CLI in v0.7.0, however since the new CLI is currently being ported from TypeScript to Rust, these functionalities will be available again in later releases.

Events

Events or data events are metadata emitted by users with Event Agent to the LOC event store, which is built on Elasticsearch.

Ideally, an event represents a business logic event and the metadata can be shared across different data processes, or to generate data lineage (which represents data flow between data processes).

You can read the following tutorials to learn more about using events:

• Emit and Inspect Events
• Events and Multiple Data Processes

In future releases events can even serve as triggers as well (invoke data processes with data events).

Other Active Metadata

Other than events, LOC offers some different types of automated logs:

• Build History (Building/compiling status logs of all cloud logic)
• Audit Logs (creation, execution and deletion of data processes) - currently not available in Studio v1.4.x