Data Pipeline: Data Process and Logic

Data Process

Data processes are data pipelines created and run in LOC and are consisted of logic. You can browse them in Explorer (using the project hierarchy) or Flat Search View (as a list).

note

A data process is "deployed" once it is created. It would only be executed either invoked by triggers or to be run manually.

See Quick Start for how to create a data process and logic in Studio.

Edit a Data Processes

A data process can be edit for

• Modifying metadata (name, description, etc.) of the data process
• Adding or removing logic
• Updating a logic to another revision
• Adding or removing agent configuration and/or logic variable for a specific logic

Modifying an existing data process creates a new revision. If the data process is linked to at least one triggers, you have to update their linked revision.

You can edit tags without editing the data process:

You can delete a data process even though some logic are already linked to it.

warning

A logic, after being linked into a data process, can setup its own agent configuration and logic variable. If a referenced agent configuration has been deleted (missing), LOC will prevent you from executing the data process until the missing agent configuration is removed or fixed.

Manual Execution

A data process can be executed directly via Studio. See Manually Trigger the Data Process for details.

Logic

Logic are code modules to be linked into one or more data processes.

Depends on how and where the logic are created, there are two types of logic can be browsed: cloud logic and native logic. Only cloud logic are editable in Studio.

You can view the logic code (all its revisions and draft) by clicking the logic name:

The saved edited code of a logic before build is referred as the draft. You can create draft from any previous revisions.

You can save the draft by clicking the "✓" icon, or click the rocket (🚀) icon to build it:

The build status can be found in Build History in the next section.

note

A logic is "deployed" once it is created. It has to be linked into a data process to be executed with it.

A logic cannot be deleted if it is linked to any data process.

Edit a Cloud Logic

A cloud logic can be edited for

• Modifying metadata (name, description, etc.) of the logic
• modifying the source code

note

Native logic (deployed from local workspaces using CLI) cannot be edited in Studio. You have to re-build the logic with CLI's loc logic build command.

Modifying a logic creates a new revision.

You can edit tags without editing the logic.

Build History

The logic build history is the log of building/compiling status of all cloud logic:

If the logic did not compiled successfully, you can instect the error log:

note

The build logs of deleted logic won't be removed (displayed in non-bold names), although the source code won't be available for view either.