Software and Architecture

What software are used in LOC and how is the system structured?

Architecture Overview

This is a simplified architecture, but it shows how developers can deploy data processes serverlessly which in turn invoked by day-to-day users. The data processes use agents to operate external data as well as emit data events as data lineages.

LOC Logic Runtime

The LOC runtime is responsible to execute logic defined in a data process and has its own dedicated resources.

Since LOC v1.0, LOC runtime supports running logic in different languages with swappable external runtimes.

Software Technologies

The majority of LOC are proprietary software built by FST Network's developer team, mostly using Rust, TypeScript and React. All LOC components (except CLI) will run as custom resource (CR) pods in a Kubernetes cluster.

However, we do recognize and relies on proven third party software and tools, most of them are open-sourced. Some notable ones include:

• Istio (gateway and service mesh)
• KrakenD (gateway for API routes)
• Apache Pulsar (trigger broker)
• PostgreSQL (internal storage)
• Redis (internal storage)
• Amazon S3, Minio or S3-compatible databases (data process code and payload storage)
• Elasticsearch (event store/data lineage)
• Kibana (user interface for Elasticsearch)
• Prometheus (performance/resource monitoring)
• Vault (secrets management and protection)
• Harbor (local image registry)
• Verdaccio (local NPM registry)

LOC SDKs are developed and published by FST Network, which are free to download locally.

Supported Logic Languages

A LOC logic can be written in one of the following programming languages:

• JavaScript
• TypeScript
• C# (C Sharp)

Supported Data Sources

LOC data processes can be triggered by the following ways:

• HTTP requests
• Message queue messages
• Scheduled jobs

LOC agents supports the following operating options (with more planned in each category):

• HTTP endpoints
• Databases:
  • MySQL
  • PostgreSQL
  • Microsoft SQL Server
  • Oracle Database
• File storages:
  • FTP
  • S3
  • SMB
• SMTP email servers

With more can be request by clients and partners.