Architecture

See the whole pipeline journey inside PostgreSQL.

Definition, schedule, execution, result, and history stay in one model. Teams can understand what happened to their data with familiar SQL tools. This architecture has evolved through more than a decade of operation across real organizations.

10+ yearsevolved on real DWH workloads

Database-nativeEngine and process ledger inside PostgreSQL

FDW-powereduniversal SQL access to sources

Runtime flow

Every run becomes a queryable record.

Pipeline definitions are expanded into concrete work before they execute. The runtime ledger then carries both planned state and execution history.

01 · Definecore.etlschedule + SQL template

02 · Plancore.plannercreate process row

03 · Materializecore.processrendered SQL + due time

04 · Executecore.handlerclaim + run work

05 · Observehistorystatus + metrics

Historical components

Proven components with clear responsibilities.

core.etl

Pipeline definition

Schedule, SQL template, parameters, and execution policy.

core.planner

Planning

Turns scheduled definitions into concrete process records.

core.process

Runtime ledger

Planned work, rendered SQL, status, history, and execution metrics.

core.handler

Execution

Claims and runs prepared SQL work asynchronously.

core.reviver

Recovery

Retries eligible failed work after a delay.

Universal data layer

One SQL layer across virtually any source.

PostgreSQL FDW hides connection protocols and exposes external data as tables. The ecosystem spans relational databases, NoSQL, Big Data, files, cloud services, geodata, and APIs, while the open interface makes new wrappers possible.

ECOSYSTEM

From Oracle to Parquet

PostgreSQL, Oracle, MySQL, SQL Server, MongoDB, Redis, Elasticsearch, BigQuery, Hadoop, Hive, CSV, Parquet, Excel, and many more.

EXTENSIBILITY

A new source becomes a new wrapper

FDW is a standard PostgreSQL extension API. When no ready connector exists, a source can join the same table and SQL model.

Reliability and operations

Built for daily warehouse operations, not a demonstration.

PARALLEL

Parallel execution

Multiple handlers use PostgreSQL resources to process independent pipelines concurrently.

RECOVERY

Retries and recovery

Failed work returns to processing, while reviver paths help pipelines complete after temporary failures and delays.

CONTROL

History and monitoring

Rendered SQL, status, runtime, counters, and notifications give operators a clear view of the warehouse.