Skip to content
Pyrula

Pipelines

Generated from the type stubs and docstrings. Do not edit by hand.

Types

Section titled “Types”

BigQuerySink

Section titled “BigQuerySink”

Fields:

  • delivery = DeliveryGuarantee.AT_LEAST_ONCE

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

BigQuerySource

Section titled “BigQuerySource”

Watermark-incremental BigQuery source. Polls WHERE wm > ? ORDER BY wm LIMIT batch and checkpoints the last watermark under {"bigquery": {"watermark": ...}}. See WatermarkSource.

Fields:

  • NAMESPACE = 'bigquery'
  • PLACEHOLDER = '?'

DatabricksSink

Section titled “DatabricksSink”

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Any) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Any]

DatabricksSource

Section titled “DatabricksSource”

open

Section titled “open”
open(self) -> None

poll

Section titled “poll”
poll(self) -> Optional[tuple[Any, Any]]

DeliveryGuarantee

Section titled “DeliveryGuarantee”

Fields:

  • EXACTLY_ONCE = 'exactly_once'
  • AT_LEAST_ONCE = 'at_least_once'

DeltaSink

Section titled “DeltaSink”

Fields:

  • delivery = DeliveryGuarantee.EXACTLY_ONCE

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

DeltaSource

Section titled “DeltaSource”

open

Section titled “open”
open(self) -> None

seek

Section titled “seek”
seek(self, checkpoint: Checkpoint) -> None

poll

Section titled “poll”
poll(self) -> Optional[tuple[Any, Checkpoint]]

commit

Section titled “commit”
commit(self, checkpoint: Checkpoint) -> None

close

Section titled “close”
close(self) -> None

KafkaSink

Section titled “KafkaSink”

Fields:

  • delivery = DeliveryGuarantee.EXACTLY_ONCE

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

KafkaSource

Section titled “KafkaSource”

Fields:

  • skipped: int

open

Section titled “open”
open(self) -> None

seek

Section titled “seek”
seek(self, checkpoint: Checkpoint) -> None

poll

Section titled “poll”
poll(self) -> Optional[tuple[Any, Checkpoint]]

commit

Section titled “commit”
commit(self, checkpoint: Checkpoint) -> None

close

Section titled “close”
close(self) -> None

Pipeline

Section titled “Pipeline”

stop

Section titled “stop”
stop(self) -> None

run_once

Section titled “run_once”
run_once(self) -> int

run

Section titled “run”
run(self, max_batches: Optional[int] = None) -> int

Run until stop()/signal (service mode) or max_batches. Returns batches committed.

PipelineError

Section titled “PipelineError”

Pipeline halted. Carries the last successfully committed checkpoint.

Fields:

  • last_checkpoint = last_checkpoint

PostgresSink

Section titled “PostgresSink”

Fields:

  • delivery = DeliveryGuarantee.EXACTLY_ONCE

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

PostgresSource

Section titled “PostgresSource”

Watermark-incremental Postgres source. Polls WHERE wm > $1 ORDER BY wm LIMIT batch and checkpoints the last watermark under {"postgres": {"watermark": ...}}. See WatermarkSource.

Fields:

  • NAMESPACE = 'postgres'
  • PLACEHOLDER = '$1'

S3Sink

Section titled “S3Sink”

Fields:

  • delivery = DeliveryGuarantee.AT_LEAST_ONCE

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

Sink

Section titled “Sink”

Fields:

  • delivery: DeliveryGuarantee

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

EXACTLY_ONCE: commit data and checkpoint atomically - raising means nothing was committed. AT_LEAST_ONCE: commit data only.

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

SnowflakeSink

Section titled “SnowflakeSink”

Fields:

  • delivery = DeliveryGuarantee.EXACTLY_ONCE

open

Section titled “open”
open(self, schema: Any) -> None

write_batch

Section titled “write_batch”
write_batch(self, batch: Any, checkpoint: Checkpoint) -> None

resume_checkpoint

Section titled “resume_checkpoint”
resume_checkpoint(self) -> Optional[Checkpoint]

close

Section titled “close”
close(self) -> None

SnowflakeSource

Section titled “SnowflakeSource”

Watermark-incremental Snowflake source. Polls WHERE wm > ? ORDER BY wm LIMIT batch and checkpoints the last watermark under {"snowflake": {"watermark": ...}}. See WatermarkSource.

Fields:

  • NAMESPACE = 'snowflake'
  • PLACEHOLDER = '?'

Source

Section titled “Source”

open

Section titled “open”
open(self) -> None

poll

Section titled “poll”
poll(self) -> Optional[tuple[Any, Checkpoint]]

Next (arrow_batch, checkpoint) or None when nothing is available.

seek

Section titled “seek”
seek(self, checkpoint: Checkpoint) -> None

Position the source so the record AFTER checkpoint is next.

commit

Section titled “commit”
commit(self, checkpoint: Checkpoint) -> None

Source-side progress mark. Advisory for EXACTLY_ONCE sinks; the resume authority for AT_LEAST_ONCE sinks.

close

Section titled “close”
close(self) -> None

Functions

Section titled “Functions”

databricks_to_postgres

Section titled “databricks_to_postgres”
databricks_to_postgres(uc_table: str, dsn: str, pg_table: str, pipeline_id: str, client: Any = None, host: Optional[str] = None, token: Optional[str] = None, mode: str = 'append_only', columns: Optional[list[str]] = None, filters: Optional[list] = None, input_schema: Any = None, output_schema: Any = None, transform: Optional[Callable[[Any], Any]] = None, slow_path: bool = False, **pipeline_kwargs: Any) -> Pipeline

Build (not run) a UC->Postgres pipeline. filters is the Delta DNF tuple list.

delta_to_postgres

Section titled “delta_to_postgres”
delta_to_postgres(delta_uri: str, dsn: str, table: str, pipeline_id: str, filters: Optional[list] = None, columns: Optional[list[str]] = None, input_schema: Any = None, output_schema: Any = None, transform: Optional[Callable[[Any], Any]] = None, slow_path: bool = False, **pipeline_kwargs: Any) -> Pipeline

Build (not run) a version-incremental Delta→Postgres pipeline.

Pushdown knobs: filters: DNF tuple list forwarded to the Delta/pyarrow scan, e.g. [("region", "=", "us"), ("year", ">", 2020)]. Different from postgres_to_delta which takes a raw SQL string. columns: Column projection forwarded to the parquet read.

Schema enforcement: input_schema / output_schema: pydantic model, dataclass, or pa.Schema. Validated columnar (once per batch) before and after the transform.

kafka_to_delta

Section titled “kafka_to_delta”
kafka_to_delta(kafka_config: Any, table_uri: str, pipeline_id: str, value_format: ValueFormat = 'raw', schema: Any = None, storage_options: Optional[dict] = None, input_schema: Any = None, output_schema: Any = None, **pipeline_kwargs: Any) -> Pipeline

Build (not run) an exactly-once Kafka→Delta pipeline.

Kafka has no read pushdown (every message is delivered), so there is no filters=. input_schema/output_schema (pydantic model, dataclass, or pa.Schema) gate the transform’s I/O columnar; slow_path/transform ride through to Pipeline.

pipeline

Section titled “pipeline”
pipeline(source: Source, sink: Sink, view: View = 'arrow', **pipeline_kwargs: Any) -> Callable[[Callable[[Any], Any]], Pipeline]

Decorate a transform function into a ready-to-run Pipeline.

The function receives and returns an Arrow batch. view reserves a future zero-copy Polars path; only “arrow” is supported today.

polars_transform

Section titled “polars_transform”
polars_transform(fn: Callable[[Any], Any]) -> Callable[[Any], Any]

postgres_to_databricks

Section titled “postgres_to_databricks”
postgres_to_databricks(dsn: str, uc_table: str, table: str, watermark_column: str, pipeline_id: str, client: Any = None, host: Optional[str] = None, token: Optional[str] = None, filters: Optional[str] = None, columns: Optional[list[str]] = None, schema_mode: Optional[str] = None, input_schema: Any = None, output_schema: Any = None, transform: Optional[Callable[[Any], Any]] = None, slow_path: bool = False, **pipeline_kwargs: Any) -> Pipeline

Build (not run) a Postgres->UC pipeline. filters is a Postgres SQL fragment.

postgres_to_delta

Section titled “postgres_to_delta”
postgres_to_delta(dsn: str, table_uri: str, table: str, watermark_column: str, pipeline_id: str, filters: Optional[str] = None, columns: Optional[list[str]] = None, input_schema: Any = None, output_schema: Any = None, transform: Optional[Callable[[Any], Any]] = None, slow_path: bool = False, **pipeline_kwargs: Any) -> Pipeline

Build (not run) a watermark-incremental Postgres→Delta pipeline.

Pushdown knobs: filters: SQL fragment added to the WHERE clause, e.g. "region = 'us'". Operator-supplied config only - never accept untrusted input here. columns: Column projection forwarded to the ADBC query.

Schema enforcement: input_schema / output_schema: pydantic model, dataclass, or pa.Schema. Validated columnar (once per batch) before and after the transform.