Parallel map

pyrula.parallel.par_map runs a function over a collection in forked worker processes on macOS and Linux. Each process has its own GIL, so CPU-bound Python runs in parallel. The callback and inputs are inherited through the fork; only results return.

from pyrula import FloatList
from pyrula.parallel import par_map

def price_option(strike):
    # stands in for heavy, pure numeric work
    return max(5_000.0 - strike, 0.0)

strikes = FloatList([float(s) for s in range(10_000)])
prices = par_map(price_option, strikes)   # FloatList in, FloatList out

Input and result types

The input type picks the path and the return type:

from pyrula import FloatList, IntList

par_map(lambda x: x * 2.0, FloatList([1.0, 2.0, 3.0]))      # -> FloatList
par_map(lambda n: n * n,   IntList([1, 2, 3]))              # -> IntList
par_map(lambda r: {"id": r["id"]}, [{"id": 1}, {"id": 2}])  # list -> list

FloatList and IntList results go through a shared-memory column. Anything else takes the object path, where each result is encoded with result_codec: "json" (default, JSON-native types only) or "pickle".

result_type selects the container instead, and excludes result_codec:

par_map(lambda x: x * 1.5, range(10_000), result_type=float)    # -> FloatList
par_map(lambda d: sha256(d).digest(), docs, result_type=bytes)  # -> BytesList
rows = par_map(build_row, ids, result_type=Row)                 # -> list[Row], validated

Row can be a Pydantic model or dataclass. Results are validated in the worker, and a failure becomes an element error. A dataclass must be defined at module level.

Per-worker setup

initializer runs once in each worker after the fork. Seed RNGs here, since forked workers share the parent’s state. If it raises, the call raises WorkerDiedError.

par_map(fn, data, initializer=lambda: random.seed(os.urandom(16)))

Errors

One bad element does not kill the batch. on_error picks how failures surface once it has run:

from pyrula.parallel import ParMapError

try:
    prices = par_map(price_option, strikes)      # "raise" (default)
except ParMapError as e:
    e.errors      # [(index, traceback), ...]
    e.partial     # results that succeeded

results = par_map(price_option, strikes, on_error="either")   # [(ok, value_or_traceback), ...]

Object-path results larger than the shared region (max(1 MiB, 128 bytes x n)) raise RegionOverflowError; pass a larger region_size.

Options

par_map(
    fn, data,
    workers=None,          # default min(8, cpu_count - 1)
    chunk_size=None,       # default max(4, n // (workers * 8))
    timeout=None,          # seconds, whole call; stragglers are killed
    on_error="raise",      # or "either"
    result_codec="json",   # or "pickle"
    result_type=None,
    region_size=None,
    initializer=None,
    initargs=(),
    unsafe_allow_threads=False,
)

Inputs under 32 elements, workers=1, or a single chunk run in-process, unless timeout is set (a timeout needs killable workers). timeout raises TimeoutError.

Callback state

fn runs in a forked child, so mutations to globals, closed-over variables or self are silently lost. Return anything the parent needs.

seen = []
par_map(lambda x: seen.append(x), data)   # `seen` stays empty in the parent

Scope

Windows raises NotImplementedError. par_map raises ForkSafetyError while other Python threads are alive, or when called inside a worker. unsafe_allow_threads=True overrides the thread check only if no live thread holds a lock a child may need. Each call forks new workers; there is no persistent pool.

Native threads (a pyrula-kafka client, a BLAS pool) are invisible to the guard, and a child that needs something they hold can hang. Set timeout when they are live.

Benchmarks

10,000 elements, 8 workers, 10-core machine, versus a serial comprehension (benchmarks/core/par_map.py):

Workload	Speedup
Heavy numeric fn (FloatList)	7.4x
Object path, heavy fn	7.1x
Validate + build model	6.8x
Trivial fn (~0.5 µs each)	0.36x

Trivial functions are dominated by fork overhead; use a comprehension.

Full signature and error types: Parallel API reference.