Maps .f over the elements of .x on a pool and returns a map handle
immediately, just as mizu_submit() returns a task. mizu_collect() on the
handle waits for completion and returns the results in input order: a list by
default, or an atomic vector (or matrix) in the style of vapply() when
.template is given. mizu_cancel() stops it. as.promise() (or then()
directly) collects it through the event loop instead; see
mizu_collect_all_async().
Usage
mizu_map(
.pool,
.x,
.f,
...,
.template = NULL,
.n_chunks = NULL,
.seed = NULL,
.timeout = Inf,
.stream = FALSE,
.collect = "copy"
)Arguments
- .pool
a pool handle from
mizu_pool()ormizu_pool_attach(); inside a task, the evaluating worker's own handle frommizu_current_pool().- .x
a vector (atomic or list) to map over. Anything else is coerced with
as.list(), aslapply()does.- .f
a function (or, as
match.fun()accepts, its name) applied as.f(.x[[i]], ...). Serialized once with its enclosing environment. Keep that environment small, as with any cross-process map. Amizu_call()specification maps over a pool of any worker language; see the Cross-language maps section.- ...
further constant arguments to
.f, staged once. As inmizu_submit(), the formals ahead of...are dot-prefixed, so an argument name passed here never collides with them.- .template
NULLfor a list result, or avapply()-styleFUN.VALUE: an atomic vector template that each result must match.- .n_chunks
the morsel count of the map (its scheduling granularity), or
NULLfor the default. See the Granularity section.- .seed
NULL(default: no RNG guarantees, no cost), or a numeric vector of length 1 or 2 that derives reproducible per-element RNG streams:seed, orc(seed, offset)to shift every element's stream byoffsetpositions. See the Reproducible RNG section.- .timeout
seconds after which submission gives up, cancels the tasks already in, and returns the
mizu_timeoutsentinel (classc("mizu_timeout", "mizu_sentinel")).Inf(the default) waits indefinitely. Submission blocks only while an injection ring is full; the collection deadline ismizu_collect()'stimeout.- .stream
TRUEstreams slices of.xto workers under a sliding window instead of staging.xinto shared memory wholesale, bounding shared-memory use by the window times the slice size. See the Streaming maps section. A streaming map handle collects withmizu_collect()only (it cannot be promise-armed).- .collect
"copy"(the default) gathers template results into an owning R vector atmizu_collect()."view"instead returns an ALTREP view over the map's shared output area, with no gather copy, for pipelines that reduce it immediately. Requires an atomic.template. The view is copy-on-write: the first write materializes a private copy. It keeps the map region alive until released, and re-sending it through a channel or pool crosses as a full copy, not by reference. The choice is stored on the handle.
Value
A map handle (class "mizu_map"). mizu_collect() on it returns a
list of the results of .f in the order of .x, with names(.x)
reapplied. With .template, an atomic vector of type typeof(.template),
or an m * length(.x) matrix when length(.template) > 1. This is an
owning vector, or a copy-on-write view over the shared output area with
.collect = "view". On .timeout expiry mid-submission, mizu_map()
itself returns the mizu_timeout sentinel, and mizu_collect() returns it
on its own timeout expiry. A length-0 .x returns a handle whose collect
yields the empty result, as lapply() does, and creates no region or
tasks.
Details
Unlike a loop of per-element mizu_submit() calls, one mizu_map() call
serializes .f, the constant arguments in ..., and .x once. It submits
one runner task per live worker, and each worker sets up that map context
at most once. The remaining per-element cost is one R function call, as in
lapply(). Runners schedule themselves: they claim contiguous batches of
elements from a shared cursor in the map region and size each batch
adaptively toward a fixed time target. A trivial .f runs in large batches
with near-zero scheduling overhead, while an expensive or skewed .f settles
on fine claims that keep the workers balanced. An atomic, non-ALTREP .x
with no attributes beyond names travels as bare bytes: the workers slice
their batches directly from shared memory without deserializing .x, and no
worker holds more than a batch of it at a time.
Runners are ordinary pool tasks, stolen and balanced like any other work.
Worker death is detected and reported (see Errors), and every pool invariant
applies unchanged. Between batches, a runner also answers a pool-wide
doorbell. When another submitter's task arrives while every worker is busy
inside a map, one runner picks it up at its next batch boundary, so pickup
latency is bounded and independent of length(.x). A map runner picked up
this way is not executed on the spot. The helper moves it onto its own deque,
where the next free worker steals it and joins that map.
Granularity
Elements are claimed in morsels: contiguous ranges of max(1, min(n %/% (workers * 256), 256)) elements. The morsel is the granularity floor for
cancellation, help, and loss reporting. Morsels go to runners in adaptively
sized batches of consecutive morsels. .n_chunks overrides the morsel count
(giving min(length(.x), .n_chunks) morsels). There is no per-morsel shared
state, so .n_chunks = length(x) costs no extra memory and suits extremely
skewed element times. A map submitted while no worker is live queues a single
runner in the injection ring and runs when a worker joins. If no worker has
ever joined the pool, mizu_map() errors instead, as mizu_submit_call()
does. If the submitter's result slots are all occupied by outstanding tasks,
mizu_map() errors immediately, before it stages anything.
Templates
.template works like vapply()'s FUN.VALUE: every result must match its
type and length exactly, or coerce upward (logical -> integer -> double ->
complex, checked on the workers per element). A classed integer64 template
(bit64's layout) is also supported. Results must then be integer64 of the
template's length, since an int64 result never widens to double, and the
gathered vector or view keeps the class. Results are written directly into a
shared output area and gathered in one copy, so no result is serialized. With
.collect = "view", the gather copy is skipped as well, and the result is a
copy-on-write view over the shared output area itself, for pipelines that
reduce it immediately. A template of length m > 1 gathers an m * length(.x) matrix with the names of the template as row names, as vapply()
does. Character templates are assembled through the generic result path
instead, so their type checks surface at assembly rather than per element on
the workers. Large results of a consistent shape belong on the template path.
A runner's generic results accumulate on the worker and publish once, so
.template both caps worker memory and moves the values across processes
without serialization. For readers of mizu_pool_stats(): a map with large
generic results publishes them through the ordinary result path, and can
add a few result-slot spills per call even when the pool's slot_size is
right for its usual traffic.
Errors, timeout, and cleanup
An error raised by .f is signalled again in the caller as the original
condition, with a mizu_map_index field naming the failing element (the
lowest index among the elements that ran). Failure is fail-fast: the erroring
runner sets the map's shared cancel word before publishing its error, so
every other runner stops within about one batch instead of finishing the
remaining elements. If a worker dies mid-map, the map raises
mizu_error_worker_died (see mizu_error) with the lost elements in a
lost field, a two-column matrix of inclusive lo, hi ranges. Loss is
reported conservatively, per runner. A dead runner's results publish only at
the end, so everything it completed is reported as lost along with what it
was executing, and a lost element is never reported as completed. On
.timeout expiry, during submit or collect, the outstanding work is
cancelled and the mizu_timeout sentinel is returned, not raised. Executing
runners observe cancellation within about one batch (one element where .f
is expensive), independent of length(.x). The slot of a published but
uncollected result is released only when the dropped handle's finalizer runs
at the next garbage collection, and the map's staging region is deleted then
too. A later map absorbs these temporarily occupied slots by using fewer
runners.
Reproducible RNG
By default, nothing is guaranteed about random draws inside .f. The workers
seed lazily and independently, and the fast path pays nothing for the option.
.seed gives every element a reproducible L'Ecuyer-CMRG stream: element i
runs under the stream i jumps of 2^127 steps from the base state that
set.seed(.seed, "L'Ecuyer-CMRG") installs. The caller's own .Random.seed
is not touched, and the RNG state of each worker is saved and restored around
its batches. Because the streams are per-element, the results are identical
for any .n_chunks value, batch sizing, worker count, or steal order. With
c(seed, offset), element i runs under the stream of element i + offset
of an unoffset run, so a map split across runs or processes reproduces the
streams of one uninterrupted run.
Nested maps
mizu_map(mizu_current_pool(), ...) inside a task expression uses the handle
of the worker running it. Runner submissions push onto the worker's own
deque. The blocked collect executes its own runners while idle peers steal
the rest, so nested parallelism has little overhead. The first nested map on
a worker claims a submitter slot, so with the default max_submitters = 8
(one held by the controller) at most 7 workers can nest at the same time.
Raise max_submitters for wider nested fan-outs.
Cross-language maps
.f may be a mizu_call() specification instead of a function. This is how
you map over a pool whose workers are not R, such as one spawned with
mizu_py_launcher(). A spec .f always stages a shared map region: the
descriptor crosses in the interchange format, and each runner task carries a
region reference that any worker language reads. The map element fills the
spec's first positional argument (name kind) or binds as x (source kind),
and the spec's own constant arguments travel with it, so ... must be empty
with a spec .f. Constants and elements must be portable values (the
interchange subset documented in mizu_send()), and a non-portable one
raises mizu_error_not_portable at stage time. .template and .collect
work unchanged, with the results gathered (or view-wrapped) by the
submitter's own language, and a per-element error crosses with its element
index. .seed crosses as the language-neutral (seed, offset) pair and each
worker language derives its own per-element streams. Batching and steal-order
invariance holds within a worker language, but the draws are not identical
across languages. A spec map runs on a same-language pool too, subject to the
same portability rules.
Streaming maps
With .stream = TRUE, the map does not stage the whole of .x into shared
memory. It streams slices of .x to workers as they take work, and the
return value is unchanged. Fixed slices of .x ride ordinary chunk tasks
under a sliding submit/collect window of at most min(.n_chunks, 2 * live workers, free result slots) outstanding tasks, so shared-memory use is
bounded by the window times the slice size instead of the size of .x. With
the default chunk count that is roughly (2 * workers) / .n_chunks of the
serialized .x. The chunk count defaults to min(length(.x), 32 * live workers), and .n_chunks overrides it (.n_chunks = length(.x) is the
mirai-style extreme of one element per task). Everything else (result order,
.template and .collect, .seed invariance, the error behavior) is the
same as for the non-streaming map. Fail-fast latency grows from about one
adaptive morsel batch to about one chunk, because the outstanding work at any
time is bounded by the window, so the bound moves with .n_chunks. Fixed
chunks have two costs: the adaptive batch sizing of the morsel scheduler is
lost (raise .n_chunks to mitigate skew), and slices cross serialized, so
each chunk pays an ordinary submit's serialization instead of the
non-streaming map's serialization-free slicing. A streaming map with R
workers shares one description of the map with every worker, and each worker
reads that description at most once. A mizu_call() spec as .f streams
too, over workers of any language. Each chunk in the window then stages a
small shared region of its own (the slice plus the task that runs on it, one
runner task per chunk), removed as soon as the chunk completes. Shared-memory
use stays bounded by the window plus a small per-worker cache (up to 8 recent
slices per worker) rather than by the size of .x. Each chunk also writes
its results separately, so they cannot combine into one view: .collect = "view" errors on a streaming spec map, as does preparing one with
mizu_map_prepare().
Very large x
The serialized runner wrapper needs a little over 200 bytes of entry inline
budget, so pools created with slot_size = 256L (224-byte budget) fit it.
The exception is .seed: its 6-word RNG state pushes the wrapper to about
270 bytes. Seeded maps on such pools work but spill a region per runner, so
keep the default slot_size on pools meant for seeded maps. For a very large
.x, two paths avoid materializing it on every worker. If the shared-memory
use is acceptable, share it first: a zero-copy view received from a channel
or a pool result, or a mori::share()d vector, reduces to its 30-byte
identifier inside the staged descriptor, and workers read elements directly
off the shared pages with OS demand paging, so no worker copies any part of
.x (mizu_map() itself never calls mori). If .x must not be staged
wholesale at all, .stream = TRUE keeps only a window of slices in shared
memory; see the Streaming maps section.
As in lapply(), .x is indexed with [[ on the workers after an
as.list() coercion of anything that is not a plain vector, so a
data.frame maps over its columns and a factor over its elements.
Examples
p <- mizu_pool()
m <- mizu_map(p, 1:10, function(i) i * 2L)
mizu_collect(m)
#> [[1]]
#> [1] 2
#>
#> [[2]]
#> [1] 4
#>
#> [[3]]
#> [1] 6
#>
#> [[4]]
#> [1] 8
#>
#> [[5]]
#> [1] 10
#>
#> [[6]]
#> [1] 12
#>
#> [[7]]
#> [1] 14
#>
#> [[8]]
#> [1] 16
#>
#> [[9]]
#> [1] 18
#>
#> [[10]]
#> [1] 20
#>
m <- mizu_map(p, rnorm(1e5), abs, .template = numeric(1))
v <- mizu_collect(m)
m <- mizu_map(p, 1:4, function(i) rnorm(2), .seed = 42L)
mizu_collect(m)
#> [[1]]
#> [1] 1.11932846 -0.07617141
#>
#> [[2]]
#> [1] -0.2084809 -1.0341493
#>
#> [[3]]
#> [1] 0.001100034 1.763058291
#>
#> [[4]]
#> [1] 0.2262605 -0.4827515
#>
mizu_pool_stop(p)