Skip to contents

mizu_as_arrow() converts a data frame, typically one received over a channel or collected from a pool, into a struct-typed nanoarrow_array_stream, ready for nanoarrow::convert_array_stream(), arrow::as_arrow_table(), or any other Arrow C Data Interface consumer.

Usage

mizu_as_arrow(x)

Arguments

x

a data frame.

Value

A struct-typed nanoarrow_array_stream.

Details

Frames arrive over channels and pools alike. Arrow data sent at top level is staged as one frame region, and Arrow data nested inside a payload (a list element, a task argument, the task expression itself) crosses by reference, so the received object holds a view in place of the table. A collected pool result is that view, and this function converts it back explicitly. One caveat on sends: the conversion warnings (a uint64 past 2^63 - 1 becoming NA, an int32 or int64 minimum reading as NA) fire at send over a channel, but at a pool publish they fire on the worker and the submitter never sees them.

Columns still backed by their received shared-memory pages export zero-copy: the stream's buffers point directly into the shared region for attribute-free double, integer, integer64, and character views (character as large_string). This covers any column backed by a shared region (the frame's own columns, columns referencing another region, and received views assembled into a data frame), with one reference held per region. A region whose producer has exited affects only its own columns, which fall back to copying. Everything else (factors, Date, POSIXct and difftime columns, logical and raw vectors, and materialized or never-shared columns) converts per column through nanoarrow::as_nanoarrow_array(). The shared region stays mapped until the stream and every array produced from it are released.

Examples

ch <- mizu_channel(quote(mizu_send(ch, mizu_recv(ch))))
mizu_send(ch, arrow::arrow_table(data.frame(x = runif(1e5), s = "a")))
df <- mizu_recv(ch, timeout = 30)
stream <- mizu_as_arrow(df)
arrow::as_arrow_table(stream)
#> Table
#> 100000 rows x 2 columns
#> $x <double>
#> $s <large_string>
mizu_close(ch)