NEWS.md
mirai() without setting daemons now exits as soon as the parent process does rather than finish the task. For previous behaviour use: with(daemons(1L, dispatcher = FALSE, autoexit = NA), mirai(...)).daemon() defaults:autoexit default of TRUE now ensures daemons are terminated along with the parent process. Set to NA to retain the previous behaviour of having them automatically exit after completing any in-progress tasks.dispatcher now defaults to TRUE.daemons() to create local daemons now errors if performed within a mirai_map() call. This guards against excessive spawning of local processes on a single machine.cluster_config() to launch remote daemons via HPC resource managers for Slurm, SGE, Torque, PBS and LSF clusters.require_daemons() as a developer function that prompts the user to set daemons if not already set, with a clickable function link if the cli package is available.launch_remote() commands are now the same irrespective of the number of launches. This is as daemons now retrieve the next RNG stream from dispatcher rather than the rs argument to daemon().call_mirai_() is now removed.mirai() argument .timeout is upgraded to automatically cancel ongoing mirai upon timeout when using dispatcher (thanks @be-marc, @sebffischer #251).serial_config() now accepts vector arguments to register multiple custom serialization configurations. Argument vec is dropped as internal optimizations mean this option no longer needs to be set.host_url() is upgraded to return all local IP addresses (named by network interface), which provides a more comprehensive solution than just using a hostname.register_serial() to register serialization configurations for all daemons() calls (may be used by package authors as a convenience).on_daemon() which returns a logical value, whether or not evaluation is taking place within a mirai call on a daemon.daemons_set() which returns a logical value, whether or not daemons are set for a given compute profile.daemons() now supports initial synchronization exceeding 10s (between host/dispatcher/daemons). This is particularly relevant for HPC setups (thanks @sebffischer, #275)..compute, this argument has a new default of NULL, which continues to use the default profile (and hence should not result in any change in behaviour).stop_mirai() failing to interrupt in certain cases on non-Windows platforms, and more robust interruption if tools::SIGINT is supplied or passed through to the autoexit argument of daemon() (thanks @LennardLux, #240).daemons() dispatcher argument ‘process’, deprecated in mirai v2.1.0, is removed.ssh_config() argument ‘port’ is removed, with the tunnel port now inferred at the time of launch, and no longer set by the configuration.local_url() adds logical argument ‘tcp’ for easily constructing an automatic local TCP URL when setting daemons() for SSH tunnelling.as.promise() method for ‘mirai_map’ objects. This will resolve upon completion of the entire map operation.register_cluster() is removed as no longer required.parallel::makeCluster(type = "MIRAI") to create a ‘miraiCluster’.call_mirai() is now user-interruptible, consistent with all other functions in the package.call_mirai_() is hence redundant and now deprecated.with() method for daemons() now propagates “.compute” so that this does not need to be specified in functions such as mirai() within the with() clause.mirai() arguments ... and .args now accept environments containing variables beginning with a dot . (#207).$call objects are now stripped of ‘srcref’ attributes (thanks @lionel-, #218).daemons() calls that error due to the remote launcher no longer leave the compute profile set up (#237).daemons() now requires an explicit reset before providing revised settings for a compute profile, and will error otherwise.mirai_map() now errors if daemons have not yet been set (rather than warn and launch one local daemon).$stack.trace as a list of calls (with srcrefs removed) without deparsing to character strings.mirai_map() improvements:cli errors when collecting a ‘mirai_map’.daemons(NULL) not causing all daemons started with autoexit = FALSE to quit, regression introduced in mirai v2.0.0.mirai_map() collection option improvements:[.progress_cli] is no longer a separate option.[.stop] now reports the index number that errored.collect_mirai() replaces ‘…’ with an ‘options’ argument, to which collection options should be supplied as a character vector. This avoids non-standard evaluation in this function.daemon() now returns an integer exit code to indicate the reason for termination.nextcode() to provide a human-readable translation of the exit codes returned by daemon().everywhere() now returns a list of at least one mirai regardless of the number of actual connections.tcp:// or tls+tcp:// connection in all cases.daemons(dispatcher = TRUE) provides a new and more efficient architecture for dispatcher. This argument reverts to a logical value, although ‘process’ is still accepted and retains the previous behaviour of the v1 dispatcher.daemons() gains argument ‘serial’ to register serialization configurations when using dispatcher. These automatically apply to all daemons that connect.stop_mirai() is now able to cancel remote mirai tasks (when using dispatcher), returning a logical value indicating whether cancellation was successful.rlang::abort() custom metadata may now be accessed using $ on the ‘miraiError’ (thanks @James-G-Hill #173).status() using the new dispatcher is updated to provide more concise and insightful information.everywhere() updates:daemon() updates:TRUE when connecting to dispatcher and FALSE when connecting directly to host.status() to track connection and disconnection events.launch_local() and launch_remote() updates:launch_local() now returns the number of daemons launched rather than invisible NULL.collect_mirai() is now interruptible and takes a ‘…’ argument accepting the collection options provided to the ‘mirai_map’ [] method, such as .flat etc.ssh_config() simplified to take the argument ‘port’ instead of ‘host’. For SSH tunnelling, this is the port that will be used, and the hostname is now required to be ‘127.0.0.1’ (no longer accepting ‘localhost’).host_url() argument ‘ws’ is removed as a TCP URL is now always recommended (although websocket URLs are still supported).saisei() is defunct as no longer required, but still available for use with the old v1 dispatcher.daemons(dispatcher = "thread") (experimental threaded dispatcher) has been retired - as this was based on the old dispatcher architecture and future development will focus on the current design. Specifying ‘dispatcher = thread’ is defunct, but will point to ‘dispatcher = process’ for the time being.nanonext >= 1.4.0.daemons(dispatcher = "thread") implements threaded dispatcher (experimental), a faster and more efficient alternative to running dispatcher in a separate process.mirai_map() adds [.progress_cli] as an alternative progress indicator, using the cli package to show % complete and ETA.daemons() gains argument ‘force’ to control whether further calls reset previous settings for the same compute profile.daemon() gains argument ‘asyncdial’ to allow control of connection behaviour independently of what happens when the daemon exits.daemons():... arguments are now dropped instead of throwing an error. This allows ... containing unused arguments to be more easily passed from other functions.mirai_map():x[.stop, .progress].mirai::mirai_map(1:4, Sys.sleep)[.progress].dispatcher() drops argument ‘asyncdial’ as it is rarely useful to set this here.everywhere() now errors if the specified compute profile is not yet set up, rather than fail silently.launch_local() and launch_remote() now strictly require daemons to be set, and will error otherwise.serial_config() now validates the arguments provided and returns them as a list. This means any saved configurations from previous package versions must be re-generated.daemons() to correctly handle a vector of URLs passed to ‘url’ again.mirai_map()[.flat] assigning a variable ‘typ’ to the calling environment.mirai(), mirai_map() and the promises method.nanonext >= 1.3.0.everywhere() adds argument ‘.serial’ to accept serialization configurations created by serial_config(). These allow normally non-exportable reference objects such as Arrow Tables or torch tensors to be used seamlessly across parallel processes without additional marshalling steps. Configurations apply on a per compute profile basis.serialization() is now deprecated in favour of the above usage of everywhere(), and will be removed in a future version.mirai_map() enhanced to perform multiple map over 2D lists/vectors, allowing advanced patterns such as mapping over the rows of a dataframe or matrix.[] method gains the option [.flat] to collect and flatten results, avoiding coercion.daemons(dispatcher = FALSE) and make_cluster() by using asynchronous launches (thanks @mtmorgan #123).everywhere() not reaching all daemons when using dispatcher.stats and utils base packages.nanonext >= 1.2.0.serialization() function signature and return value slightly modified for clarity. Successful registration / cancellation messages are no longer printed to the console.dispatcher() argument ‘retry’ now defaults to FALSE for consistency with non-dispatcher behaviour.remote_config() gains argument ‘quote’ to control whether or not to quote the daemon launch command, and now works with Slurm (thanks @michaelmayer2 #119).nanonext >= 1.1.1.mirai_map() for asynchronous parallel/distributed map using mirai, with promises integration. Allows recovery from partial failure or else early stopping, together with optional progress reporting.x[] collects the results of a mirai_map x, waiting for all asynchronous operations to complete.x[.progress] collects the results whilst showing a text progress bar.x[.stop] collects the results applying early-stopping, which stops at the first error, and aborts remaining in-progress operations.x[] as a more efficient equivalent of the interruptible call_mirai_(x)$data.collect_mirai() as a more efficient equivalent of the non-interruptible call_mirai(x)$data.unresolved(), call_mirai(), collect_mirai() and stop_mirai() now accept a list of ‘mirai’ such as that returned by mirai_map().daemons() with new settings when the compute profile is already set now implicitly resets daemons before applying the new settings instead of silently doing nothing.daemons() as automatic re-tries are no longer performed for non-dispatcher daemons.dispatcher() governs whether to auto-retry in the dispatcher case.nanonext >= 1.1.0.mirai() and everywhere():environment() to be supplied to ‘.args’ or as the only element of ‘…’.mirai(x, x = 1).ssh_config() improvements:stop_mirai() now resolves to an ‘errorValue’ 20 (operation canceled) in the case the asynchronous task was still ongoing (thanks @jcheng5 #110).[ method for ‘miraiCluster’ no longer produces a ‘miraiCluster’ object (thanks @HenrikBengtsson #83).parallel package is now only loaded when first used.nanonext >= 1.0.0.mirai() and everywhere() behaviour changed such that ‘…’ args are now assigned to the global environment of the daemon process.with() method for mirai daemons, allowing for example: with(daemons(4), {expr}), where the daemons last for the duration of ‘expr’.register_cluster() for registering ‘miraiCluster’ as a parallel Cluster type (requires R >= 4.4).is.promising() method for ‘mirai’ for the promises package.$stack.trace, and includes the trailing line break for consistency with ‘as.character.error()’.serialization adds arguments ‘class’ and ‘vec’ for custom serialisation of all reference object types.ssh_config() argument ‘timeout’ widened to 10 (seconds).daemons() specifying ‘output = FALSE’ registering as TRUE instead.everywhere() specifying ‘.args’ as an unnamed list or ‘.expr’ as a language object.local_url() helper to construct a random inter-process communications URL for local daemons (thanks @noamross #90).daemon() argument ‘autoexit’ now accepts a signal value such as tools::SIGINT in order to raise it upon exit.daemon() now records the state of initial global environment objects (e.g. those created in .Rprofile) for cleanup purposes (thanks @noamross #91).as.promise() method for ‘mirai’.serialization() for registering custom serialization and unserialization functions when using daemons.call_mirai_(), a user-interruptible version of call_mirai() suitable for interactive use.mirai() interface:everywhere() now returns invisible NULL in the case the specified compute profile is not set up, rather than error.mirai() specifying a timeout when daemons() has not been set - the timeout begins immediately rather than after the ephemeral daemon has connected - please factor in a small amount of time for the daemon to launch.make_cluster() now prints daemon launch commands where ‘url’ is specified without ‘remote’ whether or not interactive.make_cluster() specifying only ‘url’ now succeeds with implied ‘n’ of one.mirai() specifying a language object by name for ‘.expr’ in R versions 4.0 and earlier.%...>% with a ‘mirai’.everywhere() evaluates an expression ‘everywhere’ on all connected daemons for a compute profile. Resulting changes to the global environment, loaded pacakges or options are persisted regardless of the ‘cleanup’ setting (request by @krlmlr #80).host_url() implemented as a helper function to automatically construct the host URL using the computer’s hostname.daemon() adds argument ‘autoexit’, which replaces ‘asyncdial’, to govern persistence settings for a daemon. A daemon can now survive a host session and re-connect to another one (request by @krlmlr #81).daemons(NULL) implemented as a variant of daemons(0) which also sends exit signals to connected persistent daemons.dispatcher() argument ‘lock’ removed as this is now applied in all cases to prevent more than one daemon dialling into a dispatcher URL at any one time.daemon() argument ‘cleanup’ simplified to a logical argument, with more granular control offered by the existing integer bitmask (thanks @krlmlr #79).ssh_config() specifying tunnelling to the ‘remote’ argument of daemons().%>>%.make_cluster() creates a ‘miraiCluster’, compatible with all existing functions taking a ‘cluster’ object, for example in the ‘parallel’ and ‘doParallel’ / ‘foreach’ packages.status() can now take a ‘miraiCluster’ as the argument to query its connection status.launch_remote() improvements:daemons() gains the following features:launch_remote().remote_config() added to generate configurations for directly launching remote daemons, and can be supplied directly to a ‘remote’ argument.ssh_config() added as a convenience method to generate launch configurations using SSH, including SSH tunnelling.mirai() adds logical argument ‘.signal’ for whether to signal the condition variable within the compute profile upon resolution of the ‘mirai’.daemon() argument ‘exitlinger’ retired as daemons now synchronise with the host/dispatcher and exit as soon as possible (although a default ‘exitlinger’ period still applies to ephemeral daemons).daemons() or dispatcher() now raise an error early rather than attempting to launch daemons that fail.status() if there is no longer a connection to dispatcher.daemons(), dispatcher() and daemon() to clearly delineate core vs peripheral options.%>>% in favour of a recommendation to use package mirai.promises for performing side effects upon ‘mirai’ resolution.server() for daemon() is retired.daemons() gains the new argument ‘seed’ to set a random seed for generating these streams.daemon() and dispatcher() gain the argument ‘rs’ which takes a L’Ecuyer-CMRG random seed.nextstream() and nextget(), opening interfaces for packages which extend mirai.--vanilla flags for efficiency, avoiding lengthy startup configurations (thanks @alexpiper).status() occasionally failing to communicate with dispatcher.saisei():status():saisei() or launch_local().saisei() no longer resets the counter but instead turns it negative until a new daemon connects.launch_remote().daemons() and dispatcher() gain the argument ‘pass’ to support password-protected private keys when supplying TLS credentials (thanks @wlandau #76).daemons(), launch_local() or launch_remote() now works correctly as documented.tls+tcp:// or wss:// URL in daemons(). Single-use keys and certificates are automatically generated.daemons() and daemon(), such as those generated via a Ceritficate Signing Request (CSR) to a Certificate Authority (CA).launch_remote() launches daemons on remote machines and/or returns the shell command for launching daemons as a character vector.launch_remote("ws://192.168.0.1:5555", command = "ssh", args = c("-p 22 192.168.0.2", .).daemon() replaces server(), which is deprecated (although currently retained as an alias).launch_local() replaces launch_server() and now accepts a vector argument for ‘url’ as well as numeric values to select the relevant dispatcher or host URL, returning invisible NULL instead of an integer value.status() now retrieves connections and daemons status, replacing the call to daemons() with no arguments (which is deprecated). The return value of $daemons is now always the host URL when not using dispatcher.output=TRUE for daemons() or launch_local(). daemon() accepts a new ‘output’ argument.saisei() argument validation now happens prior to sending a request to dispatcher rather than on dispatcher.saisei(i = 1L) is called.daemons() status requests have been rendered more robust.saisei() always returns NULL if ‘tcp://’ URLs are being used as they do not support tokens.server() and dispatcher() argument ‘asyncdial’ is now FALSE by default, causing these functions to exit if a connection is not immediately available. This means that for distributed computing purposes, daemons() should be called before server() is launched on remote resources, or else server(asyncdial = TRUE) allows servers to wait for a connection.launch_server() now parses the passed URL for correctness before attempting to launch a server, producing an error if not valid.%>>% gains the following enhancements:.() implemented to wrap a piped expression, ensuring return of either an ‘unresolvedExpr’ or ‘resolvedExpr’.unresolved() in the same way as a ‘mirai’.mirai() gains the following enhancements (thanks @HenrikBengtsson):dispatcher() gains the following arguments:token for appending a unique token to each URL the dispatcher listens at.lock for locking sockets to prevent more than one server connecting at a unique URL.saisei() implemented to regenerate the token used by a given dispatcher socket.launch_server() replaces launch() for launching local instances, with a simpler interface directly mapping to server().mirai() with timeouts and no connection to a server.dispatcher() re-implemented using an innovative non-polling design. Efficient process consumes zero processor usage when idle and features significantly higher throughput and lower latency.launch() exported as a utility for easily re-launching daemons that have timed out, for instance.... variables in the daemons() call..compute argument in daemons() and mirai() for sending tasks with heterogeneous compute requirements.daemons() interface has a new url argument along with dispatcher for using a background dispatcher process to ensure optimal FIFO task scheduling (now the default).:0 will automatically assign a free ephemeral port, with the actual port number subsequently reported by daemons().daemons("view") functionality.dispatcher() is implemented as a new function for the dispatcher.server() gains the following arguments:asyncdial to specify how the server dials into the client.maxtasks for specifying a maximum number of tasks before exiting.idletime for specifying an idle time, since completion of the last task before exiting.walltime for specifying a soft walltime before exiting.timerstart for specifying a minimum number of task completions before starting timers.daemons() and server() now error and return immediately instead of potentially causing a hang.eval_mirai() is removed as an alias for mirai().--vanilla argument to enable site / user profile and environment files to be read.call_mirai() again (regression in 0.7.0), now returning a ‘miraiInterrupt’.is_mirai_interrupt() to test if an object is a ‘miraiInterrupt’.daemons() now takes ‘n’ and ‘.url’ arguments. ‘.url’ is an optional client URL allowing mirai tasks to be distributed across the network. Compatibility with existing interface is retained.server() is exported for creating daemon / ephemeral processes on network resources.daemons() and %>>% deferred evaluation pipe.mirai() gains a ‘.args’ argument for passing a list of objects already in the calling environment, allowing for example mirai(func(x, y, z), .args = list(x, y, z)) rather than having to specify mirai(func(x, y, z), x = x, y = y, z = z).is_mirai_error() should be used in place of is_nul_byte(), which is no longer re-exported.is_error_value() can be used to test for all errors, including timeouts where the ‘.timeout’ argument has been used.%>>% deferred evaluation pipe.mirai() to ensure a mirai always resolves even if the child process crashes etc.is_mirai() to test if an object is a mirai.$data to optimally align with the underlying implementation.mirai() added as an alias for eval_mirai(); supports evaluating arbitrary length expressions wrapped in {}.call_mirai(). Access the $value directly and an ‘unresolved’ logical NA will be returned if the async operation is yet to complete.stop_mirai() added as a function to stop evaluation of an ongoing async operation.is_nul_byte() and unresolved() re-exported from {nanonext} to test for evaluation errors and resolution of a ‘mirai’ respectively.daemons() interface to set and manage persistent background processes for receiving ‘mirai’ requests.