R/BatchtoolsFutureBackend-class.R, R/batchtools_custom.R, R/batchtools_interactive.R, and 2 more
BatchtoolsFutureBackend.RdA batchtools local future is an synchronous uniprocess future that will be evaluated in a background R session. A batchtools interactive future is an synchronous uniprocess future that will be evaluated in the current R session (and variables will be assigned to the calling environment rather than to a local one). Both types of futures will block until the futures are resolved.
A batchtools SSH future is an asynchronous multiprocess
future that will be evaluated in a background R session.
We highly recommend using future::multisession
(sic!) futures of the future package instead of
SSH batchtools futures.
BatchtoolsFutureBackend(
workers = 1L,
resources = list(),
finalize = getOption("future.finalize", TRUE),
cluster.functions = NULL,
registry = list(),
conf.file = findConfFile(),
interrupts = TRUE,
delete = getOption("future.batchtools.debug", "on-success"),
...
)
BatchtoolsCustomFutureBackend(...)
BatchtoolsInteractiveFutureBackend(fs.latency = 0, ...)
BatchtoolsLocalFutureBackend(fs.latency = 0, ...)
BatchtoolsSSHFutureBackend(workers = availableWorkers(), fs.latency = 65, ...)(optional) The maximum number of workers the batchtools
backend may use at any time. Interactive and "local" backends can only
process one future at the time (workers = 1), whereas HPC backends,
where futures are resolved via separate jobs on a scheduler, can have
multiple workers. In the latter, the default is workers = NULL, which
will resolve to getOption("future.batchtools.workers", 100).
(optional) A named list passed to the batchtools
job-script template as variable resources. This is based on how
batchtools::submitJobs() works, with the exception for specially
reserved names defined by the future.batchtools package;
resources[["details"]], if TRUE, results in the job script outputting
job details and job summaries at the beginning and at the end.
resources[["startup"]] and resources[["shutdown"]] are character
vectors of shell code to be injected to the job script as-is.
resources[["modules"]] is character vector of Linux environment
modules to be loaded.
resources[["envs"]], is an optional names character vector specifying
environment variables to be set.
resources[["rscript"]] is an optional character vector specifying
how the 'Rscript' is launched. The resources[["rscript_args"]] field
is an optional character vector specifying the 'Rscript' command-line
arguments.
resources[["asis"]] is a character vector that are passed as-is to
the job script and are injected as job resource declarations.
All remaining resources named elements are injected as named resource
specification for the scheduler.
If TRUE, a future's batchtools Registry is automatically deleted when the future is garbage collected, otherwise not.
(optional) Assigned as-is to the each future's batchtools Registry.
(optional) A named list of settings applied to each
future's batchtools Registry.
This is a more convenient alternative to using argument conf.file.
(optional) A "batchtools-configuration" R script, which
is sourced when each future's batchtools
Registry is created. Any variables
created by this script is assigned to the registry.
The default file is the one found by batchtools::findConfFile(), if any.
If FALSE, attempts to interrupt futures will not take place on this backend, even if the backend supports it. This is useful when, for instance, it takes a long time to interrupt a future.
Controls if and when the batchtools job registry folder is
deleted.
If "on-success" (default), it is deleted if the future was resolved
successfully and the expression did not produce an error.
If "never", then it is never deleted.
If "always", then it is always deleted.
[numeric(1)]
Expected maximum latency of the file system, in seconds.
Set to a positive number for network file systems like NFS which enables more robust (but also more expensive) mechanisms to
access files and directories.
Usually safe to set to 0 to disable the heuristic, e.g. if you are working on a local file system.
Additional arguments passed to BatchtoolsFutureBackend().
A future::FutureBackend object of class BatchtoolsFutureBackend
An object of class BatchtoolsFuture.
An object of class BatchtoolsUniprocessFuture.
An object of class BatchtoolsMulticoreFuture.
batchtools local futures rely on the batchtools backend set up by
batchtools::makeClusterFunctionsInteractive(external = TRUE)
and batchtools interactive futures on the one set up by
batchtools::makeClusterFunctionsInteractive().
These are supported by all operating systems.
An alternative to batchtools local futures is to use
cluster futures of the future
package with a single local background session, i.e.
plan(cluster, workers = "localhost").
An alternative to batchtools interactive futures is to use
plan(sequential, split = TRUE) futures of the future package.
batchtools SSH futures rely on the batchtools backend set
up by batchtools::makeClusterFunctionsSSH().
The batchtools SSH backend only works on operating systems
supporting the ssh and ps command-line tool, e.g. Linux and macOS.
library(future)
## Create custom cluster functions (here same as "local")
cf <- batchtools::makeClusterFunctionsInteractive(external = TRUE)
print(cf)
#> ClusterFunctions for mode: Interactive
#> List queued Jobs : FALSE
#> List running Jobs: FALSE
#> Kill Jobs : FALSE
#> Hooks : -
str(cf)
#> List of 11
#> $ name : chr "Interactive"
#> $ submitJob :function (reg, jc)
#> $ killJob : NULL
#> $ listJobsQueued : NULL
#> $ listJobsRunning : NULL
#> $ array.var : chr NA
#> $ store.job.collection: logi TRUE
#> $ store.job.files : logi FALSE
#> $ scheduler.latency : num 0
#> $ fs.latency : num 0
#> $ hooks : list()
#> - attr(*, "class")= chr "ClusterFunctions"
# Use custom batchtools backend
plan(future.batchtools::batchtools_custom, cluster.functions = cf)
print(plan())
#> batchtools_custom:
#> - args: function (..., cluster.functions = list(name = "Interactive", submitJob = function (reg, jc) { assertRegistry(reg, writeable = TRUE) assertClass(jc, "JobCollection") if (external) { runOSCommand(Rscript(), sprintf("-e \"batchtools::doJobCollection('%s', output = '%s')\"", jc$uri, jc$log.file)) } else { doJobCollection(jc, output = jc$log.file) } makeSubmitJobResult(status = 0L, batch.id = "cfInteractive") }, killJob = NULL, listJobsQueued = NULL, listJobsRunning = NULL, array.var = NA_character_, store.job.collection = TRUE, store.job.files = FALSE, scheduler.latency = 0, fs.latency = 0, hooks = list()), workers = "<NULL>")
#> - tweaked: TRUE
#> - call: plan(future.batchtools::batchtools_custom, cluster.functions = cf)
#> BatchtoolsCustomFutureBackend:
#> Inherits: BatchtoolsMultiprocessFutureBackend, BatchtoolsFutureBackend, MultiprocessFutureBackend, FutureBackend
#> UUID: ee55118d87b71543b5c919a2cb7535a3
#> Number of workers: 1
#> Number of free workers: 1
#> Available cores: 4
#> Automatic garbage collection: FALSE
#> Early signaling: FALSE
#> Interrupts are enabled: TRUE
#> Maximum total size of globals: +Inf
#> Maximum total size of value: +Inf
#> Number of active futures: 0
#> Number of futures since start: 0 (0 created, 0 launched, 0 finished)
#> Total runtime of futures: 0 secs (NaN secs/finished future)
#> batchtools configuration file: <NA>
#> batchtools cluster functions: ‘Interactive’
#> batchtools cluster functions template: <NA>
#> Cache directory: ‘/cluster-data/user-homes/elizabethb/projects/prism-pkgdocs-build/installed-pkgs/2025-09-29/future.batchtools_0.21.0/docs/reference/.future/20251001_140005-Cj9EW1’ (0 folders)
#> batchtools resources:
#> list()
message("Main process ID: ", Sys.getpid())
#> Main process ID: 942212
f <- future({
data.frame(
hostname = Sys.info()[["nodename"]],
os = Sys.info()[["sysname"]],
cores = unname(parallelly::availableCores()),
pid = Sys.getpid(),
modules = Sys.getenv("LOADEDMODULES")
)
})
info <- value(f)
#> Warning: batchtools::waitForJobs(..., timeout = 2592000) returned FALSE
#> Warning: Will not remove batchtools registry, because the status of the batchtools was ‘error’, ‘defined’, ‘expired’, ‘submitted’ and future backend argument 'delete' is ‘on-success’: ‘/cluster-data/user-homes/elizabethb/projects/prism-pkgdocs-build/installed-pkgs/2025-09-29/future.batchtools_0.21.0/docs/reference/.future/20251001_140005-Cj9EW1/batchtools_678350931’
#> Error: Future (<unnamed-1>) of class BatchtoolsCustomFuture expired, which indicates that it crashed or was killed.
#> Post-mortem details:
#> Future state: ‘running’
#> Batchtools status: ‘defined’, ‘expired’, ‘submitted’
#> No logged output file exist (at the moment)
print(info)
#> Error: object 'info' not found