This is a reference vignette of the package’s core functionality. Other package vignettes cover additional features.
1. The mirai Concept
mirai (Japanese for ‘future’) implements the concept of futures in R.
Futures are an abstraction that represent the result of code evaluation that will be available at some point in the future. The actual code evaluation is sent to and performed in a separate R process (daemon), and the result is sent back to the main (host) process when it completes.
The package has one main function: mirai()
to create a
mirai object.
This function returns almost immediately, and is never blocking. This is the essence of async: whilst the mirai evaluation is ongoing on the daemon, the host R process is free to continue with other things.
As the mirai expression is sent to another process, it must be self-contained. This means that any functions and variables used in it must be available in that process as well. This requires that:
- Package functions be namespaced using
::
, orlibrary()
calls be made within the expression. - Other functions/data/objects required by the expression should be
passed via
...
or.args
to be sent along to the daemon.
A mirai is either unresolved if the result has yet to be
received, or resolved if it has. unresolved()
is a
helper function to check the state of a mirai.
For a mirai m
, the result is available at
m$data
once it has resolved. Normally this will be the
return value of the evaluated expression. If the expression errored,
caused the process to crash, or timed out then this will be an
‘errorValue’ instead. See the section Errors in a mirai below.
Rather than repeatedly checking unresolved(m)
, it is
more efficient to wait for and collect its value by using
m[]
.
When a developer or code author writes a mirai()
call,
they should not be concerned about where or how execution of that code
actually happens. It is simply meant to be executed on the resources
that are available to it at the time it is run.
Instead, it is for the end-user running the code to declare the
resources available for evaluating mirai calls. This is done using the
package’s other main function: daemons()
.
If daemons have not been set, each mirai()
call will by
default create a new local background process (ephemeral
daemon) on which to perform its evaluation.
Instead, daemons()
sets up persistent daemons on which
to evaluate mirai expressions.
- Using persistent daemons eliminates the time and overhead of starting new processes for each evaluation, and limits the number of processes used at any one time.
- Even re-using the same daemon, cleanup steps performed between evaluations ensure that each mirai continues to be self-contained and unaffected by past evaluations.
How to set up and launch daemons is covered in sections below, starting with local daemons.
2. Example Use Cases
Parallelize compute-intensive tasks
Multiple long computes (model fits etc.) can be performed in parallel on available computing cores.
The following mimics an expensive calculation that eventually returns a random value.
library(mirai)
args <- list(time = 2L, mean = 4)
m <- mirai(
{
Sys.sleep(time)
rnorm(5L, mean)
},
time = args$time,
mean = args$mean
)
m
#> < mirai [] >
m$data
#> 'unresolved' logi NA
unresolved(m)
#> [1] TRUE
# Do other stuff
collect_mirai(m)
#> [1] 3.171449 2.518936 5.100663 4.802009 4.847599
m[]
#> [1] 3.171449 2.518936 5.100663 4.802009 4.847599
For easy programmatic use of mirai()
, ‘.expr’ accepts a
pre-constructed language object, and also a list of named arguments
passed via ‘.args’. So, the following would be equivalent to the
above:
expr <- quote({Sys.sleep(time); rnorm(5L, mean)})
args <- list(time = 2L, mean = 4)
m1 <- mirai(.expr = expr, .args = args)
m2 <- mirai(.expr = expr, .args = args)
m1[]
#> [1] 3.045729 2.936203 4.315945 3.302699 6.096261
m2[]
#> [1] 3.848957 4.245206 4.324884 4.337941 5.008054
By running the above two calculations in parallel, they take roughly half the time as running sequentially (minus a relatively inconsequential parallelization overhead).
Unblock I/O-bound Operations
Problem: high-frequency real-time data cannot be written to file/database synchronously without disrupting the execution flow of ingesting the data.
Solution: cache data in memory and use mirai()
to
perform periodic write operations asynchronously from a separate
process.
Below, ‘.args’ is used to pass environment()
, which is
the calling environment. This provides a convenient method of passing in
existing objects, as as the x
and file
arguments to the write.cv.async()
function.
library(mirai)
write.cv.async <- function(x, file) {
mirai(write.csv(x, file), .args = environment())
}
m <- write.cv.async(x = rnorm(1e6), file = tempfile())
while (unresolved(m)) {
cat("Writing file...\n")
Sys.sleep(0.5)
}
#> Writing file...
#> Writing file...
cat("Write complete:", is.null(m$data))
#> Write complete: TRUE
Resilient Pipelines
Code that can potentially fail is isolated in a separate process to ensure continued uptime.
As part of a data science / machine learning pipeline, iterations of model training may periodically fail for stochastic reasons (e.g. problematic graphics cards memory management).
Running each iteration in a mirai isolates this potentially problematic code such that it doesn’t bring down the entire pipeline, even if it fails.
library(mirai)
run_iteration <- function(i) {
# simulates a stochastic error rate
if (runif(1) < 0.1) stop("random error\n", call. = FALSE)
sprintf("iteration %d successful\n", i)
}
for (i in 1:10) {
m <- mirai(run_iteration(i), environment())
while (is_error_value(m[])) {
cat(m$data)
m <- mirai(run_iteration(i), environment())
}
cat(m$data)
}
#> iteration 1 successful
#> iteration 2 successful
#> iteration 3 successful
#> iteration 4 successful
#> iteration 5 successful
#> iteration 6 successful
#> iteration 7 successful
#> iteration 8 successful
#> iteration 9 successful
#> iteration 10 successful
By testing the return value of each mirai for errors, error-handling code is able to automate recovery and re-attempts, as above. The result is a resilient and fault-tolerant pipeline that minimizes downtime by eliminating interruptions of long computes.
3. Errors in a mirai
If execution in a mirai fails, the error message is returned as a character string of class ‘miraiError’ and ‘errorValue’ to facilitate debugging.
is_mirai_error()
may be used to test for mirai execution
errors.
m1 <- mirai(stop("occurred with a custom message", call. = FALSE))
m1[]
#> 'miraiError' chr Error: occurred with a custom message
m2 <- mirai(mirai::mirai())
m2[]
#> 'miraiError' chr Error in mirai::mirai(): missing expression, perhaps wrap in {}?
is_mirai_error(m2$data)
#> [1] TRUE
is_error_value(m2$data)
#> [1] TRUE
A full stack trace of evaluation within the mirai is recorded and
accessible at $stack.trace
on the error object.
f <- function(x) if (x > 0) stop("positive")
m3 <- mirai({f(-1); f(1)}, f = f)
m3[]
#> 'miraiError' chr Error in f(1): positive
m3$data$stack.trace
#> [[1]]
#> stop("positive")
#>
#> [[2]]
#> f(1)
Elements of the original error condition are also accessible via
$
on the error object. For example, additional metadata
recorded by rlang::abort()
is preserved:
f <- function(x) if (x > 0) stop("positive")
m4 <- mirai(rlang::abort("aborted", meta_uid = "UID001"))
m4[]
#> 'miraiError' chr Error: aborted
m4$data$meta_uid
#> [1] "UID001"
If a daemon instance is sent a user interrupt, the mirai will resolve to an object of class ‘miraiInterrupt’ and ‘errorValue’.
is_mirai_interrupt()
may be used to test for such
interrupts.
m4 <- mirai(rlang::interrupt()) # simulates a user interrupt
is_mirai_interrupt(m4[])
#> [1] TRUE
If execution of a mirai surpasses the timeout set via the ‘.timeout’ argument, the mirai will resolve to an ‘errorValue’ of 5L (timed out). This can, amongst other things, guard against mirai processes that have the potential to hang and never return.
m5 <- mirai(nanonext::msleep(1000), .timeout = 500)
m5[]
#> 'errorValue' int 5 | Timed out
is_mirai_error(m5$data)
#> [1] FALSE
is_mirai_interrupt(m5$data)
#> [1] FALSE
is_error_value(m5$data)
#> [1] TRUE
is_error_value()
tests for all mirai execution errors,
user interrupts and timeouts.
4. Local Daemons
Daemons, or persistent background processes, may be set to receive ‘mirai’ requests.
Daemons inherit the default system configuration and read in the relevant ‘.Renviron’ and ‘.Rprofile’ etc. on startup. They also load the default packages. To instead only load the
base
package (which cuts out more than half of R’s startup time), the environment variableR_SCRIPT_DEFAULT_PACKAGES=NULL
may be set prior to launching daemons.
With Dispatcher (default)
Call daemons()
specifying the number of daemons to
launch.
daemons(6)
#> [1] 6
The default dispatcher = TRUE
creates a
dispatcher()
background process that connects to individual
daemon processes on the local machine. This ensures that tasks are
dispatched efficiently on a first-in first-out (FIFO) basis to daemons
for processing. Tasks are queued at dispatcher and sent to a daemon as
soon as it can accept the task for immediate execution. Dispatcher
employs an event-driven approach that is efficient both in terms of
consuming no resources while waiting, whilst also being fully
synchronized with events.
To view the current status, status()
provides:
- The number of active connections,
- The URL daemons connect to, and
- A task summary:
-
waiting
number of tasks queued for execution at dispatcher -
assigned
number of tasks sent to a daemon for execution -
complete
number of tasks for which the result has been received (either completed or cancelled)
status()
#> $connections
#> [1] 6
#>
#> $daemons
#> [1] "ipc:///tmp/9692d9ac2f065c50bb16e481"
#>
#> $mirai
#> awaiting executing completed
#> 0 0 0
daemons(0)
#> [1] 0
Set the number of daemons to zero to reset. This reverts to the default of creating a new background process for each ‘mirai’ request.
Without Dispatcher
Alternatively, specifying dispatcher = FALSE
, the
background daemons connect directly to the host process.
daemons(6, dispatcher = FALSE)
#> [1] 6
This means that tasks are sent immediately in a round-robin fashion, which ensures that they are evenly-distributed amongst daemons. This does not however guarantee optimal scheduling, as the duration of tasks cannot be known a priori. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons are idle having already completed their tasks.
The advantage of this approach is that it is resource-light and does not require an additional dispatcher process. It is suited to working with similar-length tasks, or where concurrent tasks typically do not exceed available daemons.
Requesting the status now shows 6 connections, along with the host URL:
status()
#> $connections
#> [1] 6
#>
#> $daemons
#> [1] "ipc:///tmp/cb135b871d004dfd18dd5ee6"
Everywhere
everywhere()
may be used to evaluate an expression on
all connected daemons and persist the resultant state, regardless of a
daemon’s ‘cleanup’ setting.
The above keeps the DBI
package loaded for
all evaluations. Other types of setup task may also be performed,
including making a common resource available, such as a database
connection:
everywhere(con <<- dbConnect(RSQLite::SQLite(), file), file = tempfile())
By super-assignment, the conenction ‘con’ will be available in the global environment of all daemon instances. Subsequent mirai calls may then make use of ‘con’.
Disconnect from the database everywhere:
everywhere(dbDisconnect(con))
Sometimes it may be necessary to evaluate an expression in the global environment of each daemon. As mirai evaluation does not occur in the global environment itself, but one inheriting from it, an explicit call to
evalq(envir = .GlobalEnv)
achieves this. An example use case isbox::use()
to import a module or package:
everywhere(
evalq(
box::use(dplyr[select], mirai[...]),
envir = .GlobalEnv
)
)
daemons(0)
#> [1] 0
With Method
daemons()
has a with()
method, which
evaluates an expression with daemons created for the duration of the
expression and automatically torn down upon completion.
It was originally designed for running a Shiny app with the desired number of daemons, as in the example below:
Note: it is assumed the app is already created. Wrapping a call to
shiny::shinyApp()
would not work asrunApp()
is implicitly called when the app is printed, however printing occurs only afterwith()
has returned, hence the app would run outside of the scope of thewith()
statement.
In the case of a Shiny app, all mirai calls will be executed before the app returns as the app itself is blocking. In the case of other expressions, be sure to call the results (or collect the values) of all mirai within the expression to ensure that they all complete before the daemons are torn down.
If specifying a compute profile for
the daemons()
call, all calls with
.compute = NULL
within the with()
clause will
default to this compute profile.
5. Remote Daemons
The daemons interface may also be used to send tasks for computation to remote daemon processes on the network.
Call daemons()
specifying ‘url’ as a character string
such as: ‘tcp://10.75.32.70:5555’ at which daemon processes should
connect. Alternatively, use host_url()
to automatically
construct a valid URL. The host (or dispatcher) listens at this address,
utilising a single port.
IPv6 addresses are also supported and must be enclosed in square brackets
[]
to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address::ffff:a6f:50d
would be specified astcp://[::ffff:a6f:50d]:5555
.
For options on actually launching the daemons, please see the next section.
Below, calling host_url()
without a port value uses the
default of ‘0’. This is a wildcard value that will automatically assigns
a free ephemeral port:
The actual assigned port may be queried at any time via
status()
:
status()
#> $connections
#> [1] 0
#>
#> $daemons
#> [1] "tcp://10.37.58.114:50816"
#>
#> $mirai
#> awaiting executing completed
#> 0 0 0
The number of daemons connected at any time may be dynamically scaled up or down, according to requirements.
To reset all connections and revert to default behaviour:
daemons(0)
#> [1] 0
Closing the connection causes all connected daemons to exit automatically. If using dispatcher, it will cause dispatcher to exit, and in turn all connected daemons when their respective connections with the dispatcher are terminated.
6. Launching Remote Daemons
The launcher analogy is appropriate, as they are ways to execute a daemon on the machine of your choice, very much like launching a satellite. Once deployed, the daemon connects back to your host process through it’s own communications (TCP or TLS over TCP).
The local launcher simply runs an Rscript
instance via a
local shell. The remote launcher uses a method to run this
Rscript
command on a remote machine.
To launch remote daemons, supply a remote launch configuration to the
‘remote’ argument of daemons()
, or
launch_remote()
at any time thereafter.
There are currently 3 options for generating remote launch configurations:
-
ssh_config()
where there is SSH access to the remote machine. -
cluster_config()
to use HPC cluster resource managers such as Slurm, SGE, Torque/PBS and LSF. -
remote_config()
for a generic, flexible method that caters for other custom launchers.
The return value of all of these functions is a simple list. This means that they may be pre-constructed, saved and re-used whenever the same configuration is required.
i. SSH Direct Connection
This method is appropriate for internal networks and in trusted, properly-configured environments where it is safe for your machine to accept incoming connections on certain ports. In the examples below, the remote daemons connect back directly to port 5555 on the local machine.
In these cases, using TLS is often desirable to provide additional security to the connections.
The first example below launches 4 daemons on the machine 10.75.32.90 (using the default SSH port of 22 as this was not specified), connecting back to the host URL:
daemons(
n = 4,
url = host_url(tls = TRUE, port = 5555),
remote = ssh_config("ssh://10.75.32.90")
)
The second example below launches one daemon on each of 10.75.32.90 and 10.75.32.91 using the custom SSH port of 222:
daemons(
n = 1,
url = host_url(tls = TRUE, port = 5555),
remote = ssh_config(c("ssh://10.75.32.90:222", "ssh://10.75.32.91:222"))
)
ii. SSH Tunnelling
Use SSH tunnelling to launch daemons on any machine you are able to access via SSH, whether on the local network or the cloud. SSH key-based authentication must already be in place, but no other configuration is required.
This provides a convenient way to launch remote daemons without them needing to directly access the host. Firewall configurations or security policies often prevent opening a port to accept outside connections. In these cases, SSH tunnelling creates a tunnel once the initial SSH connection is made. For simplicity, the implementation in mirai uses the same tunnel port on both the host and daemon.
To use tunnelling, supply a URL with hostname of ‘127.0.0.1’ to ‘url’
for the daemons()
call.
-
local_url(tcp = TRUE)
does this for you. - The default uses the wildcard port of ‘0’, which assigns a free ephemeral port.
- Whilst convenient, there is a small possibility that this port may not be available on all daemons.
- It is hence preferable to specify a specific port that has been whitelisted for use, where possible.
For example, if local_url(tcp = TRUE, port = 5555)
is
specified, the tunnel is created using port 5555 on each machine. The
host listens to 127.0.0.1:5555
on its side, and the daemons
each dial into 127.0.0.1:5555
on their own respective
machines.
The below example launches 2 daemons on the remote machine 10.75.32.90 using SSH tunnelling:
daemons(
n = 2,
url = local_url(tcp = TRUE),
remote = ssh_config("ssh://10.75.32.90", tunnel = TRUE)
)
iii. HPC Cluster Resource Managers
cluster_config()
may be used to deploy daemons using a
cluster resource manager / scheduler.
- The first argument is
command
. This should be:
-
"sbatch"
if using Slurm -
"qsub"
if using SGE / Torque / PBS -
"bsub"
if using LSF.
- The second argument
options
are any options that you would normally supply in a shell script to pass to the scheduler. These are script lines typically preceded by a#
.
Slurm: "#SBATCH --job-name=mirai
#SBATCH --mem=10G
#SBATCH --output=job.out"
SGE: "#$ -N mirai
#$ -l mem_free=10G
#$ -o job.out"
Torque/PBS: "#PBS -N mirai
#PBS -l mem=10gb
#PBS -o job.out"
LSF: "#BSUB -J mirai
#BSUB -M 10000
#BSUB -o job.out"
- As per the above, it is fine to pass this as a character string with
the options each on a new line (whitespace is automatically handled), or
else by explicitly using
\n
to denote a newline. - It is also permissible to include other shell commands, for example to change working directory.
- For the avoidance of doubt, the initial shebang line of a script
such as
#!/bin/bash
is not required. - For certain HPC setups, a final line which loads environment modules may be needed. This would usually be of the form:
module load R
or for a specific R version:
module load R/4.5.0
- The third argument
rscript
defaults to"Rscript"
, which assumes that R is on the file search path. This may be substituted for the full path to a specific R executable, such as that returned byfile.path(R.home("bin"), "Rscript")
.
iv. Generic Remote Configuration
remote_config()
provides a generic, flexible framework
for running any shell command that may be used to deploy daemons.
Conceptually, this function takes an args
argument,
which must contain “.”. The correctly-configured call to
daemon()
is substituted in for this “.”, so that
command
is run with this as one of its arguments.
This can provide an alternative for cluster resource managers in
certain cases, although cluster_config()
provides an easier
and more complete interface. Using Slurm as an example, the following
uses sbatch
to launch a daemon on the cluster, with some
additional Slurm options passed via command line arguments to
sbatch
:
v. Manual Deployment
As an alternative to automated launches, calling
launch_remote()
without specifying ‘remote’ may be used to
return the shell commands for deploying daemons manually.
The printed return values may then be copy / pasted directly to a remote machine e.g. via a terminal session.
daemons(url = host_url())
#> [1] 0
launch_remote()
#> [1]
#> Rscript -e 'mirai::daemon("tcp://10.37.58.114:50820")'
daemons(0)
#> [1] 0
7. TLS Secure Connections
TLS provides a robust solution for securing communications from the local machine to remote daemons.
Automatic Zero-configuration Default
Simply specify a secure URL using the scheme tls+tcp://
when setting daemons, or use host_url(tls = TRUE)
, for
example:
Single-use keys and certificates are automatically generated and configured, without requiring any further intervention. The private key is always retained on the host machine and never transmitted.
The generated self-signed certificate is available via
launch_remote()
, where it is included as part of the shell
command for manually launching a daemon on a remote machine.
launch_remote(1)
#> [1]
#> Rscript -e 'mirai::daemon("tls+tcp://10.37.58.114:50824",tls=c("-----BEGIN CERTIFICATE-----
#> MIIFPzCCAyegAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQDDAwxMC4z
#> Ny41OC4xMTQxETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDAeFw0wMTAx
#> MDEwMDAwMDBaFw0zMDEyMzEyMzU5NTlaMDcxFTATBgNVBAMMDDEwLjM3LjU4LjEx
#> NDERMA8GA1UECgwITmFub25leHQxCzAJBgNVBAYTAkpQMIICIjANBgkqhkiG9w0B
#> AQEFAAOCAg8AMIICCgKCAgEAswpXuhaKrn95sz8K4TbwfjgLhOLCCx2zM/oVPld/
#> k+rXhJG9+Pu85GGcsbgFuyx7iPcRoDb4oeSo6PGFEFIXyDODkxo6h9OPllDBq+cd
#> Q5KjihwxANgVR1MLiCA6DsLvRhznCid9wSoAFttLRFstZOZHFcRIuIUB9hZzd7U4
#> F8v3IxA/vxCYWKPgV7IXyke8zD9rTDUrCQngq9jGr3Hq+pCqqzofbxm09DPvzx1l
#> clelJV7Q3UzMogIFH3WYBizBM7QVQSMOl4F1KiXl4g/86Kw4lP1LVJJe+gVcf4Wd
#> cJI/P63SlURmiO54OsXoMPeGYo1DQPZX8/V7uIfspUPYvlMiOu5S/Wb07N430D3L
#> GIoJY+UYJmOFCH8rSVI8HUd+W3LVVc2Qj6bcrClhIAzw9+uvUunZMDwTb8mpnywG
#> MCJNl2NBeKYAjhBQA1cgEgzyjByg8jBbBxdJ7AYE3tRO91zjd+t5xSYa4M3ddvvg
#> dX318tAmZBAgvpdOnzBXnxn7uWR3uy4N11wYMbMa1+sVGF+/njSiH5bCnGIhsFbx
#> EVdhdsp+DPF8Pe8qnphSbGxDhdneqM91yrB2uzoFAqStiUAe9L1y5wpQ3YwzeL+y
#> Ea5ZoWEWvo2uayh/ufiy9qBT3Vw4VF3Uir8uW5L3zZDx2EARpZzERY74KMF3SoVl
#> gBMCAwEAAaNWMFQwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU08pRkVxK
#> qt9Kq3gV88ndTZwjM2AwHwYDVR0jBBgwFoAU08pRkVxKqt9Kq3gV88ndTZwjM2Aw
#> DQYJKoZIhvcNAQELBQADggIBAA/ILAiOP0ncsQP2h8WGohDxjB1jH5D8xyk3dyQi
#> dqsyDtRGbX//6EtHsuYPH4DX12iVyhCe8Ho62m7yvYVm0GeHzC4uqwqDZBFqYc/H
#> V8hhaU+p39HbK8QU+GxkYfaGCcnMHSyNxEiC+lRDiKfc7QyC+Ui0xn3weqEcPcY4
#> P2Ej92r4/tvFYttU7+JJb2Fj8yiO8z/YVJKr0PXp//Prc/3hSGRd7DAhYisjHkJI
#> TzXhJGF+12uut2nQ+RqVjGtbuibhPQfklLvyZjOxrFtZW31wj/vAZ9IJLq01mFak
#> u8TA+BpNux7auoPw1Nce2Uuw5ybpU86Vs3n6jYFWhUzeLqLcGtVr29gcyYJitWjf
#> W23eBzN6zoMPVUJss/K7vPPQc9jY7zUtslOgjqJ0GY704XGBLXJJPVfryPwMXTIE
#> iI/3MI8mNcFTV+1P+YJSlBqqpv9BO9Log4vPehCaFdpqJfPqppZYX+rApUBotC9V
#> asizsjNgQ3YSNu0PhQCgmdl8a7rlr3dhf7z6yRzQkZ0WUW7ey7xQZ4wokEl7XnBO
#> WI0x8GryXEGB9n6tnY3vgji5xMUJB0BWMe66RxFhM4NkeilfXOxMGihxC8Pbo/6H
#> WHcedqb4nTbn/coz79gD45dhtiAZGz/LLlojQlAcE6hvMoHR/oZV1NyVU2VthjSK
#> 3QMt
#> -----END CERTIFICATE-----
#> ",""))'
daemons(0)
#> [1] 0
CA Signed Certificates
As an alternative to the zero-configuration default, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA). The CA may be a public CA or internal to an organisation.
- Generate a private key and CSR. The following resources describe how to do so:
- using Mbed TLS: https://mbed-tls.readthedocs.io/en/latest/kb/how-to/generate-a-certificate-request-csr/
- using OpenSSL: https://www.feistyduck.com/library/openssl-cookbook/online/ (Chapter 1.2 Key and Certificate Management)
- Provide the generated CSR to the CA for it to sign a new TLS certificate.
- The common name (CN) of the certificate must be identical to the hostname or IP address actually used for the connection. As this is verified, it will fail if not the same.
- The received certificate should comprise a block of cipher text
between the markers
-----BEGIN CERTIFICATE-----
and-----END CERTIFICATE-----
. Make sure to request the certificate in the PEM format. If only available in other formats, the TLS library used should usually provide conversion utilities. - Check also that the private key is a block of cipher text between
the markers
-----BEGIN PRIVATE KEY-----
and-----END PRIVATE KEY-----
.
- When setting daemons, the TLS certificate and private key should be
provided to the ‘tls’ argument of
daemons()
.
- If the certificate and private key have been imported as character
strings
cert
andkey
respectively, then the ‘tls’ argument may be specified as the character vectorc(cert, key)
. - Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this file may be provided to the ‘tls’ argument.
- When launching daemons, the certificate chain to the CA should be
supplied to the ‘tls’ argument of
daemon()
orlaunch_remote()
.
- The certificate chain should comprise multiple certificates, each
between
-----BEGIN CERTIFICATE-----
and-----END CERTIFICATE-----
markers. The first one should be the newly-generated TLS certificate, the same supplied todaemons()
, and the final one should be a CA root certificate. - These are the only certificates required if the certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with the TLS certificate and ends with the certificate of the CA.
- If these are concatenated together as a single character string
certchain
, then the character vector comprising this and an empty character stringc(certchain, "")
may be supplied to the relevant ‘tls’ argument. - Alternatively, if these are written to a file (and the file replicated on the remote machines), then the ‘tls’ argument may also be specified as a path/filename (assuming these are the same on each machine).
8. Compute Profiles
daemons()
has a .compute
argument to
specify separate sets of daemons (compute profiles) that
operate totally independently. This is useful for managing tasks with
heterogeneous compute requirements:
- send tasks to different daemons or clusters of daemons with the appropriate specifications (in terms of CPUs / memory / GPU / accelerators etc.)
- split tasks between local and remote computation
Simply pass a character string to .compute
to use as the
profile name (which, if NULL
, is ‘default’). The daemons
settings are saved under the named profile.
To create a ‘mirai’ task using a specific compute profile, specify
the .compute
argument to mirai()
, which uses
the ‘default’ compute profile if this is NULL
.
Similarly, functions such as status()
,
launch_local()
or launch_remote()
should be
specified with the desired .compute
argument.