Skip to content

mirai

ミライ


みらい 未来

Minimalist Async Evaluation Framework for R

→ Run R code in parallel while keeping your session free

→ Scale seamlessly from your laptop to cloud servers or HPC clusters

→ Automate actions as soon as tasks complete


Installation

Quick Start

mirai() evaluates an R expression asynchronously in a parallel process.

daemons() sets up persistent background processes for parallel computations.

library(mirai)
daemons(5)

m <- mirai({
  Sys.sleep(1)
  100 + 42
})

mp <- mirai_map(1:9, \(x) {
  Sys.sleep(1)
  x^2
})

m
#> < mirai [] >
m[]
#> [1] 142

mp
#> < mirai map [4/9] >
mp[.flat]
#> [1]  1  4  9 16 25 36 49 64 81

daemons(0)

Design Philosophy

→ Dynamic Architecture

  • Inverted topology, where daemons connect to host, enables true dynamic scaling
  • Optimal load balancing through efficient FIFO dispatcher scheduling
  • Event-driven promises complete with zero latency (and no polling overhead)

→ Modern Foundation

  • Built on NNG via nanonext, scales reliably to millions of tasks / thousands of processes
  • High performance, with round-trip times measured in microseconds, not milliseconds
  • Native support for IPC, TCP, and zero-config TLS with automatic certificate generation

→ Production First

  • Clear evaluation model with explicit dependencies prevents surprises from hidden state
  • Serialization support for cross-language data formats (torch tensors, Arrow tables)
  • OpenTelemetry integration for observability across distributed processes

→ Deploy Everywhere

  • Local, network / cloud (via SSH, SSH tunnelling) or HPC (via Slurm, SGE, PBS, LSF)
  • Modular compute profiles direct tasks to the most suitable resources
  • Combine local, remote, and HPC resources in a single compute profile

Powers the R Ecosystem

mirai serves as a foundation for asynchronous, parallel and distributed computing in the R ecosystem.

R parallel   The first official alternative communications backend for R, the ‘MIRAI’ parallel cluster, a feature request by R-Core.

purrr   Powers parallel map for purrr, a core tidyverse package.

Shiny   Primary async backend for Shiny with full ExtendedTask support.

plumber2   Built-in async evaluator enabling the @async tag in plumber2.

tidymodels   Core parallel processing infrastructure provider for tidymodels.

torch   Seamless use of torch tensors, models and optimizers across parallel processes.

Arrow   Query databases over ADBC connections natively in the Arrow data format.

Polars   R Polars leverages mirai’s serialization registration mechanism for transparent use of Polars objects.

targets   Targets uses crew as its default high-performance computing backend. Crew is a distributed worker launcher extending mirai to different computing platforms.

Acknowledgements

Will Landau for being instrumental in shaping development of the package, from initiating the original request for persistent daemons, through to orchestrating robustness testing for the high performance computing requirements of crew and targets.

Joe Cheng for integrating the ‘promises’ method to work seamlessly within Shiny, and prototyping event-driven promises.

Luke Tierney of R Core, for discussion on L’Ecuyer-CMRG streams to ensure statistical independence in parallel processing, and reviewing mirai’s implementation as the first ‘alternative communications backend for R’.

Travers Ching for a novel idea in extending the original custom serialization support in the package.

Hadley Wickham, Henrik Bengtsson, Daniel Falbel, and Kirill Müller for many deep insights and discussions.

mirai | nanonext | CRAN HPC Task View

–

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.