A note on parallel computing (with R)

The parallel computing utilizes multiple CPU cores. It is useful when we run many, say more than 100, independent programs, e.g., repetitive simulations or cross-validation. The proper usage of the parallel computing helps us save a great amount of time and speed up the research progress. A more practical way to maximize the efficiency of the parallel computing is to combine it with the high-performance computing (HPC). In this note, I demonstrate how to do the parallel computing on the local machine, followed by its implementation on HPC.

Parallel computing on local machine

Startup

The macOS and Linux users are able to use the function mclapply() from the R package parallel to implement the parallel computing. The Windows users need to use the packages foreach and doParallel, which are easy to learn with the help of documentations. In the following, I demonstrate how to use mclapply() to run 10 replicates simultaneously via a simple example.

First, let us take a look on how many CPU cores are equipped on the computer:

library(parallel)
numCores <- detectCores()
numCores

[1] 4

There are 4 CPU cores on my computer. Next, we wrap up 10 independent replicates by lapply() function. In each replicate, we implement a simple linear regression.

result1 <- lapply(1:1000, function(i){
  x <- matrix(rnorm(1000), 100, 10)
  y <- rnorm(100)
  lm(y~x)
})

Finally, we replace lapply() with mclapply(). The function mclapply() is the same as lapply() except for some additional optional arguments. One commonly used option is mc.cores, which specifies how many CPU cores are used.

result2 <- mclapply(1:1000, function(i){
  x <- matrix(rnorm(1000), 100, 10)
  y <- rnorm(100)
  lm(y~x)
}, mc.cores = numCores)

There is another interesting package called pbmcapply which provides a progress bar. It is useful when the program is time-consuming and you are curious about the progress.

library(pbmcapply)
result3 <- pbmclapply(1:10, function(i){
  x <- matrix(rnorm(1000), 100, 10)
  y <- rnorm(100)
  lm(y~x)
}, mc.cores = numCores)

| | 0%, ETA NA |=============================================================================================== | 80%, ETA 00:00 |===================================================================================================================| 100%, Elapsed 00:00

Reproducibility

The reproducibility of the code is important since it enables other people to reproduce the results appearing on our paper. To achieve the reproducibility of the code in the parallel computing, simply using set.seed() does not work. Instead, we should change the kind of Random Number Generator (RNG) using RNGkind() followed by set.seed().

library(parallel)
RNGkind("L'Ecuyer-CMRG")
set.seed(1)
result4 <- mclapply(1:10, function(i){
  x <- matrix(rnorm(1000), 100, 10)
  y <- rnorm(100)
  lm(y~x)
}, mc.cores = numCores)

Here, we select L'Ecuyer-CMRG type RNG. There are many other types of RNG (see ?RNGkind), but I did not verify if any of them also guarantees the reproducibility of the code in the parallel computing.

Next, we show how to do parallel computing on HPC.

Parallel computing on HPC

Running the parallel computing code on HPC is essentially the same what we did on the local machine. The only difference is that we need to prepare a bash file for submitting the job to the computing nodes on HPC.

We save the preceding code as the R source file simulation.R.

## R source file "simulation.R"
library(parallel)
RNGkind("L'Ecuyer-CMRG")
set.seed(1)
result4 <- mclapply(1:10, function(i){
  x <- matrix(rnorm(1000), 100, 10)
  y <- rnorm(100)
  lm(y~x)
}, mc.cores = numCores)

To submit the file simulation.R to HPC, we need a bash file. Here is an example file, which is named as job.sh.

#!/bin/bash
#SBATCH --job-name="simulation"
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --ntasks-per-core=1
#SBATCH -p genacc_q
#SBATCH --mail-type="END"
#SBATCH --mail-type="FAIL"
#SBATCH -o %x.out
#SBATCH -e %x.err
module load R/3.5.2
Rscript simulation.R

The usage of SBATCH arguments are as follows:

• N: the number of nodes. I prefer to keep it as 1.
• n: the number of CPU cores. I suggest to set it as 4 or 8 (or as large as 16 if the computing resources are available). It takes much time to wait for the allocation of too many cores.
• ntasks-per-core: the number of task on each core.
• p: the HPC partition. Use partition genacc_q if the job is time-consuming, but this partition is crowded. If the job can finish within 4 hours, then use partitions backfill and backfill2.
• o, e: specify the names of output and error files. %x represents the job-name. Therefore, the job will give the output file simulation.out and the error file simulation.err.

After we log in HPC, we submit the job using in the command line:

sbatch job.sh