Parallel Processing

Serial Computing

Most (legacy) software is written for serial computation:

Problem broken into discrete set of instructions
Instructions executed sequentially on a single processor

Figure from here

Parallel computation

Problem divided into discrete parts that can be solved concurrently
Instructions executed simultaneously on different processors
Overall control/coordination mechanism

alt text
Figure from here

Flynn’s taxonomy

A classification of computer architectures (Flynn, 1972)

Four Categories

Single Instruction, Single Data (SISD)
- No parallelization
Single Instruction, Multiple Data (SIMD)
- Run the same code/analysis on different datasets
- Examples:
  - different species in species distribution model
  - same species under different climates

Multiple Instruction, Single Data (MISD)
- Run different code/analyses on the same data
- Examples:
  - One species, multiple models
Multiple Instruction, Multiple Data streams (MIMD)
- Run different code/analyses on different data
- Examples:
  - Different species & different models

Flynn’s Taxonomy

alt text
Figure from here

Our focus: Single Instruction, Multiple Data (SIMD)

Parallel functions within an R script
- starts on single processor
- runs looped elements on multiple ‘slave’ processors
- returns results of all iterations to the original instance
- foreach, multicore, plyr, raster
Alternative: run many separate instances of R in parallel with Rscript
- need another operation to combine the results
- preferable for long, complex jobs
- NOT planning to discuss in this session

R Packages

There are many R packages for parallelization, check out the CRAN Task View on High-Performance and Parallel Computing for an overview. For example:

Rmpi: Built on MPI (Message Passing Interface), a de facto standard in parallel computing.
snow: Simple Network of Workstations can use several standards (PVM, MPI, NWS)
parallel Built in R package (since v2.14.0).
multidplyr

Foreach Package

In this session we’ll focus on the foreach package, which has numerous advantages including:

intuitive for() loop-like syntax
flexibility of parallel ‘backends’ from laptops to supercomputers (multicore, parallel, snow, Rmpi, etc.)
nice options for combining output from parallelized jobs

Documentation for foreach:

Foreach backends

doParallel best for use on multicore machines (uses fork on linux/mac and snow on windows).
doMPI: Interface to MPI (Message-Passing Interface)
doSNOW: Simple Network of Workstations

Examples

Libraries

## New Packages
library(foreach)
library(doParallel)

Sequential `for` loops

With `for()`

x=vector()
for(i in 1:3) 
  x[i]=i^2
x

## [1] 1 4 9

With `foreach()`

x <- foreach(i=1:3) %do% 
  i^2
x

## [[1]]
## [1] 1
## 
## [[2]]
## [1] 4
## 
## [[3]]
## [1] 9

Manipulating output

x <- foreach(i=1:3) %do% 
  i^2
x

## [[1]]
## [1] 1
## 
## [[2]]
## [1] 4
## 
## [[3]]
## [1] 9

Note that x is a list with one element for each iterator variable (i). You can also specify a function to use to combine the outputs with .combine. Let’s concatenate the results into a vector with c.

Sequential `foreach()` loop with `.combine`

x <- foreach(i=1:3,.combine='c') %do% 
  i^2
x

## [1] 1 4 9

Tells foreach() to first calculate each iteration, then .combine them with a c(...)

Sequential `foreach()` loop with `.combine`

x <- foreach(i=1:3,.combine='rbind') %do% 
  i^2
x

##          [,1]
## result.1    1
## result.2    4
## result.3    9

Parallel `foreach()` loop

So far we’ve only used %do% which only uses a single processor.

Must register a parallel backend with one of the do functions such as doParallel(). On most multicore systems, the easiest backend is typically doParallel(). On linux and mac, it uses fork system call and on Windows machines it uses snow backend. The nice thing is it chooses automatically for the system.

registerDoParallel(3) # register specified number of workers
#registerDoParallel() # or, reserve all all available cores 
getDoParWorkers() # check registered cores

## [1] 3

Parallel `foreach()` loop

To run in parallel, simply change the %do% to %dopar%. Wasn’t that easy?

## run the loop
x <- foreach(i=1:3, .combine='c') %dopar% 
  i^2
x

## [1] 1 4 9

Review Basic Steps

Most parallel computing:

Split problem into pieces (iterators: i=1:3)
Execute the pieces in parallel (%dopar%)
Combine the results back (.combine)

Useful `foreach` parameters

.inorder (true/false) results combined in the same order that they were submitted?
.errorhandling (stop/remove/pass)
.packages packages to made available to sub-processes
.export variables to export to sub-processes

Raster Package