PS_11_ParallelProcessing.md

title	Parallel Processing

Serial Computing

Most (legacy) software is written for serial computation:

• Problem broken into discrete set of instructions
• Instructions executed sequentially on a single processor

Figure from here

Parallel computation

• Problem divided into discrete parts that can be solved concurrently
• Instructions executed simultaneously on different processors
• Overall control/coordination mechanism

Figure from [here](https://computing.llnl.gov/tutorials/parallel_comp/)

Flynn's taxonomy

A classification of computer architectures (Flynn, 1972)

Four Categories

1. Single Instruction, Single Data (SISD)
   • No parallelization
2. Single Instruction, Multiple Data (SIMD)
   • Run the same code/analysis on different datasets
   • Examples:
     • different species in species distribution model
     • same species under different climates

---

3. Multiple Instruction, Single Data (MISD)
   • Run different code/analyses on the same data
   • Examples:
     • One species, multiple models
4. Multiple Instruction, Multiple Data streams (MIMD)
   • Run different code/analyses on different data
   • Examples:
     • Different species & different models

Flynn's Taxonomy

Figure from [here](http://en.wikipedia.org/wiki/Flynn%27s_taxonomy)

Our focus: Single Instruction, Multiple Data (SIMD)

1. Parallel functions within an R script
   • starts on single processor
   • runs looped elements on multiple 'slave' processors
   • returns results of all iterations to the original instance
   • foreach, multicore, plyr, raster
2. Alternative: run many separate instances of R in parallel with Rscript
   • need another operation to combine the results
   • preferable for long, complex jobs
   • NOT planning to discuss in this session

R Packages

There are many R packages for parallelization, check out the CRAN Task View on High-Performance and Parallel Computing for an overview. For example:

• Rmpi: Built on MPI (Message Passing Interface), a de facto standard in parallel computing.
• snow: Simple Network of Workstations can use several standards (PVM, MPI, NWS)
• parallel Built in R package (since v2.14.0).
• multidplyr

Foreach Package

In this session we'll focus on the foreach package, which has numerous advantages including:

• intuitive for() loop-like syntax
• flexibility of parallel 'backends' from laptops to supercomputers (multicore, parallel, snow, Rmpi, etc.)
• nice options for combining output from parallelized jobs

Documentation for foreach:

• foreach manual
• foreach vignette
• Nested Loops

Foreach backends

• doParallel best for use on multicore machines (uses fork on linux/mac and snow on windows).
• doMPI: Interface to MPI (Message-Passing Interface)
• doSNOW: Simple Network of Workstations

Examples

Libraries

## New Packages
library(foreach)
library(doParallel)

Sequential for loops

With for()

x=vector()
for(i in 1:3) 
  x[i]=i^2
x

## [1] 1 4 9

With foreach()

x <- foreach(i=1:3) %do% 
  i^2
x

## [[1]]
## [1] 1
## 
## [[2]]
## [1] 4
## 
## [[3]]
## [1] 9

Manipulating output

x <- foreach(i=1:3) %do% 
  i^2
x

## [[1]]
## [1] 1
## 
## [[2]]
## [1] 4
## 
## [[3]]
## [1] 9

Note that x is a list with one element for each iterator variable (i). You can also specify a function to use to combine the outputs with .combine. Let's concatenate the results into a vector with c.

Sequential foreach() loop with .combine

x <- foreach(i=1:3,.combine='c') %do% 
  i^2
x

## [1] 1 4 9

Tells foreach() to first calculate each iteration, then .combine them with a c(...)

Sequential foreach() loop with .combine

x <- foreach(i=1:3,.combine='rbind') %do% 
  i^2
x

##          [,1]
## result.1    1
## result.2    4
## result.3    9

Parallel foreach() loop

So far we've only used %do% which only uses a single processor.

Must register a parallel backend with one of the do functions such as doParallel(). On most multicore systems, the easiest backend is typically doParallel(). On linux and mac, it uses fork system call and on Windows machines it uses snow backend. The nice thing is it chooses automatically for the system.

registerDoParallel(3) # register specified number of workers
#registerDoParallel() # or, reserve all all available cores 
getDoParWorkers() # check registered cores

## [1] 3

Parallel foreach() loop

To run in parallel, simply change the %do% to %dopar%. Wasn't that easy?

## run the loop
x <- foreach(i=1:3, .combine='c') %dopar% 
  i^2
x

## [1] 1 4 9

Review Basic Steps

Most parallel computing:

1. Split problem into pieces (iterators: i=1:3)
2. Execute the pieces in parallel (%dopar%)
3. Combine the results back (.combine)

Useful foreach parameters

• .inorder (true/false) results combined in the same order that they were submitted?
• .errorhandling (stop/remove/pass)
• .packages packages to made available to sub-processes
• .export variables to export to sub-processes

Raster Package

Parallel processing with raster

Some functions in the raster package also easy to parallelize.

ncores=2
beginCluster(ncores)

fn=function(x) x^3
system.time(clusterR(r, fn, verbose=T))
endCluster()

Raster package

Does not work with:

• merge
• crop
• mosaic
• (dis)aggregate
• resample
• projectRaster
• focal
• distance
• buffer
• direction

spatial.tools

However, check out the spatial.tools package

rasterEngine executes a function on Raster* object(s) using foreach, to achieve parallel reads, executions and writes.

Summary

Each task should involve computationally-intensive work. If the tasks are very small, it can take longer to run in parallel.