Include Non-Trivial Python Examples

[MorganRodgers]

Our R example is a great comparison of several methods to do parallel processing in R. I would love to see something similar for Python. One quick way to add a useful example would be to include a comparison of Python3's standard library classes concurrent.futures.ThreadPoolExecutor and concurrent.futures.ProcessPoolExecutor. These two classes allow Python to take better advantage of a single compute node. Their performance differences could be highlighted with a comparison using the prime number checking example: https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor-example

In order to take advantage of multiple nodes Python users can take advantage of MPI (at least on Owens) using mpi4py. MPI offers a number of options, but the mpi4py offers a futures-esque interface similar to Python3's concurrent.futures: https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html

[MorganRodgers]

Something like:

#!/usr/bin/env python
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
import math
from time import time


def is_prime(n):
  if n % 2 == 0:
    return False

  sqrt_n = int(math.floor(math.sqrt(n)))
  for i in range(3, sqrt_n + 1, 2):
    if n % i == 0:
      return False
  return True

def main():
  PRIMES = list(range(1_000_000))
  start = time()
  with ProcessPoolExecutor() as executor:
    primes = executor.map(is_prime, PRIMES)
  print('Calculation using ProcessPoolExecutor with n processes', time() - start)

  start = time()
  with ThreadPoolExecutor() as executor:
    primes = executor.map(is_prime, PRIMES)
  print('Calculation using ThreadPoolExecutor with n threads', time() - start)

# MPI Example here...

if __name__ == '__main__':
  main()