multiprocessing in Python

Overview:

The Python package multiprocessing enables a Python program to create multiple python interpreter processes.
For a Python program running under CPython interpreter, it is not possible yet to make use of the multiple CPUs through multithreading due to the Global Interpreter Lock (GIL).
The multiprocessing package provides a way to incorporate both concurrency and parallelism using the CPython interpreter.

Concurrency: In a uniprocessor machine, multiple processes created through multiprocessing package are context switched by the OS, which results in concurrency.

Concurrency and Parallelism: In a multiprocessor machine, multiple processes created through this package are executed in multiple processors along with context switching, providing both concurrency and parallelism.

Creating multiple processes using Python:

With multiprocessing multiple processes can be created in one of the following ways

Using method “spawn” - It does not use the Unix fork. It uses pickling module to pickle the required function information and invoke them as part of creating the new process. This mechanism is available both in Unix and Windows operating systems.

Python multiprocessing using method spawn

Using method “fork” - This uses the system call fork(), the forking mechanism available in the Unix operating system. However forking a multithreaded program is not guaranteed to be safe.

Python multiprocessing using method fork

Using method “forkserver” - This method creates a separate single threaded server process to handle the process creation requests of the parent process. Since the server process is a single threaded process, multithreaded processes can be created without any side effects.

Python Multiprocessing using method forkserver

Example:

import multiprocessing as multiproc

# Function for process1

def procFunction1(msgQueue):

msgQueue.put("Message by Child Proc 1")

# Function for process2

def procFunction2(msgQueue):

msgQueue.put("Message by Child Proc 2")

if __name__ == '__main__':

print("Parent Process started")

# Set the process creation method to 'forkserver'

multiproc.set_start_method('forkserver')

# Create a shared message queue for the processes to produce/consume data

messageQueue = multiproc.Queue()

childProcess1 = multiproc.Process(target=procFunction1, args=(messageQueue,))

childProcess2 = multiproc.Process(target=procFunction2, args=(messageQueue,))

# Start the child processes

childProcess1.start()

childProcess2.start()

# Wait for child processes to complete

childProcess1.join()

childProcess2.join()

while not messageQueue.empty():

print(messageQueue.get())

print("Parent Process Exiting")

Output:

Parent Process started

Message by Child Proc 1

Message by Child Proc 2

Parent Process Exiting

Multiprocessing classes and their uses:

The python package multiprocessing provides several classes, which help writing programs to create multiple processes to achieve concurrency and parallelism. These classes cater to various aspects of multiprocessing which include creating the processes, communication between the processes, synchronizing the processes and managing them.

Class	Description
Process	Provides an abstraction of a task, which needs to be executed as a separate process as part of achieving multiprocessing.
Queue	A thread-safe, process-safe Queue class used for sending data between processes.

Multiprocessing in Python

Overview:

Creating multiple processes using Python:

Example:

Output:

Multiprocessing classes and their uses: