- The Python package multiprocessing enables a Python program to create multiple python interpreter processes.
- For a Python program running under CPython interpreter, it is not possible yet to make use of the multiple CPUs through multithreading due to the Global Interpreter Lock (GIL).
- The multiprocessing package provides a way to incorporate both concurrency and parallelism using the CPython interpreter.
Concurrency: In a uniprocessor machine, multiple processes created through multiprocessing package are context switched by the OS, which results in concurrency.
Concurrency and Parallelism: In a multiprocessor machine, multiple processes created through this package are executed in multiple processors along with context switching, providing both concurrency and parallelism.
Creating multiple processes using Python:
With multiprocessing multiple processes can be created in one of the following ways
- Using method “spawn” - It does not use the Unix fork. It uses pickling module to pickle the required function information and invoke them as part of creating the new process. This mechanism is available both in Unix and Windows operating systems.
- Using method “fork” - This uses the system call fork(), the forking mechanism available in the Unix operating system. However forking a multithreaded program is not guaranteed to be safe.
- Using method “forkserver” - This method creates a separate single threaded server process to handle the process creation requests of the parent process. Since the server process is a single threaded process, multithreaded processes can be created without any side effects.
import multiprocessing as multiproc
# Function for process1
msgQueue.put("Message by Child Proc 1")
# Function for process2
msgQueue.put("Message by Child Proc 2")
if __name__ == '__main__':
print("Parent Process started")
# Set the process creation method to 'forkserver'
# Create a shared message queue for the processes to produce/consume data
messageQueue = multiproc.Queue()
childProcess1 = multiproc.Process(target=procFunction1, args=(messageQueue,))
childProcess2 = multiproc.Process(target=procFunction2, args=(messageQueue,))
# Start the child processes
# Wait for child processes to complete
while not messageQueue.empty():
print("Parent Process Exiting")
Parent Process started
Message by Child Proc 1
Message by Child Proc 2
Parent Process Exiting
Multiprocessing classes and their uses:
The python package multiprocessing provides several classes, which help writing programs to create multiple processes to achieve concurrency and parallelism. These classes cater to various aspects of multiprocessing which include creating the processes, communication between the processes, synchronizing the processes and managing them.
Provides an abstraction of a task, which needs to be executed as a separate process as part of achieving multiprocessing.
|Queue||A thread-safe, process-safe Queue class used for sending data between processes.|