Multiprocessing and Parallel Programming In Python

Multiprocessing is creating, running, ending multiple processes and threads from a single process for a specific purpose. Python programming language provides a lot of different features of multiprocessing.

Multiprocessing Features

Multiprocessing provides a lot of features to the program or application developers. We will list some of the below.

• Performance is the most popular feature of multiprocessing. A single process task can be divided into multiple ones where each process can complete faster. In the end, the complete task will be completed in less time.
• Reliability is provided by creating simple tas pieces were even some piece corrupt we can create a little piece of task where other pieces to not interfere.
• Efficiency is done by using all CPU sources like threads r multiple CPUs. Modern CPU’s provides multiple processing units and threads to run multiple processes at the same time. With multiprocessing, we can use these resources efficiently.

Import Multiprocessing Module/Library

Python provides multiprocessing functions and capabilities with the multiprocessing module. So in order to use multiprocessing functions and capabilities, we have to import multiprocessing module like below.

import multiprocessing

We can also see that multiprocessing module provides process and similar functions.

Spawn Multiple Processes

We will start with a simple process spawn or create which will create a new process instance with the Process()function. Process() function basically needs two parameters where we need to specify the function name where the newly created function will run. Also, we can provide some parameters with the args set like below.

import multiprocessing

def foo(message):
    print(message)

if __name__ == '__main__':
    p = multiprocessing.Process(target=foo, args=("This is a spawn",))
    p.start()


    p = multiprocessing.Process(target=foo, args=("This is a spawn 2",))
    p.start()


    p = multiprocessing.Process(target=foo, args=("This is a spawn 3",))
    p.start()

We can see that we have created 3 processes to spawn and with the start() function we started them. We can not start a single new process multiple times so we have to define a new process with the Process() function like above.

Set Name For Created Processes

Multiprocessing provides different useful features like setting a name for a new process. We will use the name option which will be provided to the Process() function. We can access this name from the newly created process with the multiprocessing.current_process().name attribute like below.

import multiprocessing

def daemon(message):
    print(multiprocessing.current_process().name)
    print(message)

if __name__ == '__main__':
    p = multiprocessing.Process(name="This process name is daemon", target=daemon, args=("This is a spawn",))
    p.start()

Create Daemon Processes

When a normally created or spawned process completes its execution its ended. This is the normal behavior of all of the normal processes. But some special cases we need to run a process like a daemon or service where they will not stop even they have no task to complete. Daemons mainly wait for a specific event like connection, trigger some event to run. Python multiprocessing provides the daemonoption which will turn given process into a daemon which will run forever normally. But if the main process finishes the daemons will be excited too.

import multiprocessing

def daemon(message):
    print(message)


if __name__ == '__main__':
    p = multiprocessing.Process(target=daemon, args=("Waiting For Tasks",))
    p.daemon = True
    p.start()

    print("This will not printed bec")

Wait For Process To Complete

After the main process reached the end, the process will exit and all the child processes will stop. But we can change this behavior where the main process will wait for the child process to finish and exit. We will use join() function where the specified child process exit will be waiting by the main process.

import multiprocessing

def daemon(message):
print(message)


if __name__ == '__main__':
    p = multiprocessing.Process(target=daemon, args=("Waiting For Tasks",))
    p.daemon = True
    p.start()
    p.join()

    print("This will not printed bec")

Terminate Process

After a new child process is created it will exit after the task is completed. But In some cases, we may need to terminate the given child process before it finishes. We can use terminate() function which will terminate the given process.

import multiprocessing

def daemon(message):
    print(message)


if __name__ == '__main__':
    p = multiprocessing.Process(target=daemon, args=("Waiting For Tasks",))
    
    p.daemon = True
    
    p.start()
    
    p.terminate()
    
    print("Process Is Terminated")

Check If Given Child Process Is Running or Alive

After creating a child process it will run according to its task or whether it is a daemon. We may need this process current situation whether it is running or alive from the main process. We will use is_alive() function which will return boolean values True or False according to the child process situation.

import multiprocessing

def daemon(message):
    print(message)


if __name__ == '__main__':
    p = multiprocessing.Process(target=daemon, args=("Waiting For Tasks",))

    p.daemon = True

    p.start()

    print(p.is_alive())

Print Process Exit Status

The process executes, completes, and exits. But the process will have some exit or finish reasons for different exit cases. For example, a process can be exit because it’s killed or it finishes its execution successfully.  We can get or print the child process exit status with the exitcode attribute.

import multiprocessing
import time

def foo(message):
    print(message)


if __name__ == '__main__':

    p1 = multiprocessing.Process(target=foo, args=("Waiting For Tasks",))

    p2 = multiprocessing.Process(target=foo, args=("Waiting For Tasks",))

    p1.start()

    p2.start()


    p2.terminate()


    time.sleep(3)


    print(p1.exitcode)

    print(p2.exitcode)

There are meaning about the exit status.

• If equal to the 0 the process is executed and finished successfully without any error.
• If bigger than 0 there is an error related to the code.
• If lower than 0the process is killed with a signal.

Log Multiprocess Operations

Every created process has some logging mechanism. We can redirect these logs into a file or standard output with the log_to_stferr(logging.DEBUG) by specifying the level of the log verbosity which is DEBUG in this case.

def foo(message):
    print(message)


if __name__ == '__main__':

    multiprocessing.log_to_stderr(logging.DEBUG)

    p1 = multiprocessing.Process(target=foo, args=("Waiting For Tasks",))

    p2 = multiprocessing.Process(target=foo, args=("Waiting For Tasks",))

    p1.start()

    p2.start()


    p2.terminate()


    time.sleep(3)


    print(p1.exitcode)

    print(p2.exitcode)