multiproc_utils module

This module implements utilities for executing code in multiple processes and threads.

class swpt_pythonlib.multiproc_utils.ThreadPoolProcessor(threads, *, iter_args_collections, process_func, wait_seconds)

Executes a function in multiple threads.

The passed iter_args_collections function will be called from the main thread ad infinitum. It should return an iterable of a list of tuples. Then for each of the tuples in the list, the process_func will be called in worker threads, until the list is exhausted. This process will continue until the returned iterable is exhausted, and then iter_args_collections will be called again to obtain a new iterable. A pause of at least wait_seconds will be made between the sequential calls of iter_args_collections.

swpt_pythonlib.multiproc_utils.spawn_worker_processes(processes: int, target, **kwargs)

Spawns the specified number of processes, each executing the passed target function. In each worker process, the target function will be called with the passed keyword arguments (kwargs), and should performs its work ad infinitum.

Note that each worker process inherits blocked SIGTERM and SIGINT signals from the parent process. The target function must unblock them at some point, by calling try_unblock_signals().

This function will not return until at least one of the worker processes has stopped. In this case, the rest of the workers will be terminated as well.

swpt_pythonlib.multiproc_utils.try_block_signals()

Blocks HANDLED_SIGNALS on platforms that support it.

swpt_pythonlib.multiproc_utils.try_unblock_signals()

Unblocks HANDLED_SIGNALS on platforms that support it.