Table of Contents
Why Multiprocessing exists?
Python’s Global Interpreter Lock (GIL) allows only one thread to execute Python bytecode at a time, no matter how many CPU cores your machine has.
Threading in Python is excellent for I/O-bound work — a thread waiting on a network response or disk read releases the GIL so another thread can run. But for CPU-bound work — number crunching, image processing, data transformations — threads fight over the same lock and you get no real parallelism.
Process vs Thread: The Core Trade-off
A thread is cheap to create and shares memory with its parent process automatically — but in Python, threads share the GIL, so CPU-bound work doesn’t parallelise.
A process is heavier to create (the OS has to allocate a new memory space, and Python has to bootstrap a fresh interpreter inside it) and does not share memory by default — but each process gets its own GIL, so CPU-bound work genuinely runs in parallel.
This trade-off shapes almost every design decision in the multiprocessing module: how processes are started, how they communicate, and why “shared state” needs special handling.
The multiprocessing module solves this by sidestepping the GIL entirely. Instead of spinning up threads inside one process, it spins up multiple independent OS processes, each with its own Python interpreter, its own GIL, and its own memory space. Since they don’t share an interpreter, they can genuinely run on separate cores at the same time.
Process vs Thread: The Core Trade-off
A thread is cheap to create and shares memory with its parent process automatically — but in Python, threads share the GIL, so CPU-bound work doesn’t parallelise.
A process is heavier to create (the OS has to allocate a new memory space, and Python has to bootstrap a fresh interpreter inside it) and does not share memory by default — but each process gets its own GIL, so CPU-bound work genuinely runs in parallel.
This trade-off shapes almost every design decision in the multiprocessing module: how processes are started, how they communicate, and why “shared state” needs special handling.
How Processes Are Started
There are three start methods, and the choice matters more than most people realise:
fork (default on Linux): The child process is a near-exact copy of the parent’s memory at the moment of forking. This is fast and means the child inherits everything — including already-imported modules and open file descriptors — without re-running your import statements. But it can cause subtle bugs if the parent held locks, open sockets, or database connections at fork time, since the child inherits them in a possibly inconsistent state.
spawn (default on macOS and Windows since Python 3.8): A brand-new Python interpreter is started from scratch, and only the specific objects you pass in are transferred (via pickling). This is safer and more predictable but slower to start, and it requires that everything passed to the child process be picklable.
forkserver: A server process is started once, and each new process is forked from that clean server rather than from the (possibly messy) main process. This gets fork’s speed with more of spawn’s safety.
If you’ve ever seen a multiprocessing script work fine on Linux but hang or throw pickling errors on Windows, this is almost always why — the default start method differs across platforms.
Processes Don’t Share Memory — So How Do They Talk?
Since each process has its own address space, you can’t just mutate a shared Python object and expect the other process to see it. Multiprocessing offers a few mechanisms instead:
Queue and Pipe — the most common pattern. A Queue is a FIFO channel: one process puts data in, another gets it out. Under the hood, data is pickled, sent through an OS pipe, and unpickled on the other side. A pipe gives you a lower-level, two-way connection between exactly two processes.
Shared memory (Value, Array, shared_memory module) — for cases where copying data via pickling is too slow (e.g., large NumPy-style arrays), Python offers multiprocessing.shared_memory.SharedMemory, which maps the same block of physical memory into multiple processes. Value and Array are simpler wrappers for single primitives and fixed-size arrays.
Manager objects — a Manager runs a separate server process that holds Python objects (dicts, lists, etc.) and exposes proxies to other processes. Every read or write goes through the manager process, which makes it flexible but noticeably slower than shared memory, since each operation involves interprocess communication rather than a direct memory access.
The rule of thumb: use Queue/Pipe for streaming results between workers, shared memory for large numeric buffers you need to avoid copying, and Manager only when you need a shared structure like a dict and can tolerate the overhead.
Queue in action — producer/consumer across processes
from multiprocessing import Process, Queue
import time
def producer(q: Queue, n: int):
for i in range(n):
q.put(i)
time.sleep(0.01)
q.put(None) # sentinel to tell the consumer we're done
def consumer(q: Queue):
while True:
item = q.get() # blocks until something is available
if item is None:
break
print(f"consumed {item}")
if __name__ == "__main__":
q = Queue()
p1 = Process(target=producer, args=(q, 10))
p2 = Process(target=consumer, args=(q,))
p1.start()
p2.start()
p1.join()
p2.join()Note the None sentinel — a Queue doesn’t know when a producer is “done,” so you signal end-of-stream explicitly.
Pipe — direct two-way channel between exactly two processes
from multiprocessing import Process, Pipe
def worker(conn):
conn.send({"status": "done", "result": 42})
conn.close()
if __name__ == "__main__":
parent_conn, child_conn = Pipe()
p = Process(target=worker, args=(child_conn,))
p.start()
print(parent_conn.recv()) # {'status': 'done', 'result': 42}
p.join()Value and Array — shared primitives without full shared memory
from multiprocessing import Process, Value, Array
def increment(counter, arr):
with counter.get_lock(): # explicit lock — increments aren't atomic by default
counter.value += 1
for i in range(len(arr)):
arr[i] *= 2
if __name__ == "__main__":
counter = Value('i', 0) # 'i' = C int
arr = Array('d', [1.0, 2.0, 3.0]) # 'd' = C double
procs = [Process(target=increment, args=(counter, arr)) for _ in range(5)]
for p in procs: p.start()
for p in procs: p.join()
print(counter.value) # 5
print(list(arr))counter.value += 1 is not atomic across processes — two workers can read the same old value before either writes back, losing an increment. That’s why get_lock() is used explicitly around it.
shared_memory — zero-copy sharing of large numeric buffers
from multiprocessing import shared_memory, Process
import numpy as np
def modify(shm_name, shape, dtype):
shm = shared_memory.SharedMemory(name=shm_name)
arr = np.ndarray(shape, dtype=dtype, buffer=shm.buf)
arr[:] = arr * 2 # mutates the SAME physical memory the parent sees
shm.close()
if __name__ == "__main__":
data = np.array([1, 2, 3, 4, 5], dtype=np.int64)
shm = shared_memory.SharedMemory(create=True, size=data.nbytes)
shared_arr = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)
shared_arr[:] = data[:]
p = Process(target=modify, args=(shm.name, data.shape, data.dtype))
p.start()
p.join()
print(shared_arr) # [2 4 6 8 10] — no pickling, no copying, real shared memory
shm.close()
shm.unlink()This is the pattern that matters for large NumPy arrays — Queue/Pipe would pickle and copy the whole array on every send; shared_memory maps the same bytes into both processes.
Manager — a shared dict across many workers
from multiprocessing import Manager, Pool
def add_to_dict(args):
shared_dict, key, value = args
shared_dict[key] = value
if __name__ == "__main__":
with Manager() as manager:
shared_dict = manager.dict()
with Pool(4) as pool:
pool.map(add_to_dict, [(shared_dict, i, i * i) for i in range(10)])
print(dict(shared_dict)) # {0: 0, 1: 1, 2: 4, ..., 9: 81}Every shared_dict[key] = value here is actually an RPC call to the manager’s server process, not a local memory write — convenient, but an order of magnitude slower than shared_memory for high-frequency updates.
The Pool: The Pattern You’ll Actually Use Most
In practice, most multiprocessing code doesn’t manually create and join individual Process objects — it uses multiprocessing.Pool. A Pool maintains a fixed number of worker processes (often set to os.cpu_count()) and hands out chunks of work to whichever worker is free.
from multiprocessing import Pool
def square(n):
return n * n
if __name__ == "__main__":
with Pool(processes=4) as pool:
results = pool.map(square, range(20))Key methods:
- map(func, iterable) — blocks until all results are ready, returns them in order.
- imap(func, iterable) — like map, but returns an iterator, so you can start consuming results as they complete rather than waiting for everything.
- apply_async(func, args) — submits one task and returns an AsyncResult you can poll or block on later, useful when tasks aren’t uniform.
The if __name__ == “__main__”: guard is not optional boilerplate — on spawn-based platforms, the child process re-imports your script from scratch, and without the guard it would recursively spawn Pools of Pools of Pools.
imap vs map vs apply_async — same task, three retrieval styles
from multiprocessing import Pool
import time
def slow_square(n):
time.sleep(0.5)
return n * n
if __name__ == "__main__":
nums = range(8)
# map: blocks until ALL results are ready, then returns them all at once, in order
with Pool(4) as pool:
results = pool.map(slow_square, nums)
print(results)
# imap: returns an iterator — start consuming as each result finishes
with Pool(4) as pool:
for result in pool.imap(slow_square, nums):
print("got:", result) # results stream in as they complete
# apply_async: fire-and-forget, useful for non-uniform tasks; poll or block later
with Pool(4) as pool:
async_results = [pool.apply_async(slow_square, (n,)) for n in nums]
final = [r.get() for r in async_results] # .get() blocks for that one result
print(final)Handling exceptions raised inside worker processes
Exceptions raised in a worker aren’t printed to your terminal automatically the way they would be in the main process — they’re pickled and re-raised in the parent when you fetch the result:
from multiprocessing import Pool
def risky(n):
if n == 3:
raise ValueError(f"bad input: {n}")
return n * 2
if __name__ == "__main__":
with Pool(4) as pool:
async_results = [pool.apply_async(risky, (n,)) for n in range(6)]
for r in async_results:
try:
print(r.get())
except ValueError as e:
print(f"worker failed: {e}") # exception surfaces here, in the parentIf you use pool.map instead of apply_async, the first exception encountered aborts the whole call and propagates immediately — you don’t get partial results.
A quick benchmark: multiprocessing vs threading on CPU-bound work
This is the experiment worth running yourself to see the GIL’s effect directly:
import time
from multiprocessing import Pool
from concurrent.futures import ThreadPoolExecutor
def cpu_bound(n):
total = 0
for i in range(10_000_000):
total += i * i
return total
if __name__ == "__main__":
tasks = [1, 2, 3, 4]
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=4) as ex:
list(ex.map(cpu_bound, tasks))
print(f"Threads: {time.perf_counter() - start:.2f}s") # ~same as sequential — GIL serializes it
start = time.perf_counter()
with Pool(4) as pool:
pool.map(cpu_bound, tasks)
print(f"Processes: {time.perf_counter() - start:.2f}s") # ~4x faster on a 4+ core machineOn a multi-core machine, the threaded version won’t be meaningfully faster than running the four calls sequentially — the GIL lets only one thread execute Python bytecode at a time. The process-based version should scale close to linearly with core count, because each process has its own GIL.
What Actually Gets Sent Between Processes: Pickling
Every argument passed to a worker process, and every result returned from one, must be pickled (serialised) by the parent, sent across, and unpickled by the child. This has real consequences:
- Anything unpicklable — open file handles, database connections, lambdas, some class instances with unpicklable attributes — will raise an error the moment you try to send it to a worker.
- Pickling and unpickling large objects has real CPU and memory cost. If your “parallel” workload spends most of its time serializing giant arguments, the overhead can eat the benefit of parallelism entirely. This is why shared memory exists for large numeric data.
When Multiprocessing Actually Helps
Multiprocessing pays off when: the work is CPU-bound (heavy computation, not waiting on I/O), the per-task work is large enough to outweigh the fixed cost of process startup and data serialisation, and the tasks are largely independent (don’t need constant fine-grained communication with each other).
It’s usually the wrong tool when: the work is I/O-bound (use asyncio or threads instead — they’re lighter and better suited), the tasks are tiny (the overhead of pickling and process management can exceed the actual work being parallelised), or the tasks need to share and mutate a lot of state (the synchronisation overhead eats the gains).
A Concrete Rule of Thumb for Choosing a Concurrency Model
- I/O-bound, want simplicity → threading
- I/O-bound, want to scale to many thousands of concurrent operations → asyncio
- CPU-bound, need real parallel execution across cores → multiprocessing
- CPU-bound but the heavy lifting happens inside a C extension (NumPy, pandas) that releases the GIL internally → threading can work fine, because the GIL is only held during actual Python bytecode execution, not during the underlying C computation
That last point trips people up: libraries like NumPy release the GIL while running their C routines, so a thread pool crunching NumPy arrays can genuinely use multiple cores even though “Python has a GIL.” The GIL constrains Python bytecode execution, not every line of code that happens to be called from Python.
Common Pitfalls
Forgetting the __main__ guard — causes infinite recomputation or import errors, especially on Windows/macOS with spawn.
Sharing unpicklable objects — database connections, open sockets, or thread locks captured in a closure will fail to serialise; open these fresh inside each worker instead.
Assuming mutation is shared — modifying a regular Python list or dict inside a worker process does nothing to the parent’s copy. If you need shared mutable state, you must explicitly use Manager, Value, Array, or shared memory.
Over-parallelising small tasks — spawning a process per tiny task can make code slower than running it sequentially, because process creation and IPC overhead dominates.
Not sizing the pool to the workload — more processes than CPU cores usually doesn’t help for CPU-bound work; it just adds context-switching overhead.
Closing Thought
Multiprocessing exists because the GIL makes true parallel execution of Python bytecode impossible within a single process. By handing work to separate OS processes — each with its own interpreter and memory — Python trades the cost of process startup and data serialisation for real multi-core parallelism. Used well, on CPU-bound work chunked into reasonably sized tasks, it’s the most direct way to make Python actually use all the cores on a machine.
