# Python Multiprocessing: A Finesse Approach

## A declaration

This content is not AI-generated whatsoever, so help me God.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1764090252715/db1cbf96-6956-47f7-95b7-6dcb890cf7a1.png align="center")

## Introduction

It’s 2025, and the 18-year-old me would never think I would be writing a blog on Python. As a platform engineer, I find myself running migrations of databases. As I advanced in these things, I found Python scripts are things I can depend on to get the job done, simulating migrations without hard writes, etc…

Today, we aren’t here to talk about migrations, but how I’m leveraging Python `multiprocessing` to run them quickly, and Multiprocessing is not concurrency but “true” Parallelism, aka doing multiple things at the same time.

## A back story

Say you are a painter, and you have 100 rooms to paint in an apartment, options:

* **Single Processing** (traditional approach)
    
    You are one painter who has to paint room 1, finish goes to 2, and so on, till you reach room 100.
    
    The only Advantage is the memory space one painter takes up space based on the room. On the downside is time, say you take 30 mins per room, that would be 30 × 100 = `3000 mins` ~ 50 hours
    
* **Multi Processing**
    
    Another approach, say you hire 10 painters, in this case, they represent processes, each painter takes up memory space (n), during execution, `painter 1` takes up rooms 1 to 10, `painter 2` takes up rooms 11 to 20, and so forth. With a bit of memory sacrifice, you will save time, 50 hours being cut into:  
    10 rooms X 30 mins = `300 mins` ~ 5 hours. Which is way faster
    

## Python Interpretation

The above can be represented by the code snippet below

```python
# This is like having one painter
for room in rooms:
    paint_room(room)  # Takes 30 minutes each

# This is like having multiple painters
with Pool(processes=10) as pool:
    pool.map(paint_room, rooms)  # All painters work simultaneously
```

## Limitations to Multiprocessing

Based on my experience, it might be a bad thing for production usage **depending** on the use case.

1. **Number of Cores**
    
    There is no faking till you make it here. And I learnt the hard way after seeing very weird behaviours in database updates. `The more the workers never means faster execution` Here is a breakdown:
    
    ````python
    import time
    from multiprocessing import Pool
    import matplotlib.pyplot as plt
    
    def cpu_intensive_task(n):
        total = 0
        for i in range(10000000):
            total += i * n
        return total
    
    def benchmark_workers(max_workers=32, tasks=32):
        results = []
        
        for num_workers in range(1, max_workers + 1):
            start = time.time()
            
            with Pool(processes=num_workers) as pool:
                pool.map(cpu_intensive_task, range(tasks))
            
            elapsed = time.time() - start
            results.append({
                'workers': num_workers,
                'time': elapsed,
                'tasks_per_second': tasks / elapsed
            })
            print(f"Workers: {num_workers:3d} | Time: {elapsed:.2f}s | Tasks/sec: {tasks/elapsed:.2f}")
        
        return results
    
    # Run benchmark
    results = benchmark_workers(max_workers=32, tasks=32)
    ```
    
    ### Typical Results Pattern (8-core machine):
    ```
    Workers:   1 | Time: 32.00s | Tasks/sec: 1.00   ← Baseline
    Workers:   2 | Time: 16.10s | Tasks/sec: 1.99   ← ~2x faster ✓
    Workers:   4 | Time:  8.20s | Tasks/sec: 3.90   ← ~4x faster ✓
    Workers:   6 | Time:  5.60s | Tasks/sec: 5.71   ← ~6x faster ✓
    Workers:   8 | Time:  4.10s | Tasks/sec: 7.80   ← ~8x faster ✓
    Workers:  10 | Time:  4.05s | Tasks/sec: 7.90   ← Marginal improvement
    Workers:  12 | Time:  4.02s | Tasks/sec: 7.96   ← Barely faster
    Workers:  16 | Time:  4.00s | Tasks/sec: 8.00   ← Plateauing
    Workers:  20 | Time:  4.01s | Tasks/sec: 7.98   ← No improvement
    Workers:  32 | Time:  4.15s | Tasks/sec: 7.71   ← SLOWER! ✗
    ```
    
    ## Why This Happens
    
    ### With 8 Cores, 8 Workers (Optimal)
    ```
    Time →
    Core 1: [Worker 1 ████████████████]
    Core 2: [Worker 2 ████████████████]
    Core 3: [Worker 3 ████████████████]
    Core 4: [Worker 4 ████████████████]
    Core 5: [Worker 5 ████████████████]
    Core 6: [Worker 6 ████████████████]
    Core 7: [Worker 7 ████████████████]
    Core 8: [Worker 8 ████████████████]
    
    Each worker gets dedicated CPU time = FAST
    ```
    
    ### With 8 Cores, 16 Workers (Oversubscribed)
    ```
    Time →
    Core 1: [W1 ██][W9 ██][W1 ██][W9 ██]  ← Context switching
    Core 2: [W2 ██][W10 ██][W2 ██][W10 ██]
    Core 3: [W3 ██][W11 ██][W3 ██][W11 ██]
    Core 4: [W4 ██][W12 ██][W4 ██][W12 ██]
    Core 5: [W5 ██][W13 ██][W5 ██][W13 ██]
    Core 6: [W6 ██][W14 ██][W6 ██][W14 ██]
    Core 7: [W7 ██][W15 ██][W7 ██][W15 ██]
    Core 8: [W8 ██][W16 ██][W8 ██][W16 ██]
    
    Workers fight for CPU time = OVERHEAD
    ````
    
2. **Shared Resources**
    
    Think of this like `all 10 painters trying to use the same brush at once` That could be chaotic. Things like Database connections, File handling that includes reading, writing, etc, HTTP requests, Memory, and variables. For example:
    
    ```python
    # WRONG - File handle can't be shared
    file = open('data.txt', 'w')
    
    def worker(data):
        file.write(data)  # Will crash
    
    with Pool(5) as pool:
        pool.map(worker, ['data1', 'data2'])
    
    # CORRECT - Each process opens its own file
    def worker(data):
        with open(f'data_{os.getpid()}.txt', 'w') as file:
            file.write(data)
    ```
    
3. **The multiprocessing library has a manager that can manage variables**
    
    From above, we have already established that shared resources are tricky to handle in parallel. So how do you handle this?.
    
    1. For non-mutating variables, it’s fine; each worker will get a copy of the global variable
        
        ```python
        LARGE_LOOKUP_TABLE = {'painter_name': 'Patrick', ...}
        
        def worker(item):
            value = LARGE_LOOKUP_TABLE.get(item)
        ```
        
    2. Shared Variables
        
        ```python
        from multiprocessing import Value, Array, Manager
        
        # Simple
        counter = Value('i', 0)  
        def worker(n, counter):
            with counter.get_lock():
                counter.value += 1
        
        # Complex
        manager = Manager()
        results_list = manager.list() 
        results_dict = manager.dict()
        ```
        

## When to use `Multiprocessing`

As I sign out, as you are ready to prompt that agent to rewrite your backend APIs to use parallelism, there are things you need to consider.

**Good for:**

* I/O bound tasks (database queries, API calls)
    
* CPU-intensive tasks with independent work units
    
* When you have many similar tasks
    

**Not good for:**

* Tasks that need to share state frequently
    
* Simple, fast operations (overhead might make it slower)
    
* When you're near memory limits
    

This is just an introduction. Let me know if you are interested in advanced stuff like process intercommunications, which are quite similar to `Golang` channels, but a bit complicated.  
Adios ✌️