Tips & Tutorials

I frequently find myself working with large lists where I need to apply the same time-consuming function to each element in the list without concern for the order that these calculations are made. I’ve written a small class using Python’s multiprocessing module to help speed things up.

It will accept a list, break it up into a list of lists the size of the number of processes you want to run in parallel, and then process each of the sublists as a separate process. Finally, it will return a list containing all the results.

import multiprocessing

class ProcessHelper:
def __init__(self, num_processes=4):
self.num_processes = num_processes

def split_list(self, data_list):
list_of_lists = []
for i in range(0, len(data_list), self.num_processes):
list_of_lists.append(data_list[i:i+self.num_processes])
return list_of_lists

def map_reduce(self, function, data_list):
split_data = self.split_list(data_list)
processes = multiprocessing.Pool(processes=self.num_processes)
results_list_of_lists = processes.map(function, split_data)
processes.close()
results_list = [item for sublist in results_list_of_lists for item in sublist]
return results_list

To demonstrate how this class works, I’ll create a list of 20 integers from 0-19. I’ve also created a function that will square every number in a list. When I run it, I’ll pass the function (job) and the list (data). The class will then break this into a list of lists and then run the function as a separate process on each of the sublists.

def job(num_list):
return [i*i for i in num_list]

data = range(20)

p = ProcessHelper(4)
result = p.map_reduce(job, data)
print(result)

So if my data originally was a list that looked like this:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

When I split it into sublists, I’ll end up with a list of 4 lists (as I’ve indicated that I want to initialise 4 processes):

[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19]]

Finally, the result will give me the list of squared values that looks like this:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361]

I’ll continue to build this class as I identify other handy helper methods that I could add.

One of the things that may seem confusing in Python is how there appears to be two ways to test if variables are the same: ‘==’ and ‘is’:

x = 200
y = 200
print(x == y)
print(x is y)

Both comparison methods returned True, so they do the same thing right? Well, not really. To illustrate this, I’ll change the integer value assigned:

x = 500
y = 500
print(x == y)
print(x is y)

Now we have ‘==’ returning True and ‘is’ returning False. What happened? Well, ‘==’ is referred to as the equality operator, and it checks that the values of the variables are the same. That’s why it returned True both times. On the other hand, ‘is’ is referred to as the identity operator and compares the id of the object that the variable points to. I have a more detailed post on object ids available here.

So the moral of this is that usually you just want to use the equality operator (‘==’), as there is no guarantee that variables storing the same number will also point to the same object (and therefore have the same object id). The only times you really need to use the ‘is’ operator is when you explicitly need to check if one variable is actually pointing to the same object as another variable.

Did you know that every object in your Python program is given a unique identifier by the interpreter which you can return using the ‘id()’ function? Let’s see what happens when we assign variables to each other in Python and then print out the variable value and object id:

x = 5
y = x
print(x, id(x))
print(y, id(y))

Now I am printing out two things about each variable: the value it stores (the number 5) and the object id. The object id for both variables should be the same. This is telling us that both the x and y variables are referring to the same object. Now, what happens if I were to change the y variable?

x = 5
y = x
y += 1
print(x, id(x))
print(y, id(y))

Here I’ve added 1 to the y variable, and as you can see, both the value and object id of the y variable is now different to the x variable. This change has occurred because integers in Python are immutable data types. Integers, floats, strings and tuples are examples of immutable objects in Python. What this means is that when you change the number a variable refers to, the Python interpreter points the variable to a completely different object.

Now, the distinction between mutable and immutable objects in Python starts to become more important when you start working with mutable data types like lists:

x = [1]
y = x
y.append(5)
print(x, id(x))
print(y, id(y))

The thing that may surprise a lot of people is that now the list both x and y points to has changed! Now the list has a 1 and a 5 in it, even though I only appended to y. This is very easy to get tripped up on if you don’t know what Python is really doing under the hood.

Now how do you get around this? Luckily the standard library has us covered with a handy package called ‘copy’:

import copy

x = [1]
y = copy.copy(x)
y.append(5)
print(x, id(x))
print(y, id(y))

Once you use the ‘copy’ method on x, you can see from the output that y has a new object id and that changes you make to the y variable no longer affect x. Problem solved, right?

Well, there’s now just one final thing to consider – what happens if I have a list of lists and I use the copy method?

import copy

x = [[1, 2], [3, 4]]
y = copy.copy(x)
y[0].append(5)
print(x, id(x))
print(y, id(y))

You can see that I’ve made a copy of the x variable, then appended to the first list in y (the one containing [1, 2]). However, when you see the output, you’ll notice that like before, the change made to y has also modified x. You can see the reason for this by looking at the object id of the first list of x and y:

print(x[0], id(x[0]))
print(y[0], id(y[0]))

Both lists refer to the same object, only the outer list was copied and received a new object id! To solve this, we’ll need to use the ‘deepcopy’ method. This will create unique copies of all the objects throughout the list – not just a copy of the outer list itself:

import copy

x = [[1, 2], [3, 4]]
y = copy.deepcopy(x)
y[0].append(5)
print(x, id(x))
print(y, id(y))

Now we should see that any changes to the lists in x or y will not affect the variable anymore.

So how is this useful? The reason I decided to write this post this evening was because some of the code I was working on today involved initialising multiple dictionaries with many keys. I could have copied and pasted them, but it was a lot easier and more readable for me to create an initial variable, then use ‘deepcopy’ to initialise the other variables. If I hadn’t been aware of Python object ids and mutable types, then I could very easily have created a massive bug in my code by creating several variables that all modified the same dictionary.