In my previous blog article, we learned how to clean data with Python Pandas. Today we are going to learn one of the key tools that has been widely used for data processing and data intensive computing.
Python is one of the most popular languages for data analysis and development, because it’s fast for testing and coding. However, this interpreted, dynamicallytyped, highlevel language, is somewhat slow for advanced/complex computing tasks.
To get the best of both worlds (fast code development time and fast code execution time): NumPy comes in.
In this article, we’ll be going through some basic NumPy functions and how to use them effectively when working with a large dataset. We will also take a look at the difference between Pure Python and NumPy. Finally, we will learn some strategies that can help us speed up code with NumPy.
Introduction to NumPy
What is NumPy?
NumPy is a Python library used for working with multidimensional arrays. NumPy stands for “Numerical Python”. It provides a highperformance multidimensional array object, and tools for working with these arrays.
NumPy is a core Python library for scientific computing. With it, we can perform complex mathematical and logical operations faster and easier, such as:
 VectorVector multiplication
 MatrixMatrix and MatrixVector multiplication
 Elementwise operations on vectors and matrices (i.e., adding, subtracting, multiplying, and dividing by a number )
 Elementwise or arraywise comparisons
 Applying functions elementwise to a vector/matrix ( like pow, log, and exp)
 A whole lot of Linear Algebra operations can be found in NumPy.linalg
 Reduction, statistics, and much more.
Quick Start to NumPy
Let’s take a look at a few examples to understand it better.
1. Creating Arrays
With Numpy, we can create multidimensional arrays:
import numpy as np
# Create a 1D Array
arr = np.array([1, 2, 3, 4, 5])
print(arr)
# Create a 2D Array
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr)
# Create a 3D Array
arr = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])
print(arr)
Out:
array([1, 2, 3, 4, 5])

array([[1, 2, 3],
[4, 5, 6]])

array([[[1, 2, 3],
[4, 5, 6]],
[[1, 2, 3],
[4, 5, 6]]])
2. Array Indexing
Indexing arrays is similar to Python lists:
arr = np.array([1, 2, 3, 4])
# Get the first element from the array
print(arr[0])
# Get third and fourth elements from the array and sum them.
print(arr[2] + arr[3])
Out:
1

7
3. Array Slicing
We can get a slice of an array like this: [start:end]
.
arr = np.array([1, 2, 3, 4, 5, 6, 7])
# Slice elements from index 1 to index 5 from the array
arr[1:5]
Out:
array([2, 3, 4, 5])
4. Array Shape
The shape of an array is the number of elements in each dimension. We can get the current shape of an array like this:
# Create a 2D array
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(arr)
# Get the shape
print(arr.shape)
Out:
array([[1, 2, 3, 4],
[5, 6, 7, 8]])

(2, 4)
(2, 4) means that the array has two dimensions, the first dimension has two elements and the second has four.
5. Reshaping arrays
Reshape an array from 1D to 2D
# Create a 1D array
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
print(arr)
# Convert the 1D array with 12 elements into a 2D array. The outermost dimension will have 4 arrays, each with 3 elements
new_arr = arr.reshape(4, 3)
print(new_arr)
Out:
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])

array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10, 11, 12]])
For more details, see NumPy Documentation.
Python Lists vs. NumPy Arrays
A common question that pops up: what is the real difference between Python lists and NumPy arrays? The simple answer is performance.
Why is NumPy faster?
Python speed suffers from small overhead, such as type checking and reference counting, which builds up for each operation and it could become especially significant in a large loop.
What NumPy does is that it pushes those loop operations down into compiled code, so the type checking only has to happen once for the entire loop therefore they can be done quickly.
Overall, NumPy performs better in the following instances:
 Size  NumPy uses much less memory to store data.
 Performance  NumPy arrays are faster than Python lists.
 Functionality  NumPy has optimized mathematical functions builtin that are convenient to use.
Memory consumption
To allocate a list of 1000 elements, Python takes 48000 bytes of the memory, whereas NumPy only needs 8000 bytes. What a significant improvement!
import numpy as np
import sys
# Declaring a Python list of 1000 elements
ls = range(1000)
print(f"Size of each element of the list: {sys.getsizeof(ls)} bytes.")
print(f"Size of the list: {sys.getsizeof(ls)*len(ls)} bytes.")
# Declaring a Numpy array of 1000 elements
arr = np.arange(1000)
print(f"Size of each element of the array: {arr.itemsize} bytes.")
print(f"Size of the array: {arr.size*arr.itemsize} bytes.")
Out:
Size of each element of the list: 48 bytes.
Size of the list: 48000 bytes.
Size of each element of the array: 8 bytes.
Size of the array: 8000 bytes.
Performance
There’s a vast difference in performance. If we add 5 to each element of a list of 100k values:
# Python way
ls = list(range(100000))
%timeit [val + 5 for val in ls]
# Numpy way
arr = np.array(ls)
%timeit arr + 5
Out:
6.17 ms ± 35.9 µs per loop

34.8 µs ± 340 ns per loop
It turned out using about 6 milliseconds (could be varied depending on different machines) for Python to process per loop while NumPy only took 34 microseconds per loop. We have about nearly 200x speed up by taking away the loops in Python.
Strategies for speeding up code with NumPy
Now we’ve learned that NumPy is a powerful tool for working with large arrays. However, it’s not only useful for scientific computing. It can also be handy for some software/services development tasks that involve complex data processing with a large dataset. With pure Python, we would normally loop over the dataset and do it raw and that’s… not so optimal right?
In order to achieve that effectively, let’s see what NumPy is capable of and how we can use it to speed our code, by applying the following strategies. Let’s get rid of those slow loops!
 Ufuncs
 Aggregations
 Slicing, Masking and Fancy indexing
Ufuncs
Ufuncs, short for universal functions, offers a bunch of operators that operate element by element on entire arrays.
(See universal functions basics for more details about ufuncs.)
For example, we want to add the elements of list a and b. Without NumPy, we can use the builtin zip()
method:
a = list(range(1001)) # [0, 1, 2, 3, ..., 999, 1000]
b = list(reversed(range(1001))) # [1000, 999, 998, ..., 1, 0]
c = [i+j for i, j in zip(a, b)] # returns [1000, 1000, ..., 1000, 1000]
Instead of looping the list, we can simply use the add()
from Numpy ufunc:
import numpy as np
a = np.array(list(range(1001)))
b = np.array(list(reversed(range(1001))))
c = np.add(a, b) #returns array([1000, 1000, ..., 1000, 1000])
Let’s compare the execution time:
%timeit [i+j for i, j in zip(a, b)]
%timeit np.add(a, b)
Out:
61 µs ± 4.11 µs per loop

811 ns ± 29.8 ns per loop
We got 75x faster with the NumPy version and the code looks much cleaner. Isn’t that awesome?
There are many other ufuncs available:
 Arithmetic:
+
,
,*
,/
,//
,%
…  Comparison:
<
,>
,==
,<=
,>=
,!=
…  Bitwise:
&
,
,~
,^
…  Trig family:
np.sin
,np.cos
,np.tan
…  And more…
Aggregations
Aggregations are functions which summarize the values in an array (e.g. min, max, sum, mean, etc.).
Let’s see how long it takes a Python loop to get the minimum value of a list of 100k random values.
With Python builtin min()
:
from random import random
ls = [random() for i in range(100000)]
%timeit min(ls)
With NumPy aggregate functions min()
:
arr = np.array(ls)
%timeit arr.min()
Out:
1.3 ms ± 7.59 µs per loop

33.6 µs ± 248 ns per loop
Again, we got about 40x speedup without writing a single loop with NumPy!
In addition, aggregations can also work on multidimensional arrays:
# Create a 3x5 array with random numbers
arr = np.random.randint(0, 10, (3, 5))
print(arr)
# Get the sum of the entire array
print(arr.sum())
# Get the sum of all the columns in the array
print(arr.sum(axis=0))
# Get the sum of all the rows in the array
print(arr.sum(axis=1))
Out:
array([[7, 7, 5, 1, 9],
[8, 7, 0, 2, 4],
[0, 5, 6, 4, 0]])

65

array([15, 19, 11, 7, 13])

array([29, 21, 15])
There are more aggregate functions:
 sum
 mean
 product
 median
 variance
 argmin
 argmax
 And more
Slicing, Masking and Fancy indexing
We just saw how to slice and index NumPy arrays earlier. In fact, there are other faster and more convenient ways that NumPy offers:
Masking
With masks, we can index an array with another array. For example, we want to index an array a with a boolean mask. What we get out, is that only the elements that line up with True in that mask will be returned:
# Create an array
arr = np.array([1, 2, 3, 4, 5])
print(arr)
# Create a boolean mask
mask = np.array([False, False, True, True, True])
print(arr[mask])
Out:
array([1, 2, 3, 4, 5])

array([3, 4, 5])
Here we only get [3 4 5]
, because these values line up with True in the mask.
Now you might be a little confused and wonder why we would ever need that? Where masks become useful is when they are combined with ufuncs. For example, we want to get the elements from an array based on some conditions. We can index that array using a mask with the conditions:
# Create an array
arr = np.array([1, 2, 3, 4, 5])
print(arr)
# Create a mask
mask = (arr % 2 ==0)  (arr > 4)
print(arr[mask])
Out:
array([1, 2, 3, 4, 5])

array([2, 4, 5])
And the values that meet the criteria will be returned.
Fancy indexing
The idea of fancy indexing is very simple. It allows us to access multiple array elements at once by passing an array of indices.
For example, we get the 0th and 1st element from array arr
by passing another array of the indices:
arr = np.array([1, 2, 3, 4, 5])
index_to_select = [0, 1]
print(arr[index_to_select])
Out:
array([1, 2])
Let’s combine all these together:
 Access multidimensional arrays by
[row, column]
:
# Create a 2x3 array
arr = np.arange(6).reshape(2, 3)
print(arr)
# Get row 0 and column 1
print(arr[0, 1])
# Get all rows in column 1 (Mixing slices and indices)
arr[:, 1]
Out:
array([[0, 1, 2],
[3, 4, 5]])

1

array([1, 4])
 Masking multidimensional arrays:
# Create a 2x3 array
arr = np.arange(6).reshape(2, 3)
print(arr)
# Masking the entire array
mask = abs(arr3)<2
print(arr[mask])
Out:
array([[0, 1, 2],
[3, 4, 5]])

array([2, 3, 4])
 Mixing masking and slicing:
arr = np.arange(6).reshape(2, 3)
print(arr)
mask = arr.sum(axis=1) > 4
print(arr[mask, 1:])
Out:
array([[0, 1, 2],
[3, 4, 5]])

array([[4, 5]])
All of these NumPy operations can be combined in nearly limitless ways!
Conclusion
No matter how large your dataset is or how complex are the operations you need to perform with your data, NumPy functions are there to make it easier and faster.
Let’s summarize what we’ve covered:
 Writing Python is fast, but loops are slow for large dataset computing.

NumPy is a great tool to use! It pushes loops into its compiled layer, so we get:
 Faster development time, and
 Faster execution time.

Strategies for speeding up your code:
 Ufuncs  For element wise operations.
 Aggregations  For array summarization.
 Slicing, masking and fancy indexing  For quickly selecting and operating on arrays.
Looking for the final source code!
References
 Python Course  NumPy Tutorial
 NumPy Documentation