Speedup GPU vs CPU for matrix operations

Question

I am wondering how much GPU computing would help me speed up my simulations.

The critical part of my code is matrix multiplication. Basically the code looks like the following python code with matrices of order 1000 and long for loops.

import numpy as np
m_size = 1000
sim_length = 50

a = np.random.rand(m_size, m_size)
b = np.random.rand(m_size, m_size)

for j in range(sim_length):
    result = np.dot(a,b)

Note: My matrices are dense, mostly random and for loops are compiled with cython.

My naive guess would be that I have two factors:

More parallel threads (Currently of order 1 thread, GPUs of order 100 threads?) --> Speedup of order 100? [Source is quite outdated, from 2011]
Lower processor frequency (Currently 3Ghz, GPUs typically 2 Ghz) --> Neglect

I expect that this viewpoint is to naive, so what am I missing?

I suggest a good starting point would be PyCUDA if your GPU allows it. Great library that I've used in the past with numpy. — sethmlarson

kangshiyin kangshiyin · Accepted Answer · 2016-08-01T17:02:30

Matrix multiplication performance

If you use numpy, you are probably using one of the BLAS libraries as computational backend, such as ATLAS, OpenBLAS, MKL, etc. When you are using the fastest one MKL, you can find a recent performance benchmark here, between a recent Nvidia GPU K40m and Intel Xeon 12-core E5-2697 v2 @ 2.70GHz

https://developer.nvidia.com/cublas

where K40m is 6x faster than 12-thread E5-2697. Considering MKL scales well on multi-core CPU. K40m is ~72x faster than 1-thread E5-2697. Please also note 1000-dim is almost the lower bound to fully utilise both the GPU and CPU. Smaller matrix size usually leads to more performance degrade on GPU.

If you are using slower BLAS backend for numpy, say the GNU-licensed ATLAS. You could then find the comparison between MKL and ATLAS here

https://software.intel.com/en-us/intel-mkl/benchmarks#DGEMM-ATLAS

where MKL is 2~4x faster than ATLAS.

For Nvidia GPUs, the only widely used backend is CUDA's cuBLAS, so the performance won't change a lot like ATLAS vs. MKL.

Data transfer

As @janbrohl says, data transfer between host RAM and GPU device memory is an important factor that affect the overall performance. Here's a benchmark of the data transfer speed.

CUDA - how much slower is transferring over PCI-E?

Given the matrix size, you can actually calculate out the absolute time for computation and data transfer, respectively. These could help you evaluate the performance better.

To maximise the performance on GPU, you probably need re-design you program to minimise the data transfer, by moving all the computational operations to GPU, rather than matrix multiplication only.

Speedup GPU vs CPU for matrix operations

3 Answers

Matrix multiplication performance

Data transfer