Eigen::Tensor and Eigen::Matrix performance comparison

Question

I would like to replace a sequence of matrices in my code with a single 3-D Eigen::Tensor. With this in mind, I try to compare Tensor and Matrix performances.

Function "tensorContractTest" below performs a contraction of (n,n,n) rank 3 tensor with a rank 1 tensor of size n (n = 500). This contraction computes n**2 dot products, so in terms of the number of operations, it is equivalent to multiplication of two (n,n) matrices (function "matrixProductTest" below).

When run on Visual Studio 2013, function "tensorContractTest" runs ~ 40 times slower than "matrixProductTest". Probably, I am missing something. Help is appreciated.

#include <unsupported/Eigen/CXX11/Tensor>
using namespace Eigen;

// Contracts 3-dimensional (n x n x n) tensor with 1-dimensional (n) tensor. 
// By the number of operations, it's equivalent to multiplication of 
// two (n, n) matrices (matrixProdTest).
Tensor<double, 2>  tensorContractTest(int n)
{
  Tensor<double, 3> a(n, n, n);     a.setConstant(1.);  
  Tensor<double, 1> b(n);           b.setConstant(1.);
  auto indexPair            = array<IndexPair<int>, 1>{IndexPair<int>(2,0)}; 
  Tensor<double, 2> result  = a.contract(b, indexPair); 
  return result;
}

MatrixXd  matrixProductTest(int n)
{ 
  MatrixXd a = MatrixXd::Ones(n, n), result = a * a;
  return result;
}

Did you enable optimizations? Please also provide a proper benchmark using, e.g. Google Benchmark, rather than just throwing numbers out there. — Henri Menke
I used /O2 compiler flag. My timing function is below. #include <chrono> #include <utility> #include <iostream> // Timing function typedef std::chrono::high_resolution_clock::time_point TimeVar; #define duration(a) std::chrono::duration_cast<std::chrono::nanoseconds>(a).count() #define timeNow() std::chrono::high_resolution_clock::now() template<typename F, typename... Args> double funcTime(F func, Args&&... args) { TimeVar t1 = timeNow(); func(std::forward<Args>(args)...); return duration(timeNow() - t1); } — lbo

ggael ggael · Accepted Answer · 2018-01-05T13:21:45

Even though the number of floating point operations are of same order, the memory access pattern are completely different and so the two operations are not comparable at all. Generally speaking matrix-matrix operation will always be faster (in terms of FLOPS) than matrix-vector or vector-vector operations because the former enable better cache usage and thus a nearly optimal usage of the CPU's ALU. In you case, on one side you have to read a n^3 tensor versus two n^2 matrices, so the memory footprints are not comparable at all.

Internally, Tensor::contract fallback to Eigen's matrix product kernels when possible, so performance should be in pair.

Eigen::Tensor and Eigen::Matrix performance comparison

2 Answers