Can I have Eigen::VectorXd with size larger than the size limit (2^32)?

Question

I'm try to initialize a VectorXd with a size 60,000,000,000. When running my software, I get the following error because of the Vector size.

Any recommendation to solve the problem?

a.out: /usr/local/EasyBuild/software/Eigen/3.3.3-intel-2018a/include/Eigen/src/Core/PlainObjectBase.h:312: void Eigen::PlainObjectBase::resize(Eigen::Index) [with Derived = Eigen::Matrix; Eigen::Index = long int]: Assertion `((SizeAtCompileTime == Dynamic && (MaxSizeAtCompileTime==Dynamic || size<=MaxSizeAtCompileTime)) || SizeAtCompileTime == size) && size>=0' failed. Aborted (core dumped)

do you have 450GBs of contiguous free RAM? If it's sparse data, you should use sparse data structures — PeterT
"Any recommendation to solve the problem?" My recommendation is overthinking your program logic because usually people don't need a vector of 60 billion elements. What are you trying to do? If you post your code and a description of the problem that you want to solve with it, maybe we can recommend a better approach. — Blaze
Unless you have sufficient memory, this should only result in a bad-alloc: godbolt.org/z/KoNMjE. If you get the assertion above, you are doing something weird. Post a minimal reproducible example! — chtz
Thanks all for your help. Actually, the 60 billion elements are the non-zero elements from a huge sparse matrix. I think I'll save them in a matrix an access their rows and columns from other matrices. — user498036

PeterT PeterT · Accepted Answer · 2019-07-16T07:22:47

The value "Dynamic" is defined in Core/util/Constants.h to be

//Changing the value of Dynamic breaks the ABI, as Dynamic is often used as a template parameter for Matrix.
const int Dynamic = -1;

So you might get rid of the specific assertion by redefining it as a larger type such as

const long Dynamic = -1;

However, the template arguments are specified as int all over Eigen, so you might need to change it in a lot of places. Also, you should take note of the comment above. You'll need to make sure to compile everything that you link against with the same patched Eigen version.

Also, you'll need an unreasonable amount of RAM or swap space to make that program run.

Can I have Eigen::VectorXd with size larger than the size limit (2^32)?

1 Answers