As far as I've researched the size of arrays that can be handled depends on the API you compile your mex files with. You can choose the API by adding the corresponding flag in your compiling instruction. The details are in the matlab documentation under "api-release specific API".
There are 4 options avaliable: -R2017b (default) -R2018a -largeArrayDims and -compatibleArrayDims.
In term of array size -R2017b (default) -R2018a and -largeArrayDims use the Large-array-handling API which according to the matlab mex documentation can handle arrays over 231-1 and, according to the API documentation should be able to handle arrays up to 248-1 elements and sparse arrays up to 248-2.
Only the last option, -compatibleArrayDims won't handle arrays above 231-1
Apart from array size those options will change the way a few data types are handled, noticeably complex types and graphics object.
So in short:
-R2017b (default) : 248-1 elements per array
-R2018a : 248-1 elements per array
-largeArrayDims : 248-1 elements per array
-compatibleArrayDims 231-1 elements per array
Finally if you want to handle larger object the solution I see would be to write your results in files (.txt or .csv for example) in the c-part of your code and read them back in the matlab part in whole or in chunks.
Hope this helped. I'm new in here so feel free to correct me if I made a mistake or disregarded the etiquette.