Reading structured variable from MAT file

Question

I am performing an analysis which involves simulation of over 1000 cases. I extracting lots of data for each case as well (about 70MB). Currently I am saving the results for each case as:

Vessel.TotalForce
Vessel.WindForce
Vessel.CurrentForce
Vessel.WaveForce
Vessel.ConnectionForce
...

Line1.EffectiveTension
Line1.X
Line1.Y

Line2.EfectiveTension
Line2.X
Line2.Y
...

save('CaseNo1.mat')

Now, I need to perform my analysis for CaseNo1.mat to CaseNo1000. Initially I planned to create a Database.mat file by loading all cases in it and then accessing any variable using h5read. This way Matlab doesn't need to load all the data at a time. However, I am concerned now that my database file will be too big.

Is there any way I can read the structured variables from individual case files for example CaseNo1.mat without loading the CaseNo1.mat file in memory.

Matlab examples shows loading just the variables directly from MAT file without loading the whole MAT file. But I am not sure how to read structures data the same way.

x=load('CaseNo1.mat','Line1.X')

says Line1.X not found. But it's there. The command is not correct to access the data. Also tried using h5read, but it says CaseNo1.mat is not an HDF5 file.

Can anyone help with this.

Apart from this, I would also appreciate if there is any suggestion about performing such data intensive analysis.

Answer 1

I was wrong! I'm leaving my old answer for context, though I've edited it to reference this one. I thought I had used matfile() in that way before, but I hadn't. I just did a thorough search and ran a few test cases. You've actually run into a limitation of the way Matlab handles and references structures stored in .mat files. There is, however, a solution. It does involve some refactoring of your original code, but it shouldn't be too egregious.

Vessel_TotalForce 

Vessel_WindForce 

Vessel_CurrentForce 

Vessel_WaveForce

Vessel_ConnectionForce 

... 

Line1_EffectiveTension 

Line1_X 

Line1_Y 

Line2_EfectiveTension 

Line2_X 

Line2_Y 

... 

save('CaseNo1.mat')

Then to access, just use matfile (or load) as you were before. Like so:

Vessel_WaveForce = load('CaseNo1.mat'', 'Vessel_WaveForce')

It's important to note that this restriction doesn't appear to be caused by anything you've chosen to do in your program, but rather is imposed by the way Matlab interacts with it's native storage files when they contain structures.

Answer 2

EDIT: This answer works, but doesn't actually solve the problem posed in OP's question. I thought I had used matfile to generate a handle that I could access, but I was wrong. See my other answer for details.

You could use matfile, like so:

myMatFileHandle = matfile('caseNo1.mat');
thisVessel = myMatFileHandle.vessel;

Also, from the little bit I can see, you seem to be on the right track for high-volume analysis. Just remember to use sparse when applicable, and generally avoid conditionals inside of loops if possible.

Good luck!

Answer 3

The objective of storing data in structured format is:

1. To be organized
2. Easy scripting post processor where looping through data under one data set it required.

To store structured dataset containing integer, floating and string variables in MAT file and to be able to read just the required variable using h5read command was sought. Matlab load command is not able to read variable beyond first level from stored data in a MAT file. The h5write couldn't write string variables. Hence needed a work around to solve this problem.

To do this I have used following method:

filename = 'myMatFile';
Vessel.TotalForce = %store some data
Vessel.WindForce = %store some data
Vessel.CurrentForce = %store some data
Vessel.WaveForce = %store some data
Vessel.ConnectionForce = %store some data
...

Lin1.LineType = 'Wire'
Line1.ArcLength_0.EffectiveTension = %store some data
Line1.ArcLength_50.EffectiveTension= %store some data
Line1.ArcLength_100.EffectiveTension= %store some data

Lin2.LineType = 'Chain'
Line2.ArcLength_0.EffectiveTension= %store some data
Line2.ArcLength_50.EffectiveTension= %store some data
Line2.ArcLength_100.EffectiveTension= %store some data

save([filename '_temp.mat']);

PointToMat=matfile([filename '.mat'],'Writable',true);
PointToMat.(char(filename)) = load([filename '_temp.mat']);

delete([filename '_temp.mat']);

Now to read from the MAT file created, we can use h5read as usual. To extract the EffectiveTension for Line1, ArcLength_0:

EffectiveTension = h5read([filename '.mat'],['/' filename '/Line1/ArcLength_0/EffectiveTension']);

For string variables, h5read returns decimal values corresponding to each character. To obtain the actual string I used: name = char(h5read([filename '.mat'],['/' filename '/Line1/LineType']));

Tried this method on my data set which is about 200MB and I could process them pretty fast. Hope this would help someone someday.

Answer 4

Short answer: Having saved the data into a MAT file with the '-v7.3' option, use something like h5read(filename, '/Line2/X') to read just one structure field. You can even read an array partially, for example:

s.a = 1:100;
save('test.mat', '-v7.3', 's');
clear
h5read('test.mat', '/s/a', [1 10], [1 5], [1 3])

returns each third element of the 1:100 array, starting with the 10th element and returning 5 values:

 10    13    16    19    22

Long answer: See answer by @Amitava for the more elaborate code and topic coverage.