Edit generated 'importdata' function to import all files in directory in Matlab

Question

Seeking help from skillful Matlab users!

I'm kind of new to Matlab and hope somebody has the time to help me. I need to import some .txt-files from a directory. I have found a way to do this trough the import tool. There are some data using comma insted of dots, so importdata will not work, but the 'import data' tool does.

So i'm wondering (and hoping) if it is possible to edit the generated function to import all the files in the directory, in such a way as the single file is imported? I want each file to be imported as matrix variable (double). I want to import all the files in one process (loop). Also there are many files and they all have some 100 000 lines or so.

If someone see an easy way to do this i would appreciate the help. Please keep the explanation on a low level, as i'm quite novice. I get the following function using the 'import data' tool:

    function Streaming0x00x00158D00000E04621709201405 = importfile1(filename, startRow, endRow)
%IMPORTFILE1 Import numeric data from a text file as a matrix.
%   STREAMING0X00X00158D00000E04621709201405 = IMPORTFILE1(FILENAME) Reads
%   data from text file FILENAME for the default selection.
%
%   STREAMING0X00X00158D00000E04621709201405 = IMPORTFILE1(FILENAME,
%   STARTROW, ENDROW) Reads data from rows STARTROW through ENDROW of text
%   file FILENAME.
%
% Example:
%   Streaming0x00x00158D00000E04621709201405 =
%   importfile1('Streaming_0_x_0_0_x_00158D00000E0462_17-09-2014_05.32.24_part000.txt',
%   17, 137834);
%
%    See also TEXTSCAN.

% Auto-generated by MATLAB on 2015/02/04 09:28:07

%% Initialize variables.
delimiter = ';';
if nargin<=2
    startRow = 17;
    endRow = inf;
end

%% Read columns of data as strings:
% For more information, see the TEXTSCAN documentation.
formatSpec = '%s%s%[^\n\r]';

%% Open the text file.
fileID = fopen(filename,'r');

%% Read columns of data according to format string.
% This call is based on the structure of the file used to generate this
% code. If an error occurs for a different file, try regenerating the code
% from the Import Tool.
textscan(fileID, '%[^\n\r]', startRow(1)-1, 'ReturnOnError', false);
dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter', delimiter, 'ReturnOnError', false);
for block=2:length(startRow)
    frewind(fileID);
    textscan(fileID, '%[^\n\r]', startRow(block)-1, 'ReturnOnError', false);
    dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, 'Delimiter', delimiter, 'ReturnOnError', false);
    for col=1:length(dataArray)
        dataArray{col} = [dataArray{col};dataArrayBlock{col}];
    end
end

%% Close the text file.
fclose(fileID);

%% Convert the contents of columns containing numeric strings to numbers.
% Replace non-numeric strings with NaN.
raw = repmat({''},length(dataArray{1}),length(dataArray)-1);
for col=1:length(dataArray)-1
    raw(1:length(dataArray{col}),col) = dataArray{col};
end
numericData = NaN(size(dataArray{1},1),size(dataArray,2));

for col=[1,2]
    % Converts strings in the input cell array to numbers. Replaced non-numeric
    % strings with NaN.
    rawData = dataArray{col};
    for row=1:size(rawData, 1);
        % Create a regular expression to detect and remove non-numeric prefixes and
        % suffixes.
        regexstr = '(?<prefix>.*?)(?<numbers>([-]*(\d+[\.]*)+[\,]{0,1}\d*[eEdD]{0,1}[-+]*\d*[i]{0,1})|([-]*(\d+[\.]*)*[\,]{1,1}\d+[eEdD]{0,1}[-+]*\d*[i]{0,1}))(?<suffix>.*)';
        try
            result = regexp(rawData{row}, regexstr, 'names');
            numbers = result.numbers;

            % Detected commas in non-thousand locations.
            invalidThousandsSeparator = false;
            if any(numbers=='.');
                thousandsRegExp = '^\d+?(\.\d{3})*\,{0,1}\d*$';
                if isempty(regexp(thousandsRegExp, '.', 'once'));
                    numbers = NaN;
                    invalidThousandsSeparator = true;
                end
            end
            % Convert numeric strings to numbers.
            if ~invalidThousandsSeparator;
                numbers = strrep(numbers, '.', '');
                numbers = strrep(numbers, ',', '.');
                numbers = textscan(numbers, '%f');
                numericData(row, col) = numbers{1};
                raw{row, col} = numbers{1};
            end
        catch me
        end
    end
end


%% Replace non-numeric cells with NaN
R = cellfun(@(x) ~isnumeric(x) && ~islogical(x),raw); % Find non-numeric cells
raw(R) = {NaN}; % Replace non-numeric cells

%% Create output variable
Streaming0x00x00158D00000E04621709201405 = cell2mat(raw);

If something is unclear, please comment.

All help is useful, thanks :)

Dave Kielpinski Dave Kielpinski · Accepted Answer · 2015-02-04T10:48:03

If all files are the same, you can make a cell array of the filenames (NOT a standard array, they do not behave correctly on strings). Then you can loop over the cell array. For instance:

fname_arr = {'file1.txt','file2.txt'}; % your filenames go here
for k in length(fname_arr):
    filename = fname_arr{k};
    %% Open the text file.
    fileID = fopen(filename,'r');  % start of the relevant part of your codeblock
    <...>  % omitting the stuff in the middle of the code
    fclose(fileID) % end of the relevant part of your codeblock
    allDataArray{k} = DataArray
end

Then allDataArray is a cell array whose kth element contains the DataArray obtained from file fname_arr{k}.

Edit generated 'importdata' function to import all files in directory in Matlab

2 Answers