open txt file using read_csv by pandas

Question

I am trying to process txt file using pandas.
However, I get following error at read_csv

CParserError Traceback (most recent call last) in () 22 Col.append(elm) 23 ---> 24 revised=pd.read_csv(Path+file,skiprows=Header+1,header=None,delim_whitespace=True) 25 26 TimeSeries.append(revised)

C:\Users\obakatsu\Anaconda3\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision) 560 skip_blank_lines=skip_blank_lines) 561 --> 562 return _read(filepath_or_buffer, kwds) 563 564 parser_f.name = name

C:\Users\obakatsu\Anaconda3\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds) 323 return parser 324 --> 325 return parser.read() 326 327 _parser_defaults = {

C:\Users\obakatsu\Anaconda3\lib\site-packages\pandas\io\parsers.py in read(self, nrows) 813 raise ValueError('skip_footer not supported for iteration') 814 --> 815 ret = self._engine.read(nrows) 816 817 if self.options.get('as_recarray'):

C:\Users\obakatsu\Anaconda3\lib\site-packages\pandas\io\parsers.py in read(self, nrows) 1312 def read(self, nrows=None): 1313
try: -> 1314 data = self._reader.read(nrows) 1315 except StopIteration: 1316 if self._first_chunk:

pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:8748)()

pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:9003)()

pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:9731)()

pandas\parser.pyx in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:9602)()

pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:23325)()

CParserError: Error tokenizing data. C error: Expected 4 fields in line 6, saw 8

Does anyone know how I can fix this problem?
My python script and example txt file I want to process is shown below.

Path='data/NanFung/OCTA_Tower/test/'
files=os.listdir(Path)
TimeSeries=[]
Cols=[]
for file in files:
    new=open(Path+file)
    Supplement=[]
    Col=[]
    data=[]
    Header=0
    #calculate how many rows should be skipped
    for line in new:
        if line.startswith('Timestamp'):
            new1=line.split(" ")
            new1[-1]=str(file)[:-4]
            break
        else:
            Header += 1      

    #clean col name
    for elm in new1:
        if len(elm)>0:
            Col.append(elm)

    revised=pd.read_csv(Path+file,skiprows=Header+1,header=None,delim_whitespace=True)
    TimeSeries.append(revised) 
    Cols.append(Col)

txt file

history:/NIKL6215_ENC_1/CH$2d19$2d1$20$20CHW$20OUTLET$20TEMP
20-Oct-12 8:00 PM CT  to  ?

Timestamp                  Trend Flags  Status  Value (ºC)
-------------------------  -----------  ------  ----------
20-Oct-12 8:00:00 PM HKT   {start}      {ok}    15.310 ºC 
21-Oct-12 12:00:00 AM HKT  { }          {ok}    15.130 ºC

Hello, John. I have edited the question including full traceback — Katsuya Obara

John Zwinck John Zwinck · Accepted Answer · 2017-06-25T02:46:20

It fails because the part of the file you're reading looks like this:

Timestamp                  Trend Flags  Status  Value (ºC)
-------------------------  -----------  ------  ----------
20-Oct-12 8:00:00 PM HKT   {start}      {ok}    15.310 ºC 
21-Oct-12 12:00:00 AM HKT  { }          {ok}    15.130 ºC

But there are no consistent delimiters here. read_csv does not understand how to read fixed-width formats like yours. You might consider using a delimited file, such as with tab characters between the columns.

open txt file using read_csv by pandas

2 Answers

Include This line before