Loop through CSV file and create new csv file with while read?

Question

I have

while read $field1 $field2 $field3 $field4
do
  $trimmed=$field2 | sed 's/ *$//g'
  echo "$trimmed","$field3" >> new.csv
done < "$FEEDS"/"$DLFILE"

Now the problem is with read I can't make it split fields csv style, can I? See the input csv format below.

I need to get columns 3 and 4 out, stripping the padding from col 2, and I don't need the quotes.

Csv format with col numbers: 12 24(")25(,)26(")/27(Field2values) 42(")/43(,)/44(Field3 decimal values) "Field1_constant_value","Field2values ",Field3,Field4

Field1 is constant and irrelevant. Data is quoted, goes from 2-23 inside the quotes. Field2 fixed with from cols 27-41 inside quotes, with the data at the left and padded by spaces on the right. Field3 is a decimal number with 1,2, or 3 digits before the decimal and 2 after, no padding. Starts at col 74. Field4 is a date and I don't much care about it right now.

Rubens Rubens · Accepted Answer · 2012-11-17T20:08:20

Yes, you can use read; all you've got to do is reset the environment variable IFS -- Internal Field Separator --, so that it won't split lines by its current value (default to whitespace), but by your own delimiter.

Considering an input file "a.csv", with the given contents:

1,2,3,4

2,3,4,5

6,3,2,1

You can do this:

IFS=','
while read f1 f2 f3 f4; do
    echo "fields[$f1 $f2 $f3 $f4]"
done < a.csv

And the output is:

fields[1 2 3 4]

fields[2 3 4 5]

fields[6 3 2 1]

Loop through CSV file and create new csv file with while read?

2 Answers