Use SAS to import CSV file with 3070 fields

0
votes

I am using AIX 5.3 and SAS 9.1.3

I have a 2.7G CSV file.
The file has about 110k lines.
Each line has 3070 fields.

I use filename myCSVfile LRECL=32768 ;
Then, I use Proc import ... delimiter = "," ...

The SAS program seems hang up.
It still there even after running 1 hour.

I head the file to have only 40 lines and try again.
This time, also seems hang up.
Also still there after running 1 hour.
NO improvement with this small file.

The log file is 0 byte.
Also, when I ps -ef , the process has eat 10% CPU.
At last I have to kill it before leaving the office.

Any hint to solve this problem ?

Alvin SIU

2
csvimportsas
Alvin, you may also want to cross-post this to www.runsubmit.com more SAS experts checking there regularly. – Jay Stevens
Also, post the actual code you're using and more information about the input file. – Jay Stevens
This morning, I run the import again without changing anything, It can import the 40 line files using 1 hour. So, the program is correct, just running very slow. – Alvin SIU

2 Answers

2
votes

Do you need to use proc import? This might work better:

data tmp;
  infile './your_data_file' dlm=',' lrecl=as_big_as_you_need;
  input var1 var2 ...;
run;

Running it this way will also allow you to inspect what errors crop up in the log file as they happen.

0
votes

Also try adding the truncover option? Check to make sure there aren't any lines > 32767 chars as that will cause issues. Do this check using an external program (ie. not SAS).

Cheers Rob