splitting a sas program into multiple programs and finally merging output

Question

I created a file (metadata.sas7bdat) which has 100 entries with two columns directory and sasdatasets.

Directory                filename

/home                    abc.sas7bdat

/home                    def.sas7bdat

/home/sub_dir            ghj.sas7bdat

Assume I have sample.sas which just gets the means of each dataset.

Proc means data=abc.sas out=x;

Run;

The above sample.sas should run on each dataset present in metadata file. Right now I have written it in very traditional way of it loops through all the files in metadata and runs proc means on each file and appends all the data. The final dataset has the means of all the files present in metadata.

I believe if I can split the metadata file into 4 parts each having (100/4=25 entries) and submit it as 4 programs and finally merge the output from all the 4 programs It would reduce the processing time by large amount. ( think of 10,000 entries and also assume there is more processing than proc means). Its just I am not well versed with what kind of options to use to submit it as 4 programs and how to sync the output from 4 different processes.

Can you provide me the skeleton of how I should construct this program , I have my vague thoughts but I am sure I can take away some elegant answers .

Richard Richard · Accepted Answer · 2018-02-05T01:19:48

This question appears to be very similar to Is it possible to loop over SAS datasets?

Breaking into four parts will not reduce the overall processing time. For each part sure. If you think the overall there will be reduced processing, what tests or evidence is there supporting that premise ?

When there is a very large set of processing, P, broken into steps, p(1), p(2), ... p(N), you will need to construct a data structure that can store intermediate results, and rules for not repeating past processing when restarting the process after some p(i) experiences an error, or prior attempt at P stops at p(i)

Consider your case, you will need to store, or accumulate intermediate means with a step-key. The natural key would be the directory and filename.

A top-level dispatcher invokes a step macro for each item in the metadata. The step macro performs the process details when necessary and appends the results in an accumulating way.

Top-level: dispatch each step for the metadata items

%macro process_all (metadata=, results=);
  data _null_;
    set &metadata;

    invoke = cats('%process_step(libname=',libname,',memname=',memname,",results=&results)");
    put 'NOTE: ' _n_= invoke=;

    call execute ('%nrstr(' || trim(invoke) || ')');

    /* global parameter for checking testing 'restart' *

    %if %symexist(test_param_1) %then %do;
    if _n_ >= &test_param_1 then stop;
    %end;
  run;
%mend;

Step-level: core process for one metadata item. Skip step if done before, otherwise do actions and accumulate results.

%macro process_step(libname=, memname=, results=);

  %local have_results;
  %local step_done;

  %let have_results = %sysfunc(exist(&results,data));
  %let step_done = 0;

  %if &have_results %then %do;
    data _null_;
      set &results; where libname="&libname" and memname="&memname";
      call symput ('step_done', '1'); stop;
    run;
  %end;

  %if &step_done %then %do;
    %put NOTE: This step already done. &=libname &=memname;
    %return;
  %end;

  proc delete data=_step_out;
  proc means data=&libname..&memname noprint;
    var height weight;
    output out=_step_out(label="step output for &libname. . &memname.");
  run;

  data _step_out;
    length libname $8 memname $32.;
    set _step_out;
    libname = symget('libname');
    memname = symget('memname');
  run;

  proc append base=&results data=_step_out force;
  run;
%mend;

Test the scheme for various configurations of metadata

data _1 _2 _3 _4 _5 _6 _7 _8;
  set sashelp.class;
run;

data configuration_1(keep=libname memname);
  length libname $8 memname $32.;
  libname = 'work';
  do index = 1,2,3,7,8; memname='_'||cats(index); output; end;
run;

options mprint;

/*
* reset results to start from scratch
proc delete data=results_1;run;
*/

%let test_param_1 = 3; %* force stoppage after first three items;
%process_all(metadata=configuration_1, results=results_1)

%symdel test_param_1; %* force rerun of all to ignore first three already done items;
%process_all(metadata=configuration_1, results=results_1)

However you actually program the scheme you should be aware of and account for intermediate settings or data sets that might be left over from a prior run or step.

splitting a sas program into multiple programs and finally merging output

1 Answers