VSAM Search VS COBOL search/loop

0
votes

I have a file that could contain about 3 million records. Certain records of this file will need to be updated multiple times throughout the run of the program. If I need to pull specific records from this file, which of the following is more efficient:

  1. Indexed VSAM search
  2. Indexed flat file with a COBOL search all
  3. Buffering all of the data into working storage and writing a loop to handle the search
2
performancecoboljclvsam
You have some good advice in the two answers already. However, what any specific answer to your question is is unclear, as there is not enough detail. It is unusual to want to update the same KSDS record more than once from the same batch program. You are missing the "two-file-match", where you have both sets of data in the same key order and read them "in step". At quite low levels of "hits" across your KSDS (even without updates) this could be the way to go. But... "know your data" is indispensable. Bad code/bad idea is the usual performance problem. - Bill Woodger

2 Answers

5
votes

Obviously, if you can buffer all of the data into memory (and if the host system can support a working-set of pages which is big enough to allow all of it to actually remain in RAM without paging, then this would probably be the fastest possible approach.

But, be very careful to consider "hidden disk-I/O" caused by the virtual-memory paging subsystem! If the requested "in-memory" data is, in fact, not "in memory," a page-fault will occur and your process will stop in its tracks until the page has been retrieved. (And if "page stealing" occurs, well, you're in trouble. Your "in-memory" strategy just turned into a possibly very-inefficient(!) disk-based one. If keys are distributed randomly, then your process has a gigantic working-set that it is accessing randomly. If all of that memory is not actually in memory, and will stay there, you're in trouble.

If you are making updates to a large file, consider sorting the updates-delta file before processing it, so that all occurrences of the same key will be adjacent. You can now write your COBOL program to take advantage of this (and, of course, to abend if an out-of-sequence record is ever detected!). If the key in "this" record is identical to the key of the "previous" one, then you do not need to re-read the record. (And, you do not actually need to write the old record, until the key does change.) As the indexed-file access method is presented with the succession of keys, each key is likely to be "close to" the one previously-requested, such that some of the necessary index-tree pages will already be in-memory. Obviously, you will need to benchmark this, but the amount of time spent sorting the file can be far less than the amount of time spent in index-lookups. (Which actually can be considerable.)

4
votes

The answer of Mike has the important issue about "hidden I/O" in (depends on the machine, configuration, amount of data)...

If you very likely need to update many records the option Mike suggest is the most useful one.

If you very likely need to update not much records (I'd guess you're likely below 2%) another approach can be quite faster (needs a benchmark !):

  • read every key via indexed VSAM search
  • store the changed record in memory (big occurs table), if you will only change some values and the record is quite big then only store all possible changed values + key in the table without an actual REWRITE
  • before doing a VSAM search: look in your occurs table if you read the key already, take the values either from there or get a new one
  • ...
  • at program end: go through your occurs and REQRITE all records (if you have the complete record a REWRITE is enough, otherwise you'd need a READ first to get the complete record)

Performance is often: "know your data and possible program flow, then try the best 2-3 approach, benchmark and decide".