How To Turn a Reduce-Realized Sequence Back Into Lazy Vector Sequence

Question

When I run a 221 line .csv file -- parsed with clojure-csv -- into this function

(defn test-key-inclusion
    "Accepts csv-data param and an index, a second csv-data param and an index,
     and searches the second csv-data instances' rows (at index) to see if
     the first file's data is located in the second csv-data instance."

    [csv-data1 pkey-idx1 csv-data2 pkey-idx2 lnam-idx fnam-idx]

    (reduce
        (fn [out-log csv-row1]
            (let [cmp-val (nth csv-row1 pkey-idx1 nil)
                  lnam (nth csv-row1 lnam-idx nil)
                  fnam (nth csv-row1 fnam-idx)
                  temp-rc (first (key-pres? cmp-val pkey-idx2 csv-data2))]

            (concat out-log (sorted-map cmp-val (vector lnam fnam)))))
         {}
         csv-data1))

and then print the result, everything's fine.

If I run a 2672 line .csv file -- also parsed with clojure-csv -- through the function above and then try to print it, I get a stack overflow error -- Exception in thread "main" java.lang.StackOverflowError

So my questions are:

1) Should wrapping the call to this function inside lazy-seq cure my problem?

2) I don't want a list, so will wrapping the lazy-seq call inside a vec turn my sequence back into a vector without realizing the whole sequence in memory, that is make the lazy-seq un-lazy again?

Thank you.

do you want concat or merge? concat doesn't make much sense - you could map rather than reduce. reduce itself is lazy. i am very confused by this question... — andrew cooke
I want to take a bunch of csv rows, and return them -- reduced in column width -- to the same kind of data parsed by clojure-csv. I used concat instead of conj so the null row would be last. — octopusgrabbus
so why are you using reduce rather than map? it sounds like what you are going to end up with is rewriting map inside reduce (which is possible - reduce is more general - but not a good idea). — andrew cooke
I've never been able to get map to work in some of these situations; reduce always has worked. I may wind up going back and using it. — octopusgrabbus

Arthur Ulfeldt Arthur Ulfeldt · Accepted Answer · 2012-04-12T22:19:21

1) i expect that making the sequence lazy not to help because print will evaluate realize it before printing it. instead try doseq or (map print my-seq) to print it in smaller chunks.

2) yes wrapping it in vec will give you what you want :) though wrapping your reduce with an into would keep it a vector the whole time. ie: (reduce into [] [[1] [2] [3]] ) --> [1 2 3]

  (into out-log (sorted-map cmp-val (vector lnam fnam)))))

How To Turn a Reduce-Realized Sequence Back Into Lazy Vector Sequence

1 Answers