Clojure stack overflow using recur, lazy seq?

Question

I've read other people's questions about having stack overflow problems in Clojure, and the problem tend to be a lazy sequence being built up somewhere. That appears to be the problem here, but for the life of me I can't figure out where.

Here is the code and after the code is a bit of explanation:

(defn pare-all []
  "writes to disk, return new counts map"
(loop [counts (counted-origlabels)
     songindex 0]
(let [[o g] (orig-gen-pair songindex)]
  (if (< songindex *song-count*) ;if we are not done processing list
    (if-not (seq o) ;if there are no original labels
      (do
        (write-newlabels songindex g);then use the generated ones
        (recur counts (inc songindex)))
      (let [{labels :labels new-counts :countmap} (pare-keywords o g counts)] ;else pare the pairs
        (write-newlabels songindex labels)
        (recur new-counts (inc songindex))))
    counts))))

There is a map stored in "counts" originally retrieved from the function "counted-origlabels". The map have string keys and integer values. It is 600 or so items long and the values are updated during the iteration but the length stays the same, I've verified this.

The "orig-gen-pair" function reads from a file and returns a short pair of sequences, 10 or so items each.

The "write-newlabels" function just rite the passed sequence to the disk and doesn't have any other side effect nor does it return a value.

"Pare-keywords" returns a short sequence and an updated version of the "counts" map.

I just don't see what lazy sequence could be causing the problem here!

Any tips would be very much appreciated!

----EDIT----

Hello all, I've updated my function to be (hopefully) a little more idiomatic Clojure. But my original problem still remains. First, here is the new code:

(defn process-song [counts songindex]
  (let [[o g] (orig-gen-pair songindex)]
(if-not (seq o) ;;if no original labels
  (do
    (write-newlabels songindex g);then use the generated ones
    counts)
  (let [{labels :labels new-counts :countmap} (pare-keywords o g counts)] ;else pare the pairs
    (write-newlabels songindex labels)
    new-counts))))

(defn pare-all []
  (reduce process-song (counted-origlabels) (range *song-count*)))

This still ends with java.lang.StackOverflowError (repl-1:331). The stack trace doesn't mean much to me other than it sure seems to indicate lazy sequence mayhem going on. Any more tips? Do I need to post the code to the functions that process-song calls? Thanks!

animal animal · Accepted Answer · 2011-04-15T22:53:07

I cannot quite grasp what you are trying to do without a little more concrete sample data, but it's very evident you're trying to iterate over your data using recursion. You're making things way more painful on yourself than you need to.

If you can generate a function, let's call it do-the-thing, that operates correctly with a single entry in your map, then you can call (map do-the-thing (counted-origlabels)), and it will apply (do-the-thing) to each map entry in (counted-origlabels), passing a single map entry to do-the-thing as it's sole argument and returning a seq of the return values from do-the-thing.

You also look like you need indexes, this is easily solved as well. You can splice in the lazy sequence (range) as the second argument to do-the-thing, and then you'll have a series of indexes generated with each map entry; however maps in clojure are not sorted by default, so unless you are using a sorted map, this index value is relatively meaningless.

Trying to abstract away what you've writen so far, try something like:

(defn do-the-thing [entry index counts]
  (let [[o g] (orig-gen-pair index)]
    (if-not (seq o)
      (write-newlabels index g)
      (let [{labels :labels new-counts :countmap} (pare-keywords o g counts)]
        (write-newlabels index labels)))))

(map do-the-thing (counted-origlabels) (range) (constantly (counted-origlabels)))

Clojure stack overflow using recur, lazy seq?

1 Answers