How to realize a lazy seq without retaining the head, when you need to process the seq twice?

Question

When a function is given a large lazy seq, it is beneficial to avoid retaining the head, so that in case the fully realized sequence does not fit in memory, you can still process it. For example, this works fine:

(count (take 10000000 (range)))
(reduce + (take 10000000 (range)))

But this can generate an out of memory error:

(defn recount [coll n]
  [(count (take n coll))
   (reduce + (take n coll))])
(recount (range) 10000000)

because the binding of coll retains the head of the sequence when count realizes the lazy seq.

The closest thing I can come up with is a macro that forces the seq to be reevaluated instead of bound:

(defmacro recount4 [coll n]
  `[(count (take ~n ~coll))
    (reduce + (take ~n ~coll))])
(recount4 (range) 10000000)

This does not seem to be broadly applicable.

I looked at this blog, but the solution is less than satisfying due to the use of atoms and mutable state.

Answer 1

You might want to look at eduction - it creates a delayed, non-cached sequential collection and will re-evaluate the full collection on each use via reduction.

Answer 2

What you want is eduction.

It allows for iteration or reduction over a collection. It works like a collection for most purposes, but does not realize an underlying lazy seq.

count does not work on eductions, though, so we have to rewrite count as a reduction.

(defn recount5 [coll n]
  (let [s (eduction (take n) coll)]
    [(reduce (fn [r x] (inc r)) 0 s)
     (reduce + s)]))
(recount5 (range) 10000000)