lazy sequence evaluation timing issue

Question

I want to expand URLs from Twitter data and (simultaneously-ish) extract their domains. I tried doing this before in Python using requests, but I guess I screwed up somewhere, because the vast majority of the URLs are still in 'short' form (bit.ly, goo.gl, etc.) I've got the Twitter data stored as JSON. I'm using clj-http.client :as client to resolve the URLs. I've got code that looks like this, so far:

(defn expand-urls [urls] (for [url-str urls] 
                     (and url-str (last (:trace-redirects 
                                          (client/get url-str))))))

(def ^:dynamic *domain-pat* (re-pattern #"https?://([\w\.]+)/.*"))

(defn get-domains [urls] (for [url urls] (first (filter #(not= url %1) 
                           (re-find *domain-pat* url)))))

I've got the Twitter data formatted [tweet-id [{tweet-data-map} {user-data-map}]], so (get-in json-data [1 0 "urls"] returns the URLs, (get-in json-data [1 0 "domains"]) returns the domains.

When I try something like (update-in (update-in js-line [1 0 "urls"] expand-urls) [1 0 "domains"] get-domains), domains are (nil). I've independently verified that my regex works, so I suspect that the issue is that the lazy sequence returned by expand-urls isn't evaluating by the time get-domains gets called. Frustratingly, (type (doall (expand-urls some-urls))) returns clojure.lang.LazySeq, as does (type (doall (doall (expand-urls some-urls)))). I've tried doall, I've tried adding vec into expand-urls. Neither seem to work.

Is this really a laziness issue, or am I missing something else?

Are you binding the result of update-in? update-in does not mutate its argument, it returns a modified copy. Also, consider using (.getHost (java.net.URL. address)) instead of a regex. — noisesmith
The way to handle laziness is to use the result of the lazy function. It will remain a lazy seq in type whether realized (ie. via doall) or not. — noisesmith
Followup to noisesmith's comment: You can used realized? to test whether a lazy seq has been realized. — Mars
This doesn't all make sense to me yet. Here are some comments and questions that might be relevant. About your remark: 'When I try something like (update-in (update-in js-line [1 0 "urls"] expand-urls) [1 0 "domains"] get-domains), domains are (nil).', What does the word "domains" in that sentence refer to? Note that update-in does not modify its argument js-line; it returns a distinct structure that reflects the modifications you wanted. Don't expect update-in to change js-line. — Mars
In this kind of situation, if it were me, I would worry that I'm confused about the nested references that I'm trying to get update-in to handle. I would expand that into something more complicated using let for example, or play with it at the repl, and figure out the problem that way. That should make it easier to figure out what's going on, and you can put it back in the nested update-in form after that. At the very least, that would allow you to simplify the question to one that focuses on a more specific problem. — Mars

Michiel Borkent Michiel Borkent · Accepted Answer · 2014-06-12T19:43:39

You can rewrite your solution into

(defn expand-urls [urls] (
  (mapv #(last (:trace-redirects (client/get %))
    (remove nil? urls)))

assuming you don't want nils in your result. Mapv is the strict counterpart of the lazy map and always returns a vector.

lazy sequence evaluation timing issue

2 Answers