How to crawl a feed

Question

My application needs to keep track of RSS/Atom feeds and save the new entries in a database. My question is, What is the most reliable method to determine whether an entry in a feed has already been crawled or not?

I use Universal Feed Parser module to parse the feeds. My current implementation keeps record of the latest value of feed.entry[i].updated_parsed, when crawling if updated_parsed value of an entry is greater than the recorded value, then that entry is saved in the database. The problem here is that many feeds don't have a published date or an updated date.

Answer 1

You should be determining whether you've already crawled an entry by reference to its <guid> primarily (falling back to <link> in the absence of a <guid>), and anything to do with dates only as a secondary analysis.