Cron Jobs, Task Queue or Deferred Tasks?

3
votes

In Google App Engine, for a task like scanning some RSS feeds and adding new entries from the feed to the datastore every 10-15 seconds, should I use Cron Jobs, Task Queue or Deferred Tasks? I'm really confused.

2
pythongoogle-app-enginetask
Why do you need to insert the data so often? - Adam Matan
@Adam Matan: What's wrong with that? - Alon Gubkin
Nothing wrong, just curious - it's quite uncommon to read an RSS feed so frequently, and it might get you blocked as if you're DDOS-ing the feed. - Adam Matan
Chances are whoever provides the RSS feed is going to block you - or possibly hunt you down and leave burning poo on your doorstep - for polling every 15 seconds. - Nick Johnson

2 Answers

0
votes

Invoke a cron job every 1 minute, which would get the RSS and sleep for 15 seconds four times. You can locks to prevent overlapping (although the database insertion provides some measure of concurrency control).

Python-like pseudo code:

if cant_get_lock:
    exit
else:
for i in (1,2,3,4):
    get RSS
    sleep 15 seconds
0
votes
  • I think if it happens every 15 seconds(not skipping) than I would cron jobs because that is easiest to implement. But if you need to be able cancel task then you should use task queue.
  • BTW you should use PubSubHubbub(hubbub) to receive updates for the feeds in realtime if I understand you correctly.