Adding REST resources to database. Struggling to loop through enumerated list and commit changes.

Question

I'm trying to extract data from an API that gives me data back in JSON format. I'm using SQLalchemy and simplejson in a python script to achieve this. The database is PostgreSQL.

I have a class called Harvest, it specifies the details for the table harvest.

Here is the code I suspect is incorrect.

def process(self):

            req = urllib2.Request('https://api.fulcrumapp.com/api/v2/records/', headers={"X-ApiToken":"****************************"})
            resp = urllib2.urlopen(req)
            data = simplejson.load(resp)

            for i, m in enumerate(data['harvest']):
                    harvest = Harvest(m)
                    self.session.add(harvest)
                    self.session.commit()

Is there something wrong with this loop? Nothing is going through to the database.

Can you check if you get data back! Are you sure the url you provide is correct. It might return an error in the json and you never see it. Have you debugged it? — ipinak

Chris Travers Chris Travers · Accepted Answer · 2013-03-20T03:57:11

I suspect that if there is anything wrong with the loop, it is that the loop is getting skipped. One thing you can do to verify this is:

ALTER USER application_user SET log_statements='all';

Then the statements will show up in your logs. When you are done:

ALTER USER application_user RESET log_statements;

This being said one thing I see in your code that may cause trouble later is the fact that you are committing per line. This will cause extra disk I/O. You probably want to commit after the loop.

Adding REST resources to database. Struggling to loop through enumerated list and commit changes.

1 Answers