Python: MySQLdb LOAD DATA INFILE silently fails

Question

I am attempting to use a Python script to import a csv file into a MySQL database. It seems to fail silently.

Here is my code:

#!/usr/bin/python

import MySQLdb

class DB:
    host = 'localhost'
    user = 'root'
    password = '**************'
    sqldb = 'agriculture'
    conn = None

    def connect(self):
        self.conn =  MySQLdb.connect(self.host,self.user,self.password,self.sqldb )

    def query(self, sql, params=None):
        try:
            cursor = self.conn.cursor()
            if params is not None:
                cursor.execute(sql, params)
            else:
                cursor.execute(sql)
        except (AttributeError, MySQLdb.OperationalError):
            self.connect()
            cursor = self.conn.cursor()
            if params is not None:
                cursor.execute(sql, params)
            else:
                cursor.execute(sql)

        print vars(cursor)
        return cursor

    def load_data_infile(self, f, table, options=""):
        sql="""LOAD DATA LOCAL INFILE '%s' INTO TABLE %s FIELDS TERMINATED BY ',';""" % (f,table)
        self.query(sql)


db = DB()

pathToFile = "/home/ariggi/722140-93805-sqltest.csv"
table_name = "agriculture.degreedays"

db.load_data_infile(pathToFile, table_name)

In an attempt to debug this situation I am dumping the cursor object to the screen within the "query()" method. Here is the output:

{'_result': None, 'description': None, 'rownumber': 0, 'messages': [], '_executed': "LOAD DATA LOCAL INFILE '/home/ariggi/722140-93805-sqltest.csv' INTO TABLE degreedays FIELDS TERMINATED BY ',';", 'errorhandler': >, 'rowcount': 500L, 'connection': , 'description_flags': None, 'arraysize': 1, '_info': 'Records: 500 Deleted: 0 Skipped: 0 Warnings: 0', 'lastrowid': 0L, '_last_executed': "LOAD DATA LOCAL INFILE '/home/ariggi/722140-93805-sqltest.csv' INTO TABLE agriculture.degreedays FIELDS TERMINATED BY ',';", '_warnings': 0, '_rows': ()}

If I take the "_last_executed" query, which is

LOAD DATA LOCAL INFILE '/home/ariggi/722140-93805-sqltest.csv' INTO TABLE agriculture.degreedays FIELDS TERMINATED BY ',';

and run it through the mysql console it works as expected and fills the table with rows. However when I execute this script my database table remains empty.

I am pretty stumped and could use some help.

holdenweb holdenweb · Accepted Answer · 2014-02-27T23:29:13

Try calling db.conn.commit() at the end of your code to make the changes permanent. Python by default does not use the "autocommit" mode, so until you issue a commit the DB module regards your changes as part of an incomplete transaction.

As @AirThomas points out in a comment it helps to us a "context manager" - though I'd say the correct formulation was

with conn.cursor() as curs:
    do_something_with(curs)

because this will automatically commit any changes unless the controlled code raises an exception.

Python: MySQLdb LOAD DATA INFILE silently fails

1 Answers