Switch Application to Unicode

1
votes

My model.py looks like:

import csv

with open("organizationTest.txt","rU") as f:
        reader = csv.reader(f)
        for row in reader:
            _, created = Company.objects.get_or_create(
                Name=row[0],
                Site=row[1],
                )

"Name" and "Site" are both described in the model!

Whenever I try and run my server I get

django.db.utils.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

How can I switch my application to Unicode strings as it's said or if there is any other way to fix this issue I would love to know!

I would be happy to answer any questions! Thank you in advance!

EDIT: Here is the full error:

Traceback (most recent call last): File "manage.py", line 10, in execute_from_command_line(sys.argv) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/management/init.py", line 338, in execute_from_command_line utility.execute() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/management/init.py", line 312, in execute django.setup() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/init.py", line 18, in setup apps.populate(settings.INSTALLED_APPS) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate app_config.import_models(all_models) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models self.models_module = import_module(models_module_name) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "/Users/aghodsib/Desktop/soroush_programming/Python/slik/companies/models.py", line 58, in description=row[15], File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 405, in get_or_create return self.get(**lookup), False File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get num = len(clone) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 144, in len self._fetch_all() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all self._result_cache = list(self.iterator()) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator results = compiler.execute_sql() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 837, in execute_sql cursor.execute(sql, params) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute return super(CursorDebugWrapper, self).execute(sql, params) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/utils.py", line 97, in exit six.reraise(dj_exc_type, dj_exc_value, traceback) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 318, in execute return Database.Cursor.execute(self, query, params) django.db.utils.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

3
pythondjangopython-2.7unicodedjango-models
That error isn't coming from Django itself. Please post the full traceback so we can see the issue in context. – Daniel Roseman
I have added the full error! – Soroush Ghodsi
try Name=row[0].decode('UTF-8') – Rolf of Saxony
I get: "UnicodeDecodeError: 'utf8' codec can't decode byte 0xcc in position 0: invalid continuation byte" – Soroush Ghodsi
In that case Name=row[0].decode('cp1252') – Rolf of Saxony

3 Answers

2
votes

You appear to be getting characters from an unusual codec.
The source of the csv file should be able to tell you what they are using.
However, you can shift from one codec to another like this:

import csv

with open("organizationTest.txt","rU") as f:
        reader = csv.reader(f)
        for row in reader:
            _, created = Company.objects.get_or_create(
                Name=row[0].decode('latin-1').encode('utf8'),
                Site=row[1].decode('latin-1').encode('utf8'),
                )

I suggest that you give that a try. If it still doesn't work contact the creator of the csv file.

2
votes

From the Python CSV docs:

The csv module doesn’t directly support reading and writing Unicode, but it is 8-bit-clean save for some problems with ASCII NUL characters. So you can write functions or classes that handle the encoding and decoding for you as long as you avoid encodings like UTF-16 that use NULs. UTF-8 is recommended.

unicode_csv_reader() below is a generator that wraps csv.reader to handle Unicode CSV data (a list of Unicode strings).

import csv

def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs):
    # csv.py doesn't do Unicode; encode temporarily as UTF-8:
    csv_reader = csv.reader(utf_8_encoder(unicode_csv_data),
                            dialect=dialect, **kwargs)
    for row in csv_reader:
        # decode UTF-8 back to Unicode, cell by cell:
        yield [unicode(cell, 'utf-8') for cell in row]

def utf_8_encoder(unicode_csv_data):
    for line in unicode_csv_data:
        yield line.encode('utf-8')


with open("organizationTest.txt","rU") as f:
        reader = unicode_csv_reader(f)
        for row in reader:
            _, created = Company.objects.get_or_create(
                Name=row[0],
                Site=row[1],
                )
1
votes

try this:

import csv

with open("organizationTest.txt","rU") as f:
        reader = csv.reader(f)
        for row in reader:
            _, created = Company.objects.get_or_create(
                Name=unicode(row[0]),
                Site=unicode(row[1]),
                )