UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a in position 12

2
votes

I'm developing a chatbot with the chatterbot library. The chatbot is in my native language --> Slovene, which has a lot of strange characters (for example: š, č, ž). I'm using python 2.7.

When I try to train the bot, the library has trouble with the characters mentioned above. For example, when I run the following code:

chatBot.set_trainer(ListTrainer)
chatBot.train([
            "Koliko imam še dopusta?",
            "Letos imate še 19 dni dopusta.",
        ])

it throws the following error:

UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a in position 12: invalid start byte

I added the # -*- coding: utf-8 -*- line to the top of my file, I also changed the encoding of all used files via my editor (Sublime text 3) to utf-8, I changed the system default encoding with the following code:

import sys
reload(sys)
sys.setdefaultencoding('utf8')

The strings are of type unicode.

When I try to get a response, with these strange characters, it works, it has no issues with them. For example, running the following code in the same execution as the above training code(when I change 'š' to 's' and 'č' to 'c', in the train strings), throws no errors:

chatBot.set_trainer(ListTrainer)
chatBot.train([
            "Koliko imam se dopusta?",
            "Letos imate se 19 dni dopusta.",
        ])    
chatBot.get_response("Koliko imam še dopusta?")

I can't find a solution to this issue. Any suggestions? Thanks loads in advance. :)

EDIT: I used from __future__ import unicode_literals, to make strings of type unicode. I also checked if they really were unicode with the method type(myString)

I would also like to paste this link.

EDIT 2: @MallikarjunaraoKosuri - s code works, but in my case, I had one more thing inside the chatbot instance intialization, which is the following:

chatBot = ChatBot(
    'Test',
    trainer='chatterbot.trainers.ListTrainer',
    storage_adapter='chatterbot.storage.JsonFileStorageAdapter'
)

This is the cause of my error. The json storage file the chatbot creates, is created in my local encoding and not in utf-8. It seems the default storage (.sqlite3), doesn't have this issue, so for now I'll just avoid the json storage. But I am still interested in finding a solution to this error.

2
python-2.7encodingutf-8chatterbot
You say the strins are of type unicode: are you using from __future__ import unicode_literals? Also, which line raises the decode error? Because if the strings are unicode, they shouldn't be decoded (they are all already decoded), so there shouldn't be any decode errors either. - lenz
Don't change the default encoding. setdefaultencoding is disabled for a reason (libraries expect the default to be ascii). - Mark Tolonen
#coding declares the encoding of your source file. Make sure you actually save your source file in the declared encoding. - Mark Tolonen
@lenz yes i am using from __future__ import unicode_literals. The decode error is raised inside the train("Koliko imam še dopusta?", "Letos imate še 19 dni dopusta.") method. - matiOS
@MarkTolonen, ok, noted, I will remove that from my code. I saw that in some other stackoverflow answer to a similar question, and it was marked as correct in that thread. I think it is saved as utf-8, I did that thing in sublime, which the answer below is suggesting. That's what i meant with "I also changed the encoding of all used files via my editor (Sublime text 3) to utf-8". But how do i know that after doing that my file is actualy in utf-8 encoding? When I save, it writes a status in the program footer, on where the file is saved and then in parentheses it says utf-8. - matiOS

2 Answers

0
votes

The strings from your example are not of type unicode.

Otherwise Python would not throw the UnicodeDecodeError.
This type of error says that at a certain step of program's execution Python tries to decode byte-string into unicode but for some reason fails.

In your case the reason is that:

  • decoding is configured by utf-8
  • your source file is not in utf-8 and almost certainly in cp1252: 
    import unicodedata

b = '\x9a'

# u = b.decode('utf-8') # UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a 
                        # in position 0: invalid start byte

u = b.decode('cp1252')

print unicodedata.name(u) # LATIN SMALL LETTER S WITH CARON
print u # š

    So, the 0x9a byte from your cp1252 source can't be decoded with utf-8.

    The best solution is to do nothing except convertation your source to utf-8.
    With Sublime Text 3 you can easily do it by: File -> Reopen with Encoding -> UTF-8.
    But don't forget to Ctrl+C your source code before the convertation beacuse just after that all your š, č, ž chars wil be replaced with ?.

    • 0
    votes

    Some of our friends are already suggested good part solutions, However again I would like combine all the solutions into one.

    And author @gunthercox suggested some guidelines are described here http://chatterbot.readthedocs.io/en/stable/encoding.html#how-do-i-fix-python-encoding-errors

    # -*- coding: utf-8 -*-
from chatterbot import ChatBot

# Create a new chat bot named Test
chatBot = ChatBot(
    'Test',
    trainer='chatterbot.trainers.ListTrainer'
)

chatBot.train([
    "Koliko imam še dopusta?",
    "Letos imate še 19 dni dopusta.",
])

    Python Terminal

    >>> # -*- coding: utf-8 -*-
... from chatterbot import ChatBot
>>> 
>>> # Create a new chat bot named Test
... chatBot = ChatBot(
...     'Test',
...     trainer='chatterbot.trainers.ListTrainer'
... )
>>> 
>>> chatBot.train([
...     "Koliko imam še dopusta?",
...     "Letos imate še 19 dni dopusta.",
... ])
List Trainer: [####################] 100%
>>>