SQLAlchemy - how to avoid an error: UnicodeEncodeError: 'latin-1' codec can't encode characters

2
votes

Here is model column: 

title = Column(Unicode(100))

When I try to add a record to db with cyrillic letters in title I get an error:

UnicodeEncodeError: latin-1' codec can't encode characters on position ...

If i encode it with utf-8 

title = request.POST['title'].encode('utf-8')

I get next error:

ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit btystrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

When I decode utf-8

title = request.POST['title'].decode('utf-8')

There is new error

UnicodeDecodeError: 'ascii' codec can't encode characters...

What to do?

EDIT:

I'm using Sqlite3, I thought it may be important.

EDIT 2: (code and traceback)

My code:

title = request.POST['title']

new_model = Model(
    ...
    title = title,
    ...
)
DBSession.add(new_model)
DBSession.flush()

And Traceback (most recent call last):

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/performance.py", line 55, in resource_timer_handler

result = handler(request)

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/tweens.py", line 20, in excview_tween

response = handler(request)

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid_tm-0.3-py2.6.egg/pyramid_tm/init.py", line 61, in tm_tween

response = handler(request)

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/router.py", line 164, in handle_request

response = view_callable(context, request)

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/config/views.py", line 316, in rendered_view

result = view(context, request)

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/config/views.py", line 426, in _requestonly_view

response = view(request)

File "/home/p/work/SUN/sunviver/sunviver/views/advert.py", line 187, in create

return HTTPFound(location=new_advert.url())

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/httpexceptions.py", line 444, in init

body_template=body_template, location=location, **kw)

File "/home/p/work/SUN/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/httpexceptions.py", line 213, in init

Response.init(self, status=status, **kw)

File "build/bdist.linux-i686/egg/webob/response.py", line 141, in init

setattr(self, name, value)

File "build/bdist.linux-i686/egg/webob/descriptors.py", line 112, in fset

value = value.encode('latin-1')

And new_advert.url():

def url(self):
    return '/%s/%s' % (self.id, self.title.replace(' ', '_'))

@J.F. Sebastian

Traceback (most recent call last): File "/usr/lib/python2.6/wsgiref/handlers.py", line 93, in run

self.result = application(self.environ, self.start_response)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/router.py", line 187, in call

response = self.handle_request(request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/toolbar.py", line 157, in toolbar_tween

toolbar.process_response(response)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/toolbar.py", line 55, in process_response

vars, request=request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/renderers.py", line 81, in render

return helper.render(value, None, request=request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/renderers.py", line 420, in render

result = renderer(value, system_values)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 162, in call

reraise(MakoRenderingException(errtext), None, exc_info[2])

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 154, in call

result = template.render_unicode(**system)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/template.py", line 311, in render_unicode

as_unicode=True)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 660, in _render

**_kwargs_for_callable(callable_, data))

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 692, in _render_context

_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 718, in _exec_template

callable_(context, *args, **kwargs)

File "pyramid_debugtoolbar_templates_toolbar_mako", line 117, in render_body

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/headers.py", line 54, in content

vars, self.request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/init.py", line 24, in render

return render(template_name, vars, request=request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/renderers.py", line 81, in render

return helper.render(value, None, request=request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/renderers.py", line 420, in render

result = renderer(value, system_values)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 162, in call

reraise(MakoRenderingException(errtext), None, exc_info[2])

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 154, in call

result = template.render_unicode(**system)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/template.py", line 311, in render_unicode

as_unicode=True)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 660, in _render

**_kwargs_for_callable(callable_, data))

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 692, in _render_context

_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 718, in _exec_template

callable_(context, *args, **kwargs)

File "pyramid_debugtoolbar_panels_templates_headers_mako", line 34, in render_body

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/MarkupSafe-0.15-py2.6.egg/markupsafe/_native.py", line 21, in escape

return Markup(unicode(s)

MakoRenderingException:

Traceback (most recent call last):

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 154, in call

result = template.render_unicode(**system)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/template.py", line 311, in render_unicode

as_unicode=True)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 660, in _render

**_kwargs_for_callable(callable_, data))

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 692, in _render_context

_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 718, in _exec_template

callable_(context, *args, **kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/templates/toolbar.mako", line 61, in render_body

${panel.content()|n}

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/headers.py", line 54, in content

vars, self.request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/init.py", line 24, in render

return render(template_name, vars, request=request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/renderers.py", line 81, in render

return helper.render(value, None, request=request)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/renderers.py", line 420, in render

result = renderer(value, system_values)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 162, in call

reraise(MakoRenderingException(errtext), None, exc_info[2])

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 154, in call

result = template.render_unicode(**system)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/template.py", line 311, in render_unicode

as_unicode=True)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 660, in _render

**_kwargs_for_callable(callable_, data))

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 692, in _render_context

_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 718, in _exec_template

callable_(context, *args, **kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/templates/headers.mako", line 13, in render_body

${value|h}

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/MarkupSafe-0.15-py2.6.egg/markupsafe/_native.py", line 21, in escape

return Markup(unicode(s)

MakoRenderingException:

Traceback (most recent call last):

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid-1.3a3-py2.6.egg/pyramid/mako_templating.py", line 154, in call

result = template.render_unicode(**system)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/template.py", line 311, in render_unicode

as_unicode=True)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 660, in _render

**_kwargs_for_callable(callable_, data))

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 692, in _render_context

_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/Mako-0.5.0-py2.6.egg/mako/runtime.py", line 718, in _exec_template

callable_(context, *args, **kwargs)

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/pyramid_debugtoolbar-0.9.7-py2.6.egg/pyramid_debugtoolbar/panels/templates/headers.mako", line 13, in render_body

${value|h}

File "/home/ponomar/v/ENV/lib/python2.6/site-packages/MarkupSafe-0.15-py2.6.egg/markupsafe/_native.py", line 21, in escape

return Markup(unicode(s)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 7: ordinal not in range(128)

1
pythonsqlalchemy
Check that there are no non-ascii characters in new_advert.url(). - jfs
@J.F.Sebastian I've added new_advert.url() to the bottom of the question - Vitalii Ponomar
try path = '/%s/%s' % (self.id, self.title.replace(' ', '_')); return urllib.quote(path.encode('utf-8')) - jfs
@J.F.Sebastian - Didn't help. I think that problem may be in Pyramid framework urls, because I've tried to open any article with cyrillic letters in title and it shows me now A server error occurred. Please contact the administrator. even in debug mode (without any traceback), but it opens articles without Russian letters. Will search at Pyramid docs. But thanks anyway, you really helped when pointed on urls. - Vitalii Ponomar
"didn't help" is not very specific. Is it the same error? Show traceback. - jfs

1 Answers

2
votes

Encoding converts Unicode strings to byte strings. You need to go in the other direction, decoding: request.POST['title'].decode('utf-8') (assuming request.POST['title'] is a UTF8-encoded str).