Context Navigation

← Previous Ticket
Next Ticket →

#30887 closed Bug (duplicate)

UnicodeEncodeError logging error with saving binary with PyMySQL driver.

Reported by:	Louis	Owned by:	nobody
Component:	Database layer (models, ORM)	Version:	2.1
Severity:	Normal	Keywords:
Cc:		Triage Stage:	Unreviewed
Has patch:	no	Needs documentation:	no
Needs tests:	no	Patch needs improvement:	no
Easy pickings:	no	UI/UX:	no

Description

from django.db import models

class JobContext(models.Model):
    data = models.BinaryField()

    class Meta:
        verbose_name = 'JobContext'

JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')

raise exception below:
Traceback (most recent call last):

File "/usr/local/lib/python3.7/logging/init.py", line 1028, in emit

stream.write(msg + self.terminator)

UnicodeEncodeError: 'utf-8' codec can't encode character '\udc80' in position 173: surrogates not allowed

due to File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 111, in execute

logger.debug(                 
                '(%.3f) %s; args=%s', duration, sql, params,                                   
                extra={'duration': duration, 'sql': sql, 'params': params}                                   
            )

Change History (9)

follow-up: 3 comment:1 by Carlton Gibson, 5 years ago

Resolution:	→ worksforme
Status:	new → closed

Hmmm. This seems to be working for me on master.

Creating a model as described works as expected.

In [1]: from app.models import JobContext                                                                                    

In [2]: JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')                                                                                                                      
2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E'); args=[<memory at 0x103da5888>] [utils.py:130]
2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E'); args=[<memory at 0x103da5888>] [utils.py:130]
Out[2]: <JobContext: JobContext object (1)>

In particular the error isn't thrown at the logger.debug() call.

What are your Python/Django versions? (I see you tagged 2.0)

Are you able to provide a sample project that reproduces?

Thanks.

comment:2 by Simon Charette, 5 years ago

It would also be useful to know which database backend you were using because params might be massaged to different types depending on which backend your are using.

Version 1, edited 5 years ago by Simon Charette (previous) (next) (diff)

in reply to: 1 comment:3 by Louis, 5 years ago

Replying to Carlton Gibson:

Hmmm. This seems to be working for me on master.

Creating a model as described works as expected.

In [1]: from app.models import JobContext                                                                                    

In [2]: JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')                                                                                                                      
2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E'); args=[<memory at 0x103da5888>] [utils.py:130]
2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E'); args=[<memory at 0x103da5888>] [utils.py:130]
Out[2]: <JobContext: JobContext object (1)>

In particular the error isn't thrown at the logger.debug() call.

What are your Python/Django versions? (I see you tagged 2.0)

Python 3.7.4 with Django 2.1.3

Are you able to provide a sample project that reproduces?

https://github.com/somedayiamold/djangotest
Thanks.

Last edited 5 years ago by Carlton Gibson (previous) (diff)

comment:4 by Louis, 5 years ago

Version:	2.0 → 2.1

comment:5 by Carlton Gibson, 5 years ago

Resolution:	worksforme
Status:	closed → new

OK, thanks for the follow-up. Let me reopen to investigate.

follow-up: 7 comment:6 by Carlton Gibson, 5 years ago

Resolution:	→ invalid
Status:	new → closed
Summary:	UnicodeEncodeError problem while saving binary to database → UnicodeEncodeError logging error with saving binary with PyMySQL driver.

Thanks for the docker image. Very helpful!

I think this is a bug with PyMySQL. I suggest moving to mysqlclient, which is the recommended driver now.

I'm able to reproduce the bug with your image but, changing the driver (and removing the couple of lines from djangotest/__init__.py) resolves the issue:

root@11f9f5ba1dfb:/usr/src/app# pip freeze
Django==2.1.3
PyMySQL==0.9.3
pytz==2019.3
root@11f9f5ba1dfb:/usr/src/app# pip uninstall PyMySQL
Uninstalling PyMySQL-0.9.3:
  Would remove:
    /usr/local/lib/python3.7/site-packages/PyMySQL-0.9.3.dist-info/*
    /usr/local/lib/python3.7/site-packages/pymysql/*
Proceed (y/n)? y
  Successfully uninstalled PyMySQL-0.9.3
root@11f9f5ba1dfb:/usr/src/app# pip install mysqlclient
Collecting mysqlclient
  Downloading https://files.pythonhosted.org/packages/4d/38/c5f8bac9c50f3042c8f05615f84206f77f03db79781db841898fde1bb284/mysqlclient-1.4.4.tar.gz (86kB)
     |████████████████████████████████| 92kB 824kB/s 
Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/a0/04/57/031b9b01df38999df7dc7f4ee998a98ecdbd2d781f73e3ffbf
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.4.4
WARNING: You are using pip version 19.1.1, however version 19.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py makemigrations
Migrations for 'demo':
  demo/migrations/0001_initial.py
    - Create model JobContext
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, demo, sessions
Running migrations:
  No migrations to apply.
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py shell
Python 3.7.3 (default, Jun 11 2019, 01:05:09) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from demo.models import JobContext
>>> JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
<JobContext: JobContext object (2)>
>>> 
now exiting InteractiveConsole...
root@11f9f5ba1dfb:/usr/src/app# pip freeze
Django==2.1.3
mysqlclient==1.4.4
pytz==2019.3

I'm going to close on that basis.

in reply to: 6 comment:7 by Louis, 5 years ago

Replying to Carlton Gibson:

Thanks for the docker image. Very helpful!

I think this is a bug with PyMySQL. I suggest moving to mysqlclient, which is the recommended driver now.

I'm able to reproduce the bug with your image but, changing the driver (and removing the couple of lines from djangotest/__init__.py) resolves the issue:

root@11f9f5ba1dfb:/usr/src/app# pip freeze
Django==2.1.3
PyMySQL==0.9.3
pytz==2019.3
root@11f9f5ba1dfb:/usr/src/app# pip uninstall PyMySQL
Uninstalling PyMySQL-0.9.3:
  Would remove:
    /usr/local/lib/python3.7/site-packages/PyMySQL-0.9.3.dist-info/*
    /usr/local/lib/python3.7/site-packages/pymysql/*
Proceed (y/n)? y
  Successfully uninstalled PyMySQL-0.9.3
root@11f9f5ba1dfb:/usr/src/app# pip install mysqlclient
Collecting mysqlclient
  Downloading https://files.pythonhosted.org/packages/4d/38/c5f8bac9c50f3042c8f05615f84206f77f03db79781db841898fde1bb284/mysqlclient-1.4.4.tar.gz (86kB)
     |████████████████████████████████| 92kB 824kB/s 
Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/a0/04/57/031b9b01df38999df7dc7f4ee998a98ecdbd2d781f73e3ffbf
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.4.4
WARNING: You are using pip version 19.1.1, however version 19.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py makemigrations
Migrations for 'demo':
  demo/migrations/0001_initial.py
    - Create model JobContext
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, demo, sessions
Running migrations:
  No migrations to apply.
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py shell
Python 3.7.3 (default, Jun 11 2019, 01:05:09) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from demo.models import JobContext
>>> JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
<JobContext: JobContext object (2)>
>>> 
now exiting InteractiveConsole...
root@11f9f5ba1dfb:/usr/src/app# pip freeze
Django==2.1.3
mysqlclient==1.4.4
pytz==2019.3

I'm going to close on that basis.

OK, it turned out to be pymysql compatible problem, thanks for the help, will follow your suggestion then.

follow-up: 9 comment:8 by Simon Charette, 5 years ago

Resolution:	invalid → duplicate

For the record this is tracked in #30380 and was fixed in a41b09266dcdd01036d59d76fe926fe0386aaade which is part of the upcoming Django 3.0 release.

in reply to: 8 comment:9 by Louis, 5 years ago

Replying to Simon Charette:

For the record this is tracked in #30380 and was fixed in a41b09266dcdd01036d59d76fe926fe0386aaade which is part of the upcoming Django 3.0 release.

good to know, thanks.

Note: See TracTickets for help on using tickets.

Download in other formats:

Issues