Opened 3 years ago

Closed 11 months ago

#23222 closed Bug (wontfix)

Empty BinaryField != b'' on Python 2

Reported by: wkschwartz@… Owned by: Grzegorz Ślusarek
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: py2
Cc: cmawebsite@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by William Schwartz)

Problem

In Python 2, at least under SQLite, the initial value for an empty binary field behaves inconsistently. The ORM thinks it's an empty bytes: b''. The database connection manager thinks it's a buffer. Now, the buffer evaluates to False and has zero length. So it'll mostly work. But not always -- and most importantly to me, not in my unit tests!

See http://stackoverflow.com/q/12625557/1286628

Note this was not a problem under Python 3.4.

Steps to Reproduce

Using Python 2.7.8, SQLite, and either Django 1.7rc2 or Django @ edcc75e5ac5b9dc2f174580e7adacd3be586f8bd (HEAD at the time of this writing; the error exists in both places)

  1. Make a new project and a new app and add the app to settings.py
  2. Fill in app/models.py as follows
from django.db import models                                                                                                         
class BinModel(models.Model):                                                                                                        
    data = models.BinaryField()                                                                                                      
  1. Run from the command line:
(venv) $ ./manage.py makemigrations app && ./manage.py migrate && ./manage.py shell
  1. Run from the resulting Python shell
>>> from app import models; m = models.BinModel(); m.save(); n = models.BinModel.objects.get()
>>> m.data
''
>>> m.data == b''
True
>>> n.data
<read-write buffer ptr 0x10eaa62b0, size 0 at 0x10eaa6270>
>>> n.data == b''
False
>>> bool(n.data)
False
>>> len(n.data)
0
>>> bytes(n.data)
''

Note that the same problem persisted when I had a default value for the field. There was no problem under Python 3.4.

Change History (9)

comment:1 Changed 3 years ago by Collin Anderson

Cc: cmawebsite@… added
Severity: Release blockerNormal
Version: 1.7-rc-2master

I'm seeing the same behavior on django 1.6.0 and stable/1.6.x, so I don't think this is a release blocker.

Last edited 3 years ago by Collin Anderson (previous) (diff)

comment:2 Changed 3 years ago by Areski Belaid

I dont think it's SQLite related, I have the same output using PostgreSQL

comment:3 Changed 3 years ago by Collin Anderson

Summary: Empty BinaryField != b'' in Python 2/SQLiteEmpty BinaryField != b'' on Python 2

comment:4 Changed 3 years ago by Baptiste Mispelon

Triage Stage: UnreviewedAccepted

Marking this as accepted per previous comments.

This seem like it might be caused by six using buffers on python 2 [1].

Thanks.

[1] https://github.com/django/django/blob/master/django/utils/six.py#L681

comment:5 Changed 3 years ago by ppYang

I'm not sure, if it's only a problem with empty BinaryField.

I try to add a test function in model_fields.tests.BinaryFieldTests, as below:

    def test_retrieve_compare_non_empty_field(self):
        dm = DataModel(data=b'test')
        self.assertTrue(dm.data == b'test')
        dm.save()
        dm = DataModel.objects.get(pk=dm.pk)
        self.assertTrue(dm.data == b'test')

On python2, the first assertion will be passed, and the second one will be failed.

As the data field is a buffer type after retrieving, we can't do a simple compare.

We can see the definition:

class BinaryField(Field):
    ...
    def get_db_prep_value(self, value, connection, prepared=False):
        value = super(BinaryField, self).get_db_prep_value(value, connection, prepared)
        if value is not None:
            return connection.Database.Binary(value)
        return value

I can't go further as I know little about six, and hope this can help:)

Last edited 3 years ago by ppYang (previous) (diff)

comment:6 Changed 3 years ago by Grzegorz Ślusarek

Owner: changed from nobody to Grzegorz Ślusarek
Status: newassigned

comment:7 Changed 3 years ago by William Schwartz

Description: modified (diff)

comment:8 Changed 18 months ago by Tim Graham

Keywords: py2 added

comment:9 Changed 11 months ago by Tim Graham

Resolution: wontfix
Status: assignedclosed

Closing due to the end of Python 2 support in master in a couple weeks.

Note: See TracTickets for help on using tickets.
Back to Top