Opened 2 years ago

Last modified 6 months ago

#23222 assigned Bug

Empty BinaryField != b'' on Python 2

Reported by: wkschwartz@… Owned by: Grzegorz Ślusarek
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: py2
Cc: cmawebsite@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by William Schwartz)

Problem

In Python 2, at least under SQLite, the initial value for an empty binary field behaves inconsistently. The ORM thinks it's an empty bytes: b''. The database connection manager thinks it's a buffer. Now, the buffer evaluates to False and has zero length. So it'll mostly work. But not always -- and most importantly to me, not in my unit tests!

See http://stackoverflow.com/q/12625557/1286628

Note this was not a problem under Python 3.4.

Steps to Reproduce

Using Python 2.7.8, SQLite, and either Django 1.7rc2 or Django @ edcc75e5ac5b9dc2f174580e7adacd3be586f8bd (HEAD at the time of this writing; the error exists in both places)

  1. Make a new project and a new app and add the app to settings.py
  2. Fill in app/models.py as follows
from django.db import models                                                                                                         
class BinModel(models.Model):                                                                                                        
    data = models.BinaryField()                                                                                                      
  1. Run from the command line:
(venv) $ ./manage.py makemigrations app && ./manage.py migrate && ./manage.py shell
  1. Run from the resulting Python shell
>>> from app import models; m = models.BinModel(); m.save(); n = models.BinModel.objects.get()
>>> m.data
''
>>> m.data == b''
True
>>> n.data
<read-write buffer ptr 0x10eaa62b0, size 0 at 0x10eaa6270>
>>> n.data == b''
False
>>> bool(n.data)
False
>>> len(n.data)
0
>>> bytes(n.data)
''

Note that the same problem persisted when I had a default value for the field. There was no problem under Python 3.4.

Change History (8)

comment:1 Changed 2 years ago by Collin Anderson

Cc: cmawebsite@… added
Severity: Release blockerNormal
Version: 1.7-rc-2master

I'm seeing the same behavior on django 1.6.0 and stable/1.6.x, so I don't think this is a release blocker.

Last edited 2 years ago by Collin Anderson (previous) (diff)

comment:2 Changed 2 years ago by Areski Belaid

I dont think it's SQLite related, I have the same output using PostgreSQL

comment:3 Changed 2 years ago by Collin Anderson

Summary: Empty BinaryField != b'' in Python 2/SQLiteEmpty BinaryField != b'' on Python 2

comment:4 Changed 2 years ago by Baptiste Mispelon

Triage Stage: UnreviewedAccepted

Marking this as accepted per previous comments.

This seem like it might be caused by six using buffers on python 2 [1].

Thanks.

[1] https://github.com/django/django/blob/master/django/utils/six.py#L681

comment:5 Changed 23 months ago by ppYang

I'm not sure, if it's only a problem with empty BinaryField.

I try to add a test function in model_fields.tests.BinaryFieldTests, as below:

    def test_retrieve_compare_non_empty_field(self):
        dm = DataModel(data=b'test')
        self.assertTrue(dm.data == b'test')
        dm.save()
        dm = DataModel.objects.get(pk=dm.pk)
        self.assertTrue(dm.data == b'test')

On python2, the first assertion will be passed, and the second one will be failed.

As the data field is a buffer type after retrieving, we can't do a simple compare.

We can see the definition:

class BinaryField(Field):
    ...
    def get_db_prep_value(self, value, connection, prepared=False):
        value = super(BinaryField, self).get_db_prep_value(value, connection, prepared)
        if value is not None:
            return connection.Database.Binary(value)
        return value

I can't go further as I know little about six, and hope this can help:)

Last edited 23 months ago by ppYang (previous) (diff)

comment:6 Changed 21 months ago by Grzegorz Ślusarek

Owner: changed from nobody to Grzegorz Ślusarek
Status: newassigned

comment:7 Changed 21 months ago by William Schwartz

Description: modified (diff)

comment:8 Changed 6 months ago by Tim Graham

Keywords: py2 added
Note: See TracTickets for help on using tickets.
Back to Top